c45f8a2946
fix: Import data from parquet file in streaming way ( #29514 )
...
issue: #29292
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-27 15:30:46 +08:00
2b05460ef9
enhance: Make import-related error message clearer ( #28978 )
...
issue: #28976
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-08 10:12:38 +08:00
f5f4f0872e
enhance: Support importing data with parquet file ( #28608 )
...
issue: #28272
Numpy does not support array type import.
Array type data is imported through parquet.
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-29 20:52:27 +08:00
1b7a503f89
enhance: Revert import support csv format ( #28760 )
...
Revert import support csv format.
issue: #28778
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-28 14:32:27 +08:00
c29b60e18e
enhance: Support Array DataType for bulk_insert ( #28341 )
...
issue: #28272
Support array DataType for bulk_insert with json, binlog files.
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-27 13:50:27 +08:00
3f6b203018
Fix bulkinsert bug that segments are compacted after import ( #28192 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-11-07 15:14:26 +08:00
e88212ba4b
Add CSV file import function ( #27149 )
...
Signed-off-by: kuma <675613722@qq.com>
Co-authored-by: kuma <675613722@qq.com>
2023-10-31 22:47:23 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
a6808e6484
Fix bulkinsert row count error ( #25869 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-07-25 11:29:00 +08:00
96c987ed62
Bulkinsert supports partition keys ( #25284 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-07-11 15:18:28 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
3cb75997a3
Fix a bug of bulkinsert dynamic schema ( #24405 ) ( #24527 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-05-31 14:41:41 +08:00
7744573d3d
support parms for import maxfilesize( #24191 ) ( #24192 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-05-18 18:59:27 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
9f8095996b
Report bulkinsert progress ( #21612 ) ( #21638 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2023-01-11 17:37:44 +08:00
aab20a4031
Streaming read numpy ( #21579 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-01-10 14:53:38 +08:00
b3da554705
Increase bulk insert file size limit ( #21526 )
...
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2023-01-06 13:09:36 +08:00
33a102c6fc
Refine bulkinsert ( #20986 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-12-07 14:53:18 +08:00
8da44638ac
Refine json parser for import ( #20545 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-11-16 19:05:08 +08:00
53850e340c
Dont expire pending import tasks ( #20431 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-11-10 10:25:08 +08:00
a7bed1c927
Pass backup flag through Import request options ( #20334 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-11-08 11:33:03 +08:00
b847c425e1
Fix listImportTasks bug ( #20301 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-11-07 17:11:02 +08:00
cd3b3d3da6
Only allow one file for row-based import ( #20234 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-11-02 10:23:35 +08:00
c6151ad351
Parse utf32 string of numpy file ( #20176 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-10-31 17:39:34 +08:00
bee66631e3
Refine bulkload ( #19671 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-10-27 16:21:34 +08:00
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
8b8df0a5e9
Do not use DataCoord context when DataNode is handling import task ( #19732 )
...
So that when DataCoord is done, DataNode can still proceed.
/kind bug
issue: #19730
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-10-14 15:15:24 +08:00
a561614d48
Fix bulkload failed bug ( #19702 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-10-12 10:27:23 +08:00
2599a3ece0
Import native binlog files ( #19447 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-09-30 10:32:54 +08:00
838a633584
Add ctx parameter for ChunkManager methods ( #19546 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-29 16:18:56 +08:00
b30c9d4f60
Support Bulk Load in Milvus 2.2 ( #18982 )
...
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-09-26 18:06:54 +08:00
d7f38a803d
Separate some proto files ( #19218 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
371a44f018
Fix a bug of bulkload ( #17620 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-06-17 20:52:11 +08:00
ab9b186f15
Change bulkload bufsize for row-based parser ( #17473 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-06-14 16:18:09 +08:00
9210299706
Fix bulkload OOM issue ( #17461 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-06-09 16:52:07 +08:00
d525e95550
Bulkload not allow duplicate numpy file ( #17105 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-05-20 10:27:56 +08:00
168935f4cf
Fix bulkload bug for string primary key ( #16958 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-05-13 16:07:54 +08:00
8736372fd2
Fix bulkload bugs ( #16760 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-05-06 11:21:50 +08:00
21b5973253
Refine list import task api ( #16690 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-04-28 17:21:47 +08:00
bc5e9ec929
Add list import tasks api ( #16605 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-04-25 17:37:46 +08:00
a6a3b69d91
Report bulk load state ( #16555 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-04-21 21:37:42 +08:00
289e468a7a
Make bulk load fully work ( #16512 )
...
issue: #15604
/kind enhancement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-20 14:03:40 +08:00
4781db8a2a
Add datanode import ( #16414 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-04-12 22:19:34 +08:00
be8d9a8b6b
Support import numpy file ( #16348 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-04-03 11:27:29 +08:00
bd241cd28b
Add performance test case for import ( #16296 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-04-01 10:07:28 +08:00
ffa06c77b6
Import util functions ( #16237 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-03-30 16:25:30 +08:00
cai.zhang
groot
KumaJie
SimFG
congqixia
MrPresent-Han
jaime
Ten Thousand Leaves
wayblink
godchen