0a62d6d509
enhance: Add Size interface to FileReader to eliminate the StatObject call during Read ( #42911 )
...
issue: #42907
master pr: #42908
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-06-25 15:26:42 +08:00
9b17108b50
fix: [2.5] Fix import reader goroutine leak ( #41870 )
...
Close the chunk manager's reader after the import completes to prevent
goroutine leaks.
issues: https://github.com/milvus-io/milvus/issues/41868
pr: https://github.com/milvus-io/milvus/pull/41869
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-05-15 22:20:23 +08:00
18eb627533
fix: [2.5] Update logging context and upgrade dependencies ( #41319 )
...
- issue: #41291
- pr: #41318
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-04-24 23:50:40 +08:00
cba8feade3
enhance: [2.5] Add UTF-8 string validation for import ( #40746 )
...
issue: https://github.com/milvus-io/milvus/issues/40684
pr: https://github.com/milvus-io/milvus/pull/40694
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-31 21:08:21 +08:00
b0ad3d9444
enhance: [2.5] Improve import error msgs ( #40597 )
...
issue: https://github.com/milvus-io/milvus/issues/40208
pr: https://github.com/milvus-io/milvus/pull/40567
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-12 14:36:08 +08:00
709594f158
enhance: [2.5] Use v2 package name for pkg module ( #40117 )
...
Cherry-pick from master
pr: #39990
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-23 00:46:01 +08:00
f896b0e3e8
fix: Bulk insert failed when the nullable/default_value field is not exist( #39063 ) ( #39111 )
...
pr: #39063
issue: #39036
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-01-09 19:27:04 +08:00
b0bd290a6e
enhance: Use internal json(sonic) to replace std json lib ( #37708 )
...
Related to #35020
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-18 10:46:31 +08:00
b45cf2d49f
enhance: Add max length check for csv import ( #37077 )
...
1. Add max length check for csv import.
2. Tidy import options.
3. Tidy common import util functions.
issue: https://github.com/milvus-io/milvus/issues/34150
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-25 14:37:29 +08:00
463c47ced1
enhance: support default value in import ( #36700 )
...
https://github.com/milvus-io/milvus/issues/31728
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-17 12:05:24 +08:00
82c5cf2fa2
feat: add bulk insert support for Functions ( #36715 )
...
issue: https://github.com/milvus-io/milvus/issues/35853 and
https://github.com/milvus-io/milvus/issues/35856
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-10-12 17:19:20 +08:00
50fcfe8ef1
enhance: add nan and inf check ( #35683 )
...
fix #35594
add float check on files
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-08-25 15:22:57 +08:00
80a7c78f28
enhance: import supports null in parquet and json formats ( #35558 )
...
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-20 16:50:55 +08:00
3948bd4e79
fix: Added check for validating varchar,array max length ( #35499 )
...
issue : https://github.com/milvus-io/milvus/issues/34150
This is for numpy,parquet,json readers.
---------
Signed-off-by: Nischay Yadav <nischay.yadav@ibm.com>
2024-08-20 11:42:55 +08:00
b1d46eb34b
fix: Fix multiple vector fields import ( #33723 )
...
1. Fix dim mismatch with multi-vector fields and JSON import
2. Enhance: do not display file ID in GetImportResponse.
issue: https://github.com/milvus-io/milvus/issues/33681 ,
https://github.com/milvus-io/milvus/issues/33682
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-10 21:57:54 +08:00
4004e4c545
enhance: Optimize bulk insert unittest ( #33224 )
...
Issue: #22837
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-24 10:23:41 +08:00
b560602885
enhance: Store SparseFloatVector into parquet as JSON string ( #33101 )
...
Issue: #22837
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-17 15:01:37 +08:00
4ef163fb70
enhance: Support readable JSON file import for Float16/BFloat16/SparseFloat ( #33064 )
...
Issue: #22837
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-16 14:47:35 +08:00
dc89c6f810
enhance: remove duplicated data generation APIs for bulk insert test ( #32889 )
...
Issue: #22837
including following changes:
1. Add API CreateInsertData() and BuildArrayData() in
internal/util/testutil
2. Remove duplicated test APIs from importutilv2 unittest and bulk
insert integration test
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-10 15:27:31 +08:00
bcdbd1966e
feat: Support sparse float vector bulk insert for binlog/json/parquet ( #32649 )
...
Issue: #22837
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-07 18:43:30 +08:00
5fc439c600
feat: Bulk insert support fp16/bf16 ( #32157 )
...
Issue: #22837
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-22 10:05:22 +08:00
1b5554c8cb
enhance: Support $meta key for json import ( #32013 )
...
During JSON import:
1. Allow the specification of the $meta key
2. Prohibit duplicated keys within the $meta field, for instance,
`{"id": 1, "vector": [], "x": 6, "$meta": {"x": 8}}`
issue: https://github.com/milvus-io/milvus/issues/31835
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-10 17:27:17 +08:00
31cf849f68
enhance: Support retriving file size from importutilv2.Reader ( #31533 )
...
To reduce the overhead caused by listing the S3 objects, add an
interface to importutil.Reader to retrieve file sizes.
issue: https://github.com/milvus-io/milvus/issues/31532 ,
https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-25 20:29:07 +08:00
de2c95d00c
enhance: Constraint dynamic field as key-value format ( #31183 )
...
issue: #31051
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-12 12:45:03 +08:00
c5918290e6
feat: Add import executor and manager for datanode ( #29438 )
...
This PR introduces novel importv2 roles for datanode:
1. Executor: To execute tasks, a import task will be divided into the
following steps: read data -> hash data -> sync data;
2. Manager: To manage all the tasks;
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-31 20:45:04 +08:00
23183ffb0f
feat: Add import reader for json ( #29252 )
...
This PR implements a new json reader for import.
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-05 18:12:48 +08:00
cai.zhang
yihao.dai
SimFG
congqixia
smellthemoon
Buqian Zheng
Xiaofan
nish112022
Cai Yudong