b45cf2d49f
enhance: Add max length check for csv import ( #37077 )
...
1. Add max length check for csv import.
2. Tidy import options.
3. Tidy common import util functions.
issue: https://github.com/milvus-io/milvus/issues/34150
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-25 14:37:29 +08:00
31cf849f68
enhance: Support retriving file size from importutilv2.Reader ( #31533 )
...
To reduce the overhead caused by listing the S3 objects, add an
interface to importutil.Reader to retrieve file sizes.
issue: https://github.com/milvus-io/milvus/issues/31532 ,
https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-25 20:29:07 +08:00
c5918290e6
feat: Add import executor and manager for datanode ( #29438 )
...
This PR introduces novel importv2 roles for datanode:
1. Executor: To execute tasks, a import task will be divided into the
following steps: read data -> hash data -> sync data;
2. Manager: To manage all the tasks;
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-31 20:45:04 +08:00
23183ffb0f
feat: Add import reader for json ( #29252 )
...
This PR implements a new json reader for import.
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-05 18:12:48 +08:00
yihao.dai