Commit Graph

6 Commits (f813fb45637633f61ba73ace5d1a0acabd815897)

Author SHA1 Message Date
OxalisCu 3a381bc247
enhance: Bulkinsert supports null in csv formats (#35912)
see details in this issue
https://github.com/milvus-io/milvus/issues/35911

---------

Signed-off-by: OxalisCu <2127298698@qq.com>
2024-09-09 19:17:07 +08:00
OxalisCu ed4eaffc9d
enhance: add csv support for bulkinsert (#34938)
See this issue for details: #34937

---------

Signed-off-by: OxalisCu <2127298698@qq.com>
2024-08-21 17:47:01 +08:00
yihao.dai 1594122c0a
enhance: Make the dynamic field file optional during numpy import (#32596)
1. Make the dynamic field file optional during numpy import
2. Add integration importing test with dynamic
3. Disallow file of pk when autoID=true during numpy import

issue: https://github.com/milvus-io/milvus/issues/32542

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-28 19:39:25 +08:00
yihao.dai 31cf849f68
enhance: Support retriving file size from importutilv2.Reader (#31533)
To reduce the overhead caused by listing the S3 objects, add an
interface to importutil.Reader to retrieve file sizes.

issue: https://github.com/milvus-io/milvus/issues/31532,
https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-25 20:29:07 +08:00
yihao.dai c5918290e6
feat: Add import executor and manager for datanode (#29438)
This PR introduces novel importv2 roles for datanode:
1. Executor: To execute tasks, a import task will be divided into the
following steps: read data -> hash data -> sync data;
2. Manager: To manage all the tasks;

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-31 20:45:04 +08:00
yihao.dai 3561586edf
feat: Add import reader for binlog (#28910)
This PR defines the new import reader interfaces and implement a binlog
reader for import.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-05 11:48:47 +08:00