c5918290e6
feat: Add import executor and manager for datanode ( #29438 )
...
This PR introduces novel importv2 roles for datanode:
1. Executor: To execute tasks, a import task will be divided into the
following steps: read data -> hash data -> sync data;
2. Manager: To manage all the tasks;
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-31 20:45:04 +08:00
6cf2f09b60
feat: Support tencent cloud object storage for milvus ( #30163 )
...
issue: #30162
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-01-23 11:28:56 +08:00
6bfa826320
fix: Fix bug for read data from azure ( #30007 )
...
issue: #30005
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-01-22 15:44:54 +08:00
e429965f32
Add float16 approve for multi-type part ( #28427 )
...
issue：https://github.com/milvus-io/milvus/issues/22837
Add bfloat16 vector, add the index part of float16 vector.
Signed-off-by: Writer-X <1256866856@qq.com>
2024-01-11 15:48:51 +08:00
f18a7191f2
enhance: make `ColumnBasedInsertMsgToInsertData` check field missing ( #29758 )
...
fix : #29757
In previous code, `ColumnBasedInsertMsgToInsertData` adds empty field if
the insertMsg parameter does not have the column schema defined. This
may lead to unexpected behavior of caller functions.
This PR:
- Add column missing check
- Add column length check
- Generate BlobInfo for ColumnBasedInsertMsgToInsertData result
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-09 11:50:48 +08:00
3d07b6682c
feat: Add import reader for numpy ( #29253 )
...
This PR implements a new numpy reader for import.
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-08 19:42:49 +08:00
97e4ec5a69
enhance: use random root path for minio unit tests ( #29753 )
...
this avoids the conflicts while running multiple unit tests
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2024-01-08 15:58:48 +08:00
23183ffb0f
feat: Add import reader for json ( #29252 )
...
This PR implements a new json reader for import.
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-05 18:12:48 +08:00
1c1f2a1371
enhance:change some logs ( #29579 )
...
related #29588
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-05 16:12:48 +08:00
3561586edf
feat: Add import reader for binlog ( #28910 )
...
This PR defines the new import reader interfaces and implement a binlog
reader for import.
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-05 11:48:47 +08:00
dc8b5c1130
enhance: Read azure file without ReadAll ( #29602 )
...
issue: #29292
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-01-04 20:50:46 +08:00
3f46c6d459
feat: support inverted index ( #28783 )
...
issue: https://github.com/milvus-io/milvus/issues/27704
Add inverted index for some data types in Milvus. This index type can
save a lot of memory compared to loading all data into RAM and speed up
the term query and range query.
Supported: `INT8`, `INT16`, `INT32`, `INT64`, `FLOAT`, `DOUBLE`, `BOOL`
and `VARCHAR`.
Not supported: `ARRAY` and `JSON`.
Note:
- The inverted index for `VARCHAR` is not designed to serve full-text
search now. We will treat every row as a whole keyword instead of
tokenizing it into multiple terms.
- The inverted index don't support retrieval well, so if you create
inverted index for field, those operations which depend on the raw data
will fallback to use chunk storage, which will bring some performance
loss. For example, comparisons between two columns and retrieval of
output fields.
The inverted index is very easy to be used.
Taking below collection as an example:
```python
fields = [
FieldSchema(name="pk", dtype=DataType.VARCHAR, is_primary=True, auto_id=False, max_length=100),
FieldSchema(name="int8", dtype=DataType.INT8),
FieldSchema(name="int16", dtype=DataType.INT16),
FieldSchema(name="int32", dtype=DataType.INT32),
FieldSchema(name="int64", dtype=DataType.INT64),
FieldSchema(name="float", dtype=DataType.FLOAT),
FieldSchema(name="double", dtype=DataType.DOUBLE),
FieldSchema(name="bool", dtype=DataType.BOOL),
FieldSchema(name="varchar", dtype=DataType.VARCHAR, max_length=1000),
FieldSchema(name="random", dtype=DataType.DOUBLE),
FieldSchema(name="embeddings", dtype=DataType.FLOAT_VECTOR, dim=dim),
]
schema = CollectionSchema(fields)
collection = Collection("demo", schema)
```
Then we can simply create inverted index for field via:
```python
index_type = "INVERTED"
collection.create_index("int8", {"index_type": index_type})
collection.create_index("int16", {"index_type": index_type})
collection.create_index("int32", {"index_type": index_type})
collection.create_index("int64", {"index_type": index_type})
collection.create_index("float", {"index_type": index_type})
collection.create_index("double", {"index_type": index_type})
collection.create_index("bool", {"index_type": index_type})
collection.create_index("varchar", {"index_type": index_type})
```
Then, term query and range query on the field can be speed up
automatically by the inverted index:
```python
result = collection.query(expr='int64 in [1, 2, 3]', output_fields=["pk"])
result = collection.query(expr='int64 < 5', output_fields=["pk"])
result = collection.query(expr='int64 > 2997', output_fields=["pk"])
result = collection.query(expr='1 < int64 < 5', output_fields=["pk"])
```
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-12-31 19:50:47 +08:00
ed644983e2
enhance: add param for bloomfilter( #29388 ) ( #29490 )
...
related: #29388
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-28 18:10:46 +08:00
6a86ac0ac6
fix: Align minio object storage ut to new minio server behavior ( #29014 )
...
See also #29013
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-06 15:42:43 +08:00
b4353ca4ce
enhance: Remove vector chunk manager ( #28569 )
...
We have implemented the chunkcache (in cpp) to retrieve vectors, hence
rendering the vectorchunkcache (in golang) obsolete.
issue: https://github.com/milvus-io/milvus/issues/28568
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-11-30 18:00:33 +08:00
aae7e62729
feat: Add levelzero compaction in DN ( #28470 )
...
See also: #27606
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-30 14:30:28 +08:00
f5f4f0872e
enhance: Support importing data with parquet file ( #28608 )
...
issue: #28272
Numpy does not support array type import.
Array type data is imported through parquet.
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-29 20:52:27 +08:00
4bd426dbe7
fix: Fix minio latency monitoring for get operation ( #28510 )
...
see also: https://github.com/milvus-io/milvus/issues/28509
Currently Minio latency monitoring for get operation only collects the
duration of getting object (which just returns an io.Reader and does not
really read from minio), this pr will correct this behavior.
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-11-28 10:00:27 +08:00
8a9ab69369
fix: Skip statslog generation flushing empty L0 segment ( #28733 )
...
See also #27675
When L0 segment contains only delta data, merged statslog shall be
skiped when performing sync task
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-25 15:10:25 +08:00
cc952e0486
enhance: optimize forwarding level0 deletions by respecting partition ( #28456 )
...
- Cache the level 0 deletions after loading level0 segments
- Divide the level 0 deletions by partition
related: #27349
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 18:24:22 +08:00
2b3fa8f67b
fix: Add length check for `storage.NewPrimaryKeyStats` ( #28576 )
...
See also #28575
Add zero-length check for `storage.NewPrimaryKeyStats`. This function
shall return error when non-positive rowNum passed.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-21 10:28:21 +08:00
59355cb3dc
Update arrow version to v12 ( #28425 )
...
issue: https://github.com/milvus-io/milvus/issues/28423
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-15 10:36:19 +08:00
e576271a24
Fix buffer FieldData has no `ElementType` and array logsize always zero ( #28295 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-09 14:16:20 +08:00
ece592a42f
Deliver L0 segments delete records ( #27722 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-07 01:44:18 +08:00
0252871d30
fix azure ListObjects ( #27931 )
...
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-01 11:34:14 +08:00
8ae9c947ae
Use OpenDAL to access object store ( #25642 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-11-01 09:00:14 +08:00
9658367a3c
Refine chunk manager errors ( #27590 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-31 12:18:15 +08:00
6c3f29d003
Identify service providers based on addresses ( #27907 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-10-25 17:28:10 +08:00
6060dd7ea8
Add chunk manager request timeout ( #27692 )
...
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-10-23 20:08:08 +08:00
7358c3527b
Add iterators ( #27643 )
...
See also: #27606
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-10-18 19:34:08 +08:00
2f201c25e2
Remove deprecated io/ioutil usage ( #27747 )
...
`io/ioutil` package is deprecated, use `io`,`os` package replacement
also added golangci-lint rule to block future reference
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: guoguangwu <guoguangwu@magic-shield.com>
2023-10-17 20:32:09 +08:00
2f16339aac
Enhance InsertData and FieldData ( #27436 )
...
1. Add NewInsertData
2. Add GetRowNum(), GetMemorySize(), and, Append() for InsertData
3. Add AppendRow() for FieldData for compaction
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-10-17 17:36:11 +08:00
670cb386e7
Add back `gocritic` linter and fix related issues ( #27289 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-22 10:05:26 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
cc9974979f
Add staticcheck linter and fix existing problems ( #27174 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-19 10:05:22 +08:00
4feb3fa7c6
support azure ( #26398 )
...
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-09-19 10:01:23 +08:00
9166011c4a
Add float16 vector ( #25852 )
...
Signed-off-by: Writer-X <1256866856@qq.com>
2023-09-08 10:03:16 +08:00
548c82eca5
Refactor storage.MergeInsertData() to optimize the merging process ( #26839 )
...
Benchmark Milvus with https://github.com/qdrant/vector-db-benchmark and
specify the datasets as 'deep-image-96-angular'. Meanwhile, do perf
profiling during 'upload + index' stage of vector-db-benchmark and see
the following hot spots.
39.59%--github.com/milvus-io/milvus/internal/storage.MergeInsertData
|
|--21.43%--github.com/milvus-io/milvus/internal/storage.MergeFieldData
| |
| |--17.22%--runtime.memmove
| |
| |--1.53%--asm_exc_page_fault
| ......
|
|--18.16%--runtime.memmove
|
|--1.66%--asm_exc_page_fault
......
The hot code path is in storage.MergeInsertData() which updates
buffer.buffer by creating a new 'InsertData' instance and merging both
the old buffer.buffer and addedBuffer into it. When it calls golang
runtime.memmove to move buffer.buffer which is with big size (>1M), the
hot spots appear.
To avoid the above overhead, update storage.MergeInsertData() by
appending addedBuffer to buffer.buffer, instead of moving buffer.buffer
and addedBuffer to a new 'InsertData'. This change removes the hot spots
'runtime.memmove' from perf profiling output. Additionally, the 'upload
+ index' time, which is one performance metric of vector-db-benchmark,
is reduced around 60% with this change.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
2023-09-05 21:41:48 +08:00
fb0705df1b
Decouple basetable and componentparam ( #26725 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-05 10:31:48 +08:00
411f9ac823
Upgrade minio-go and add region and virtual host config for segcore chunk manager ( #26194 )
...
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-08-11 10:37:36 +08:00
2770ac4df5
Fix nilness linter errors ( #26218 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-09 11:31:15 +08:00
2c6c7749e2
Enable print_log support json data type ( #26118 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-08-04 11:27:05 +08:00
f33451b3d8
Write the cache file to the cacheStorage.rootpath dir ( #25715 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-07-28 10:59:02 +08:00
94d6cbb238
Fix querynode panic when binlog ts wrong ( #25635 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-07-18 10:41:20 +08:00
33c2012675
Add more metrics ( #25081 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-26 17:52:44 +08:00
e8911ebda7
Add retry time when lazy load BF ( #25096 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-06-25 11:32:43 +08:00
3f4356df10
fix the spelling of `field` ( #25008 )
...
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-06-21 14:00:42 +08:00
8bc5282eb3
Fix datanode always retries to load stats even file corrupted ( #25012 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-20 16:40:42 +08:00
1ef8f0fceb
Remove cgo PayloadWriter ( #24892 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-06-14 18:04:38 +08:00
a9dccec03a
Add go payload writer ( #24656 ) ( #24762 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-09 13:52:39 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
ebd0279d3f
Check error by Error() and NoError() for better report message ( #24736 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-08 15:36:36 +08:00
d3af451d92
Upgrade golangci-lint ( #24707 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-06-07 19:34:36 +08:00
b71c967ed7
Fix NoSuchKey error caused by special stats log ( #24670 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-06-06 17:34:36 +08:00
c84bdcea49
merge stats log when segment flushing or compacting ( #23570 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-05-29 10:21:28 +08:00
73a181d226
Fix get vector it timeout and improve some string const usage ( #24141 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-16 17:41:22 +08:00
546080dcdd
Support to retrieve json ( #23563 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-04-21 11:46:32 +08:00
967a97b9bd
Support json & array types ( #23408 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
2023-04-20 11:32:31 +08:00
9288020da3
Fix bug for batch delete files on gcp of minio ( #23052 ) ( #23083 ) ( #23090 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-04-09 16:22:30 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
081572d31c
Refactor QueryNode ( #21625 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
2023-03-27 00:42:00 +08:00
9aa99aedbb
[Cherry-Pick] Remove arrow uasge in FieldData ( #22726 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-03-20 10:41:56 +08:00
024beddfe6
Make GCS OAuth token thread-safe ( #22714 )
...
Signed-off-by: huanghaoyuan <haoyuan.huang@zilliz.com>
2023-03-14 18:09:54 +08:00
732986aa04
Remove fmt.Print from internal package ( #22722 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-14 17:36:05 +08:00
32581e6452
Support aliyun oss as object storage with ak or IAM ( #22376 )
...
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2023-03-09 16:33:52 +08:00
90a5aa6265
Refine errors, re-define error codes ( #22501 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-09 15:47:52 +08:00
d126f06946
Decouple mq module from internal proto definition ( #22536 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-03-04 23:21:50 +08:00
ea4ab9e4a4
Replace ErrorList with multiErrors ( #22532 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-03 14:59:49 +08:00
697dedac7e
Use cockroachdb/errors to replace other error pkg ( #22390 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
5936723904
Refine Read method of MinioChunkManager ( #22235 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-02-21 16:22:26 +08:00
f2575e5fa8
Add unconvert & durationcheck linters and fix issues ( #22161 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-02-15 17:22:34 +08:00
6004867853
Add bucket name in error message ( #21887 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-02-10 17:44:31 +08:00
949d5d078f
Fix memory calculation in dataCodec ( #21800 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-28 11:09:52 +08:00
5986106037
Make paramtable init only once ( #21782 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-01-19 14:53:44 +08:00
264f08953d
Refactor Rocksmq's config ( #21304 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-26 19:11:30 +08:00
166e9f0da5
Refactor GrpcConfig ( #21142 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-16 15:59:23 +08:00
908023a06e
Add more test on file reader ( #21082 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-12-09 11:55:19 +08:00
eb7ef01b9a
Fix bug for ListWithPrefix from minio with recursive is false ( #21059 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-12-08 15:47:18 +08:00
89b810a4db
Refactor all params into ParamItem ( #20987 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
548e90ec68
Support an analogous Java Guava cache implementation ( #20831 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-12-05 20:29:18 +08:00
f745d7f489
Fix compaction target segment rowNum is always 0 ( #20937 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-12-01 20:33:17 +08:00
f76ea292d2
Change LRU to a generic type ( #20744 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-22 12:11:11 +08:00
bf2107ecf5
Fix a regression of local storage ( #20653 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-11-21 10:19:10 +08:00
633a749880
Recude IndexCodec Load Memory ( #20621 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-11-18 10:47:08 +08:00
c05b9ad539
Add event dispatcher for config ( #20393 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-17 18:59:09 +08:00
09ea38615e
Fix gcp oauth token not cached ( #20380 )
...
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-11-08 12:43:02 +08:00
766957e75a
Recuce memory consumption in MinioChunkManager ( #20284 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-11-03 12:19:35 +08:00
9fac1476f3
Add support for GCS(GoogleCloudStorage) with IAM ( #20164 )
...
go mod tidy
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-11-01 11:07:35 +08:00
2bfecf5b4e
Refine bloomfilter and memory usage ( #20168 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-31 17:41:34 +08:00
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
89c9cb3680
Fix scan may break GC limitation ( #19670 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-15 15:13:24 +08:00
1919353f02
Fix parse segment id error from delta log ( #19566 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-09-30 14:18:55 +08:00
838a633584
Add ctx parameter for ChunkManager methods ( #19546 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-29 16:18:56 +08:00
b30c9d4f60
Support Bulk Load in Milvus 2.2 ( #18982 )
...
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-09-26 18:06:54 +08:00
928a213e31
Fix Chunk Manager Path mis use ( #19141 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-09-23 14:40:51 +08:00
f24e16a66f
Handle multiple parquet RowGroup/Column ( #19283 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-21 09:12:50 +08:00
d7f38a803d
Separate some proto files ( #19218 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
b161aec95e
Support input empty string ( #19111 ) ( #19144 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-09-13 13:36:29 +08:00
4de1bfe5bc
Add cpp data codec ( #18538 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: zhagnlu lu.zhang@zilliz.com
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-09-09 22:12:34 +08:00
ce9ba0dd02
Remove debug log in minio chunk manager ( #19134 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-09 17:32:37 +08:00
db23a256c2
Make minioChunkManager ListObject level by level when recursive is true ( #19096 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-08 14:58:34 +08:00
68a6587374
Set insert&stats binlog timestamp range ( #19005 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-04 09:05:09 +08:00
c91bb0b015
Unify rootPath in configs and ChunkManager ( #18808 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-08-25 19:32:53 +08:00
70254c3ec5
Unified catalog interface for segment ( #18289 ) ( #18290 )
...
Signed-off-by: kejiang <ke.jiang@zilliz.com>
Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-08-20 10:24:51 +08:00
e40061b864
Update binlog event format ( #18347 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-08-11 14:06:38 +08:00
efa5dfaa7b
Fix Parse binlog path failure ( #18583 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-08-10 17:04:37 +08:00
6e7831470c
Revert settings of Pulsar address and refine config code ( #18494 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-08-02 21:26:33 +08:00
c9174d55ba
Refine merge operation during compacting phase ( #18399 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-07-28 14:52:31 +08:00
e9d777b336
Fix ParseSegmentIDBinlog panicks with bad input ( #18413 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-07-26 19:32:30 +08:00
d90313a1a8
Use chunkManager rather than minio.Client in datanode garbage collection ( #18092 )
...
Signed-off-by: xingzhao <xing.zhao@zilliz.com>
Co-authored-by: xingzhao <xing.zhao@zilliz.com>
2022-07-22 22:10:28 +08:00
6d82ef8c20
fix garbage collector err handling ( #18277 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-07-15 13:54:26 +08:00
0ad15a22c8
IndexCoord handle events correctly ( #17878 )
...
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
2022-07-07 14:44:21 +08:00
4ae1ca2cac
Fix arrow builder nullptr check in FinishPayloadWriter ( #17873 )
...
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-06-28 20:04:17 +08:00
16c3aedc15
refine complie configuration ( #17502 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-06-24 21:12:15 +08:00
37464a281d
Fix datasorter swap vector logic ( #17661 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-06-21 11:38:12 +08:00
04f836f516
Fix s3 delete file ( #17574 )
...
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-06-16 18:14:10 +08:00
70f8bea4b4
Avoid growing slice as deserializing binlogs ( #17421 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-06-08 11:46:06 +08:00
76eaa3fc50
Support access s3 through AWS IAM role ( #17292 )
...
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-06-02 19:42:03 +08:00
d28a2db46c
move arrow from storage to core ( #17061 )
...
Signed-off-by: Enwei Jiao <jiaoew2011@gmail.com>
2022-05-22 20:03:58 +08:00
5355153805
Minio Error is not Handled gracefully ( #17003 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-05-16 19:25:55 +08:00
72bbe40254
Make PayloadReader read column data in batch ( #16826 )
...
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-05-10 11:37:52 +08:00
2be46a01c0
Reduce the number of retries and add error log ( #16754 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-05-05 09:31:51 +08:00
5b2b917987
Benchmark go/cgo parquet payload readers ( #16736 )
...
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-04-29 15:59:47 +08:00
515d0369de
Support string type in segcore ( #16546 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
2022-04-29 13:35:49 +08:00
68b1b82faf
Remove DataKV interface ( #16692 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-04-28 21:03:47 +08:00
7af02fa531
Improve load performance, load binlogs concurrently per file, deserialize binlogs concurrently per field/segment ( #16514 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-04-25 15:57:47 +08:00
4ef2df8cb9
Fix an issue where query coord calls MinioChunkManager during init ( #16629 )
...
It causes Milvus not runnable with local storage.
issue: #15604
/kind enhancement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-25 11:11:46 +08:00
9769426e84
Use fmt.Errorf instead of string concat in local_chunk_manager.go ( #16442 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-04-19 18:39:39 +08:00
e76a8c5ec2
Update compile options and config for embedded Milvus ( #16472 )
...
/kind enhancement
issue: #15711
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-14 19:57:34 +08:00
4781db8a2a
Add datanode import ( #16414 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-04-12 22:19:34 +08:00
bb7a0766fe
Add dependency factory ( #16204 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-04-07 22:05:32 +08:00
99984b88e1
Support delete varChar value ( #16229 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-04-02 17:43:29 +08:00
ba37531456
Add support for loading multiple indexes ( #16138 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-03-30 21:11:28 +08:00
801eeffbcc
Replace cgo parquet reader to go parquet reader ( #16199 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-03-30 15:21:28 +08:00
b6b3c9863c
Fix incompatibility of stats log ( #16277 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-03-30 10:15:28 +08:00
478f6ca11e
Add cache for vector chunk ( #15912 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-03-26 22:05:26 +08:00
205c92e54b
Support insert string data ( #15993 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-03-25 14:27:25 +08:00
3cd28420f1
Support compile under windows ( #15786 )
...
This patch makes compile milvus under windows(MSYS), including:
- some cpp adaptation for compile under msys/gcc-10.3
- install toolchain scripts for setup from MinGW/MSYS `scripts/install_deps_msys.sh`
- adaptation for POSIX API use in golang
* using gofrs/flock instead of syscall.Flock
* using x/exp/mmap instead of syscall.Mmap
- introducing github actions for build milvus.exe under windows/MSYS
- rocksdb's patch for MSYS
- adaptation for compile knowhere under windows
- a windows package script for pack zip file, `scripts/package_windows.sh`
issue #7706
Signed-off-by: Ji Bin <matrixji@live.com>
2022-03-17 17:17:22 +08:00
d4225117ee
Enable Zstd compression for binlog ( #15840 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-03-14 16:00:01 +08:00
3121619758
Chunk manager support scalar data ( #16010 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-03-11 14:39:59 +08:00
f71651e294
Support column-based insert data in message stream ( #15802 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-03-04 15:09:56 +08:00
92c8e32ebd
Let MemoryKV.Load return error when key not exist ( #15814 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-02 18:51:55 +08:00
503724be19
Optimize CMakeLists.txt under internal/core ( #15770 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-01 10:31:55 +08:00
2e4391e6c5
Add lru cache ( #15733 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-02-28 21:43:55 +08:00
adca79facc
Fix centos build error ( #15760 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-25 18:49:54 +08:00
fcf0887d92
Add and implement chunk manager interface. ( #15541 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-02-24 23:53:53 +08:00
307a8ce535
Support compile and run on Mac ( #15491 )
...
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
2022-02-09 14:27:46 +08:00
d83f69bb21
Support windows compliation ( #15448 )
...
- remove dl dependency while compiling some lib under MSYS
- int types aligement in golang
- core/src/utils adaptation for compile under MSYS
issue: #7706
Signed-off-by: Ji Bin <matrixji@live.com>
2022-02-07 23:43:46 +08:00
4369e08f2a
Fix storage memory leak caused by runtime.SetFinalizer ( #15100 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-01-10 17:13:35 +08:00
998ebcce4e
Fix file not close when ReadAll returns error in local chunk manager ( #15068 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-01-10 11:53:34 +08:00
yihao.dai
cai.zhang
Xu Tong
congqixia
yah01
smellthemoon
Jiquan Long
MrPresent-Han
XuanYang-cn
Bingyi Sun
PowderLi
Enwei Jiao
zhenshan.cao
zhagnlu
SimFG
bjzhjing
xige-16
Xiaofan
yah01
aoiasd
jaime
huanghaoyuanhhy
shaoyue
groot
Ten Thousand Leaves
codeman
xing.zhao
Letian Jiang
Enwei Jiao
godchen
Ji Bin
Cai Yudong