2c9bb4dfa3
feat: Support stats task to sort segment by PK ( #35054 )
...
issue: #33744
This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-02 14:19:03 +08:00
42f7800b5b
enhance: add bitmap offset cache to speed up retrieve raw data ( #35498 )
...
#35458
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-24 01:40:58 +08:00
c992a61a23
enhance: Separate allocator pkg in datacoord ( #35622 )
...
Related to #28861
Move allocator interface and implementation into separate package. Also
update some unittest logic.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-22 10:06:56 +08:00
731d45abbe
enhance: provide more general configuration to control mmap behavior ( #35359 )
...
- issue: #35273
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-21 00:22:54 +08:00
feb13cdf07
enhance: Skip pick worker when task doesn't need to execute actually ( #34348 )
...
issue: #34347
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-03 15:52:09 +08:00
27cc9f2630
enhance: Support analyze data ( #33651 )
...
issue: #30633
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
2024-06-06 17:37:51 +08:00
54797b4286
enhance: Refine frequent log in datacoord ( #33449 )
...
This PR changes:
- Frequent `ListIndexes` success log to debug level
- Aggregate collection missing log after collection dropped in
`meta.GetCollectionIndexFilesSize`
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-29 19:15:43 +08:00
ed39a38953
enhance: Reduce the frequency of logs describing indexing failures ( #33212 )
...
issue: #33001 #33102
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-05-21 19:27:39 +08:00
4382cf5283
enhance: Reduce the frequency of index ralted logs ( #33050 )
...
issue: #33001
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-05-15 14:37:34 +08:00
cedb33ceec
enhance: Improve datacoord segment filtering with collection ( #32831 )
...
See also #32165
This PR modify the `SelectSegments` interface to utilizing collection id
information when selecting segment with provided collection
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-08 21:37:29 +08:00
6af5275066
fix: Refine index param check for Create/Alter Index ( #32712 )
...
See also #32711
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-06 15:43:29 +08:00
bdc70dfc6a
feat: Add global mmap enable configuration ( #31267 )
...
https://github.com/milvus-io/milvus/issues/31279
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-18 15:17:10 +08:00
4871786a7b
enhance: When describing an index, fetch the index info in batches ( #31238 )
...
issue: #29313
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-15 16:37:09 +08:00
5c0bb40549
fix: merge index params when creating index ( #31127 )
...
issue: https://github.com/milvus-io/milvus/issues/31102
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-11 17:31:03 +08:00
425da78b38
fix: alter index request's index name can not be empty ( #31128 )
...
issue: https://github.com/milvus-io/milvus/issues/31138
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-11 13:05:02 +08:00
d81ba164c8
enhance: Add `ListIndexes` API from datacoord ( #31104 )
...
See also #31103
This PR add `listIndexes` API for datacoor server to list all indexes
for provided collection.
Comparing to the existing `DescribeIndex` API, the new one does NOT
check the segment index building progress to ease the burden when
invoking it
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-07 17:37:01 +08:00
4b0c3dd377
enhance: index meta use independent rather than global meta lock ( #30869 )
...
issue: https://github.com/milvus-io/milvus/issues/30837
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-03-04 16:56:59 +08:00
f6ff2588cd
enhance: Optimize DescribeIndex to reduce lock contention ( #30939 )
...
issue: #29313
issue: #30443
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-03 19:00:59 +08:00
816ed671aa
fix: alter_index should return error if index not found ( #30786 )
...
issue: https://github.com/milvus-io/milvus/issues/30932
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-02-29 15:55:01 +08:00
0907d76253
enhance: pass partition key scalar info if enabled when build vector index ( #29931 )
...
issue: #29892
Pass optional scalar IVF offsets to Cardinal
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-01-24 00:04:55 +08:00
1c8ce33eea
fix: report error if the altering index doesn't support mmap ( #29832 )
...
this also checks the param value
fix #29909
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2024-01-17 16:40:54 +08:00
02673914a0
feat: Support multiple vector indexes in a collection ( #27700 )
...
issue: #25639
/kind improvement
Signed-off-by: xige-16 <xi.ge@zilliz.com>
---------
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-12-29 11:44:45 +08:00
a0e1a1eb31
feat: support enable/disable mmap for index ( #29005 )
...
support enable/disable mmap for index, the user could alter the index's
mode by `AlterIndex` method
related: https://github.com/milvus-io/milvus/issues/21866
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-21 18:07:24 +08:00
1b4d2674b3
fix: Set the default index name to the name of the existing index ( #29275 )
...
issue: #29269
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-20 17:22:41 +08:00
5b45a138b1
disable auto balance when old node exists ( #28191 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-07 14:02:20 +08:00
be980fbc38
Refine state check ( #27541 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-11 21:01:35 +08:00
41495ed266
Improve the error message for getting all indexes of collection ( #27389 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-08 21:23:32 +08:00
370fdaf50d
Record engine version for segment index ( #27384 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-09-28 18:03:28 +08:00
dedb90f85f
Fix error message for creating scalar index ( #27382 )
...
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-09-27 10:33:27 +08:00
6539a5ae2c
Refine DataCoord status ( #27262 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-26 17:15:27 +08:00
5db4a0489e
dynamic index version control ( #27335 )
...
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-25 21:39:27 +08:00
370b6fde58
milvus support multi index engine ( #27178 )
...
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-22 09:59:26 +08:00
22d7fa4e1c
Use GetXXX instead of directly using dots to get value ( #27045 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-09-15 10:07:19 +08:00
0901b76732
Avoid the panic when the status of rpc response is nil ( #26910 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-07 19:23:15 +08:00
d7cd1f2a6d
Fix bug for get index state with compaction segment ( #26822 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-09-04 17:29:48 +08:00
3349db4aa7
Refine errors to remove changes breaking design ( #26521 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-04 09:57:09 +08:00
6e1f284485
Refine index log ( #26191 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-08-11 11:14:39 +08:00
eaf53c7b39
Refine describe index that doesn't wait compacted segment indexed ( #25899 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-07-28 11:15:02 +08:00
760a2d9aa7
Support AllocTimestamp api for Milvus ( #25784 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-07-25 10:05:00 +08:00
66fdc71479
Refactor logs in DataCoord & DataNode ( #25574 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-07-14 15:56:31 +08:00
04082b3de2
Migrate the ability to upload and download binlog to cpp ( #22984 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-25 14:38:44 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
93ea9c4925
Support return pending index rows when describe index ( #24588 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-06-01 18:14:32 +08:00
899702f13c
Implement GetIndexStatistics interface ( #23603 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-05-06 10:34:39 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
b8b7d1f47e
CreateIndex should wait all segments which has data insert before CreateIndex finish ( #22561 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-03-20 10:21:56 +08:00
319f1773af
Fix segment not found if forward delete to empty segment ( #22528 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-03 14:13:49 +08:00
eefa29a5e8
Fix bug for get index state ( #22396 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-02-28 10:29:50 +08:00
f02ccfb422
Set GetIndexBuildProgress is compatible with SDK version 2.1 ( #21622 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-11 16:59:42 +08:00
e127cf7b99
Reset indexpb for upgrade ( #21620 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-11 14:35:40 +08:00
cai.zhang
zhagnlu
congqixia
SimFG
Bingyi Sun
jaime
Patrick Weizhi Xu
yah01
xige-16
wei liu
yah01
Jiquan Long
foxspy
Enwei Jiao
wayblink
aoiasd