milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
19,479 Commits
150 Branches
122 Tags
583 MiB
Commit Graph

56 Commits (aba4993c6ce8d5e9b865dbddd81035d53affbf2a)

Author SHA1 Message Date
wei liu 4c8cc6ceff
fix: Avoid acquire index meta's lock for each segment () 
issue:   

during FilterIndexedSegment in GetRecoveryInfo, it try to acquire index
meta's read lock for every segment. when a collection has thousands of
segments, which may blocked for more than 10 seconds and even longer.
cause `AddSegmentIndex` may also triggered frequently, which try to get
the write lock.

This PR avoid acquire index meta's lock for each segment

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-04-01 15:49:13 +08:00
Bingyi Sun bdc70dfc6a
feat: Add global mmap enable configuration () 
https://github.com/milvus-io/milvus/issues/31279

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-03-18 15:17:10 +08:00
jaime 4b0c3dd377
enhance: index meta use independent rather than global meta lock () 
issue: https://github.com/milvus-io/milvus/issues/30837

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-03-04 16:56:59 +08:00
foxspy e1e87d572b
fix: compatibility for diskann cache param () 
patch search cache param from index configs when index meta could not
get the search cache size key
#issue: 

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-02-26 16:54:55 +08:00
XuanYang-cn fd19e419f9
fix: Use size bucket for compacted segment size metric () 
See also: 

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-01-26 10:53:02 +08:00
Patrick Weizhi Xu 0907d76253
enhance: pass partition key scalar info if enabled when build vector index () 
issue:  

Pass optional scalar IVF offsets to Cardinal

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-01-24 00:04:55 +08:00
xige-16 02673914a0
feat: Support multiple vector indexes in a collection () 
issue:  

/kind improvement
Signed-off-by: xige-16 <xi.ge@zilliz.com>

---------

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-12-29 11:44:45 +08:00
aoiasd 8a4cfb7d6a
enhance: add l0 metric and fix datacoord no need drop l0 segment when flush () 
relate: https://github.com/milvus-io/milvus/issues/27675

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-11-24 15:58:24 +08:00
XuanYang-cn 40d5c902b6
Enable getting multiple segments in plan result () 
Compaction plan result contained one segment for one plan. For l0
compaction would write to multiple segments, this PR expand the segments
number in plan results and refactor some names for readibility.

- Name refactory: - CompactionStateResult -> CompactionPlanResult -
CompactionResult -> CompactionSegment

See also: 

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-14 15:56:19 +08:00
aoiasd 1d4be0d257
Adjust datacoord for L0 Delta () 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-11-06 15:26:16 +08:00
yah01 6539a5ae2c
Refine DataCoord status () 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-26 17:15:27 +08:00
Xu Tong 9166011c4a
Add float16 vector () 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-09-08 10:03:16 +08:00
XuanYang-cn b2e7cbdf4b
Remove TimeTravel in compactor () 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-04 17:41:48 +08:00
congqixia 597a4d9227
Treat small segment without index as sealed () 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-02 19:50:23 +08:00
xige-16 33c2012675
Add more metrics () 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-26 17:52:44 +08:00
congqixia 41af0a98fa
Use go-api/v2 for milvus-proto () 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
congqixia 73a181d226
Fix get vector it timeout and improve some string const usage () 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-16 17:41:22 +08:00
congqixia 5aa9db0d38
Add collection level auto compaction enabled config () 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-10 17:45:20 +08:00
jaime c9d0c157ec
Move some modules from internal to public package () 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-04-06 19:14:32 +08:00
Enwei Jiao 697dedac7e
Use cockroachdb/errors to replace other error pkg () 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-02-26 11:31:49 +08:00
cai.zhang e5f408dceb
Merge IndexCoord and DataCoord () 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-04 19:37:36 +08:00
Enwei Jiao 89b810a4db
Refactor all params into ParamItem () 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-07 18:01:19 +08:00
Ten Thousand Leaves c373b44c2c
Verify and adjust segment # of rows before saving and passing SegmentInfo () 
issue: 

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-12-01 16:31:16 +08:00
Xiaofan 192151bc11
Refine Data Log () 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-10-25 19:31:30 +08:00
yah01 52cd40fbc4
Remove codes related to deprecated field () 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-10-19 18:29:26 +08:00
SimFG a55f739608
Separate public proto files () 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-10-16 20:49:27 +08:00
congqixia 2a2d32c4fc
Fix datacoord panics when collection info not found () 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-10-11 21:39:24 +08:00
jaime 31db01b6ae
Support TTL for collection level () 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-10-10 20:31:22 +08:00
yah01 ee9d634a43
Compaction requires the source segments indexed () 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-09-22 18:34:50 +08:00
yah01 71d4a32ce6
Fix deadlock () 
The IsParentDropped() acquires read lock of meta,
which leads to recursively acquire read lock

Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-09-22 15:48:50 +08:00
Jiquan Long 5141e05c47
Add policy to seal long time idle segment () 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-09-20 20:54:50 +08:00
SimFG d7f38a803d
Separate some proto files () 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-09-16 16:56:49 +08:00
yah01 11b352c3ae
Guarantee flushed segments are all indexed () 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-09-16 11:32:48 +08:00
Ten Thousand Leaves 0ebe407221
Make DataNode not panic if stale segment is not found during flush () 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-07-20 15:06:30 +08:00
Xiaofan 6d82ef8c20
fix garbage collector err handling () 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-07-15 13:54:26 +08:00
Xiaofan 54aeb077e9
Refine compaction selection poliy () 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-15 23:14:10 +08:00
cai.zhang bcf3b7426a
Add distributed lock for segment refgerence () 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-05-31 16:36:03 +08:00
Cai Yudong 6323907566
Use CommonConfig in GlobalParams for all components () 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-10 19:03:35 +08:00
Xieql 1c19aec7a8
[skip e2e] Improve annotation () 
Signed-off-by: Xieql <qianglong.xie@zilliz.com>
 2021-12-30 16:33:41 +08:00
zhenshan.cao 32a5126705
Add the limitation of the traceable interval to timeTravel () 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-12-29 21:29:45 +08:00
Jiquan Long 00b3fcb98b
Add log for flowgraph () 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-12-29 14:55:21 +08:00
Cai Yudong 891c202b73
Use GlobalParamTable for all components () 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-12-23 18:39:11 +08:00
edward.zeng 8ede9c70b2
[skip e2e] Refine datacoord util comment () 
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
 2021-12-21 19:07:15 +08:00
Bingyi Sun b8c5239784
Add compaction retention config () 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-16 11:47:10 +08:00
sunby 05681364c9
Fix some review comments in compaction. related pr () 
issue: 
Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2021-11-08 21:45:00 +08:00
sunby c26cd79532
Add compaction () 
issue: 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-11-05 22:25:00 +08:00
XuanYang-cn e5a88374bd
[skip ci]Update license for dc server () 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-25 19:46:28 +08:00
congqixia 1e11a11c27
[skip ci] Fix golint issue for datacoord/util.go () 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-15 20:40:53 +08:00
sunby 044b06421a
Use etcd to watch channel on datanode () 
issue: 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-14 15:44:34 +08:00
sunby 82dd762ef0
Change copyright () 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-22 19:59:59 +08:00