1b0ea49d25
fix: incorrect memory size and delta count in SegmentInfo ( #35369 )
...
issue: #35368
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-13 17:20:19 +08:00
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 ( #34394 )
...
issue: #34252
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
4939f82d4f
enhance: Seal by total growing segments size ( #34692 )
...
Seals the largest growing segment if the total size of growing segments
of each shard exceeds the size threshold(default 4GB). Introducing this
policy can help keep the size of growing segments within a suitable
level, alleviating the pressure on the delegator.
issue: https://github.com/milvus-io/milvus/issues/34554
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-17 21:45:41 +08:00
67324eb809
enhance: Add l0 segment entry num quota ( #34733 )
...
See also #34670
This PR add quota configuration for l0 segment entry number per
collection. If l0 compaction cannot keep up the insertion/upsertion
rate, this feature could back press the related rate.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-17 17:35:41 +08:00
1a248f2668
enhance: Add param item for segmentFlushInterval ( #34629 )
...
See also #28817
Add paramitem for segment flush interval
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-12 18:59:35 +08:00
a1232fafda
feat: Major compaction ( #33620 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
c61fb1eff5
enhance: do check when add not empty logpath ( #33640 )
...
meta only store logid
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-06-07 10:19:51 +08:00
180d754158
fix: speed up segment lookup via channel name in datacoord ( #33530 )
...
issue: #33342
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-06-03 14:47:47 +08:00
3d29907b6e
enhance: decrease cpu overhead during filter segments on datacoord ( #33130 )
...
issue: #33129
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-05-28 19:17:43 +08:00
e895cfed84
fix: reduce redundant map operations in datacoord ( #33343 )
...
More refactories will be added.
issue: #33342
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-05-24 12:47:40 +08:00
6ea7633bd5
enhance: Add memory size for binlog ( #33025 )
...
issue: #33005
1. add `MemorySize` field for insert binlog.
2. `LogSize` means the file size in the storage object.
3. `MemorySize` means the size of the data in the memory.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2024-05-15 12:59:34 +08:00
cedb33ceec
enhance: Improve datacoord segment filtering with collection ( #32831 )
...
See also #32165
This PR modify the `SelectSegments` interface to utilizing collection id
information when selecting segment with provided collection
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-08 21:37:29 +08:00
2586c2f1b3
enhance: use WalkWithPrefix api for oss, enable piplined file gc ( #31740 )
...
issue: #19095,#29655,#31718
- Change `ListWithPrefix` to `WalkWithPrefix` of OOS into a pipeline
mode.
- File garbage collection is performed in other goroutine.
- Segment Index Recycle clean index file too.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-25 20:41:27 +08:00
037de8e4d3
enhance: speed up minor functions calls in datacoord ( #32389 )
...
Related to https://github.com/milvus-io/milvus/issues/32165
1. nodeid based channel store access should use map access instead of
iteration.
2. The join-ish functions calls are slow when # collections/segments
increases (e.g. 10k).
e.g.
getNumRowsOfCollectionUnsafe is O(num_segments); GetAllCollectionNumRows
is of O(num_collections*num_segments).
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-04-20 07:55:21 +08:00
0fe5e90e8b
enhance: Remove import v1 ( #31403 )
...
Remove all code and logic related to import v1.
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-22 15:29:09 +08:00
8f7019468f
fix: starve lock caused by slow GetCompactionTo method when too much segments ( #30963 )
...
issue: #30823
Signed-off-by: chyezh <chyezh@outlook.com>
2024-03-05 10:04:59 +08:00
4b0c3dd377
enhance: index meta use independent rather than global meta lock ( #30869 )
...
issue: https://github.com/milvus-io/milvus/issues/30837
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-03-04 16:56:59 +08:00
e52ce370b6
enhance:don't store logPath in meta to reduce memory ( #28873 )
...
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-18 22:06:31 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
6375236533
Make SegmentInfo.size atomic ( #24457 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-29 11:31:28 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
d126f06946
Decouple mq module from internal proto definition ( #22536 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-03-04 23:21:50 +08:00
78fd1987eb
Fix data race caused by no clone while updating index ( #21772 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-01-17 18:11:42 +08:00
204db183f3
Prevent panicking when index segment info not exists ( #21664 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-01-12 21:39:40 +08:00
e5f408dceb
Merge IndexCoord and DataCoord ( #21267 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-04 19:37:36 +08:00
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
b30c9d4f60
Support Bulk Load in Milvus 2.2 ( #18982 )
...
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-09-26 18:06:54 +08:00
5141e05c47
Add policy to seal long time idle segment ( #19222 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-09-20 20:54:50 +08:00
d7f38a803d
Separate some proto files ( #19218 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
fc42ee5bb3
Fix # of rows of recovering segment ( #18736 )
...
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-08-19 19:50:50 +08:00
e5fe46124b
Fix isCompacting flag lost after segment clone ( #18248 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-07-13 10:08:28 +08:00
54aeb077e9
Refine compaction selection poliy ( #17486 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-06-15 23:14:10 +08:00
8fe4af3584
[skip e2e]Sort imports ( #14809 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-01-05 14:06:48 +08:00
ef2a62387b
[skip e2e]Sort imports ( #14329 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2021-12-27 18:54:37 +08:00
c1cb69ed7e
[skip ci]Fix segment info typo ( #13643 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2021-12-21 09:19:53 +08:00
1dcd06cc78
Add log file info for all kind of segment binlogs ( #13606 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-19 20:00:42 +08:00
6b74aed32a
[skip e2e] Refine datacoord segment info comment ( #13602 )
...
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
2021-12-17 18:54:26 +08:00
c26cd79532
Add compaction ( #9906 )
...
issue: #9904
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-11-05 22:25:00 +08:00
548634ff05
[skip ci]Update license for dc segmentinfo ( #10588 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-25 19:44:37 +08:00
7ce7cb7a5e
Add a timed flush trigger mechanism ( #10197 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-20 15:02:36 +08:00
32a029fbed
[skip ci]Fix golint error in datacoord ( #10135 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-10-18 21:32:49 +08:00
044b06421a
Use etcd to watch channel on datanode ( #7115 )
...
issue: #6578
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-10-14 15:44:34 +08:00
aba9aa450d
[skip ci]Add comment for SetFlushTime ( #9668 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-10-11 21:06:37 +08:00
6fc64b836a
[skip ci]Add comment for SetBinlogs ( #9667 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-10-11 21:04:42 +08:00
1e0c3cbcb9
[skip ci]Add comment for SetCurrentRows ( #9526 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-10-09 00:16:56 +08:00
7834649dbc
[skip ci]Add comment for AddAllocation ( #9525 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-10-09 00:15:13 +08:00
292aeeac63
[skip ci]Add comment for SetAllocations ( #9524 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-10-09 00:13:21 +08:00
d828a63c94
[skip ci] Add comment for SegmentsInfo AddBinglogPaths ( #9386 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-08 17:31:50 +08:00
21655d54be
[skip ci]Add comment for AddSegmentBinlogs ( #9389 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-10-07 18:32:50 +08:00
jaime
wei liu
yihao.dai
congqixia
wayblink
smellthemoon
yiwangdr
cai.zhang
chyezh
SimFG
yah01
Ten Thousand Leaves
Jiquan Long
Xiaofan
godchen
edward.zeng
sunby
XuanYang-cn
zhenshan.cao