a0a6510db9
fix: [2.5] Only mark segment compacting for sort stats task ( #42517 )
...
issue: #42506
master pr: #42516
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-06-04 21:40:32 +08:00
1a6761ac69
enhance: [cp25]Replace currRows with NumOfRows ( #40074 ) ( #40681 )
...
See also: #40068
pr: #40074
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-03-20 11:24:13 +08:00
709594f158
enhance: [2.5] Use v2 package name for pkg module ( #40117 )
...
Cherry-pick from master
pr: #39990
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-23 00:46:01 +08:00
e46c8ba7fb
fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats ( #39490 )
...
issue: #39333
master pr: #39489
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2025-01-22 10:27:05 +08:00
2e4a1052aa
enhance: [2.5] Reduce mutex contention in datacoord meta ( #38904 )
...
1. Using secondary index to avoid retrieving all segments at
GetSegmentsChanPart.
2. Perform batch SetAllocations to reduce the number of times the meta
lock is acquired.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38219
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 00:55:00 +08:00
95809ca767
enhance: make new go package to manage proto ( #39128 )
...
issue: #39095
pr: #39114
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:53:01 +08:00
18a64782dc
enhance: Remove unnecessary clone in SetState ( #37697 )
...
issue: https://github.com/milvus-io/milvus/issues/37637
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-15 15:06:30 +08:00
5e6c3df253
fix: l0RowCount metrics value always empty ( #37306 )
...
See also: #36953
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-11 17:00:27 +08:00
4e0ea39235
fix: Remove neighbors if compactTo is unindexed ( #36503 )
...
See also: #36360
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-08 14:15:19 +08:00
d2c774fb6d
fix: Return all compactTo segments after support split ( #36361 )
...
Related to #36360
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-20 14:11:11 +08:00
1b0ea49d25
fix: incorrect memory size and delta count in SegmentInfo ( #35369 )
...
issue: #35368
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-13 17:20:19 +08:00
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 ( #34394 )
...
issue: #34252
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
4939f82d4f
enhance: Seal by total growing segments size ( #34692 )
...
Seals the largest growing segment if the total size of growing segments
of each shard exceeds the size threshold(default 4GB). Introducing this
policy can help keep the size of growing segments within a suitable
level, alleviating the pressure on the delegator.
issue: https://github.com/milvus-io/milvus/issues/34554
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-17 21:45:41 +08:00
67324eb809
enhance: Add l0 segment entry num quota ( #34733 )
...
See also #34670
This PR add quota configuration for l0 segment entry number per
collection. If l0 compaction cannot keep up the insertion/upsertion
rate, this feature could back press the related rate.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-17 17:35:41 +08:00
1a248f2668
enhance: Add param item for segmentFlushInterval ( #34629 )
...
See also #28817
Add paramitem for segment flush interval
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-12 18:59:35 +08:00
a1232fafda
feat: Major compaction ( #33620 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
c61fb1eff5
enhance: do check when add not empty logpath ( #33640 )
...
meta only store logid
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-06-07 10:19:51 +08:00
180d754158
fix: speed up segment lookup via channel name in datacoord ( #33530 )
...
issue: #33342
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-06-03 14:47:47 +08:00
3d29907b6e
enhance: decrease cpu overhead during filter segments on datacoord ( #33130 )
...
issue: #33129
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-05-28 19:17:43 +08:00
e895cfed84
fix: reduce redundant map operations in datacoord ( #33343 )
...
More refactories will be added.
issue: #33342
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-05-24 12:47:40 +08:00
6ea7633bd5
enhance: Add memory size for binlog ( #33025 )
...
issue: #33005
1. add `MemorySize` field for insert binlog.
2. `LogSize` means the file size in the storage object.
3. `MemorySize` means the size of the data in the memory.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2024-05-15 12:59:34 +08:00
cedb33ceec
enhance: Improve datacoord segment filtering with collection ( #32831 )
...
See also #32165
This PR modify the `SelectSegments` interface to utilizing collection id
information when selecting segment with provided collection
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-08 21:37:29 +08:00
2586c2f1b3
enhance: use WalkWithPrefix api for oss, enable piplined file gc ( #31740 )
...
issue: #19095,#29655,#31718
- Change `ListWithPrefix` to `WalkWithPrefix` of OOS into a pipeline
mode.
- File garbage collection is performed in other goroutine.
- Segment Index Recycle clean index file too.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-25 20:41:27 +08:00
037de8e4d3
enhance: speed up minor functions calls in datacoord ( #32389 )
...
Related to https://github.com/milvus-io/milvus/issues/32165
1. nodeid based channel store access should use map access instead of
iteration.
2. The join-ish functions calls are slow when # collections/segments
increases (e.g. 10k).
e.g.
getNumRowsOfCollectionUnsafe is O(num_segments); GetAllCollectionNumRows
is of O(num_collections*num_segments).
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-04-20 07:55:21 +08:00
0fe5e90e8b
enhance: Remove import v1 ( #31403 )
...
Remove all code and logic related to import v1.
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-22 15:29:09 +08:00
8f7019468f
fix: starve lock caused by slow GetCompactionTo method when too much segments ( #30963 )
...
issue: #30823
Signed-off-by: chyezh <chyezh@outlook.com>
2024-03-05 10:04:59 +08:00
4b0c3dd377
enhance: index meta use independent rather than global meta lock ( #30869 )
...
issue: https://github.com/milvus-io/milvus/issues/30837
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-03-04 16:56:59 +08:00
e52ce370b6
enhance:don't store logPath in meta to reduce memory ( #28873 )
...
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-18 22:06:31 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
6375236533
Make SegmentInfo.size atomic ( #24457 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-29 11:31:28 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
d126f06946
Decouple mq module from internal proto definition ( #22536 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-03-04 23:21:50 +08:00
78fd1987eb
Fix data race caused by no clone while updating index ( #21772 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-01-17 18:11:42 +08:00
204db183f3
Prevent panicking when index segment info not exists ( #21664 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-01-12 21:39:40 +08:00
e5f408dceb
Merge IndexCoord and DataCoord ( #21267 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-04 19:37:36 +08:00
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
b30c9d4f60
Support Bulk Load in Milvus 2.2 ( #18982 )
...
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-09-26 18:06:54 +08:00
5141e05c47
Add policy to seal long time idle segment ( #19222 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-09-20 20:54:50 +08:00
d7f38a803d
Separate some proto files ( #19218 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
fc42ee5bb3
Fix # of rows of recovering segment ( #18736 )
...
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-08-19 19:50:50 +08:00
e5fe46124b
Fix isCompacting flag lost after segment clone ( #18248 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-07-13 10:08:28 +08:00
54aeb077e9
Refine compaction selection poliy ( #17486 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-06-15 23:14:10 +08:00
8fe4af3584
[skip e2e]Sort imports ( #14809 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-01-05 14:06:48 +08:00
ef2a62387b
[skip e2e]Sort imports ( #14329 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2021-12-27 18:54:37 +08:00
c1cb69ed7e
[skip ci]Fix segment info typo ( #13643 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2021-12-21 09:19:53 +08:00
1dcd06cc78
Add log file info for all kind of segment binlogs ( #13606 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-19 20:00:42 +08:00
6b74aed32a
[skip e2e] Refine datacoord segment info comment ( #13602 )
...
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
2021-12-17 18:54:26 +08:00
c26cd79532
Add compaction ( #9906 )
...
issue: #9904
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-11-05 22:25:00 +08:00
548634ff05
[skip ci]Update license for dc segmentinfo ( #10588 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-25 19:44:37 +08:00
cai.zhang
XuanYang-cn
congqixia
yihao.dai
Zhen Ye
jaime
wei liu
wayblink
smellthemoon
yiwangdr
SimFG
yah01
Ten Thousand Leaves
Jiquan Long
Xiaofan
godchen
edward.zeng
sunby