281a4b0300
fix: [2.5] Ignore growing segment without start pos for seal policy ( #41131 )
...
Cherry-pick from master
pr: #41130
Related to #41129
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-04-07 23:42:27 +08:00
1a6761ac69
enhance: [cp25]Replace currRows with NumOfRows ( #40074 ) ( #40681 )
...
See also: #40068
pr: #40074
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-03-20 11:24:13 +08:00
d8a2c1a907
enhance: [2.5] Add channel seal policy based on blocking l0 ( #40505 ) ( #40535 )
...
Cherry-pick from 2.5
pr: #40505
Related to #40502
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-14 10:02:12 +08:00
709594f158
enhance: [2.5] Use v2 package name for pkg module ( #40117 )
...
Cherry-pick from master
pr: #39990
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-23 00:46:01 +08:00
96922c0a87
fix: [2.5] Use start pos ts instead for sealSegmentByLifetime policy ( #39994 )
...
Cherry-pick from master
pr: #39982
Related to #39981
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-19 17:48:55 +08:00
cc6b2ff150
fix: [cp25]A segment may never transfer from sealed to flushing ( #39996 )
...
See also: #39717
pr: #39993
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-02-19 16:50:59 +08:00
95809ca767
enhance: make new go package to manage proto ( #39128 )
...
issue: #39095
pr: #39114
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:53:01 +08:00
36bb87d9b6
fix: Fix large growing segment ( #37388 )
...
Consider the `sealProportion` factor during segment allocation.
issue: https://github.com/milvus-io/milvus/issues/37387
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-07 17:12:25 +08:00
f3d9d05a28
fix: use binlog counter to trigger flush but not stats log ( #37037 )
...
issue: #36804
Signed-off-by: chyezh <chyezh@outlook.com>
2024-10-23 15:07:29 +08:00
196a7986b3
enhance: Change the fixed value to a ratio for clustering segment size ( #35076 )
...
issue: #34495
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-01 22:04:14 +08:00
b22e549844
enhance: Rename config of sealing by growing segmetns size ( #34787 )
...
/kind improvement
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-19 20:27:41 +08:00
4939f82d4f
enhance: Seal by total growing segments size ( #34692 )
...
Seals the largest growing segment if the total size of growing segments
of each shard exceeds the size threshold(default 4GB). Introducing this
policy can help keep the size of growing segments within a suitable
level, alleviating the pressure on the delegator.
issue: https://github.com/milvus-io/milvus/issues/34554
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-17 21:45:41 +08:00
8b5754f7fe
enhance: Add segment seal proportion jitter ( #34636 )
...
See also #34574
Add jitter for segment seal proportion to avoid seal operation burst in
short period of time.
This PR also fix license header in paramtable pkg.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-15 14:47:39 +08:00
1a248f2668
enhance: Add param item for segmentFlushInterval ( #34629 )
...
See also #28817
Add paramitem for segment flush interval
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-12 18:59:35 +08:00
be92147723
enhance: Print reason when seal segments ( #34575 )
...
See also #34574
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-11 11:40:51 +08:00
7ce876a072
fix: Decoupling importing segment from flush process ( #30402 )
...
This pr decoups importing segment from flush process by:
1. Exclude the importing segment from the flush policy, this approch
avoids notifying the datanode to flush the importing segment, which may
not exist.
2. When RootCoord call Flush, DataCoord directly set the importing
segment state to `Flushed`.
issue: https://github.com/milvus-io/milvus/issues/30359
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-02-03 13:01:12 +08:00
1d4be0d257
Adjust datacoord for L0 Delta ( #28021 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-06 15:26:16 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
35cb0b5ba6
fix function name in comment ( #24879 )
...
Signed-off-by: cui fliter <imcusg@gmail.com>
2023-06-14 16:20:38 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
697dedac7e
Use cockroachdb/errors to replace other error pkg ( #22390 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
748a935a11
Modify segment seal policy with number of statslog files ( #21957 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-02-03 17:07:52 +08:00
2213ed7bed
Fix bulkload segment can not be flushed ( #21919 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-02-02 13:11:51 +08:00
d401608899
Add a segment seal policy by number of binlog files ( #21263 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-12-20 14:09:25 +08:00
89b810a4db
Refactor all params into ParamItem ( #20987 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
59f8e96522
Change compactor policy from loading all binlogs to loading serially ( #13716 ) ( #19280 )
...
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
2022-09-25 20:12:52 +08:00
5141e05c47
Add policy to seal long time idle segment ( #19222 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-09-20 20:54:50 +08:00
d7f38a803d
Separate some proto files ( #19218 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
891c202b73
Use GlobalParamTable for all components ( #13885 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-23 18:39:11 +08:00
0c39e0370c
Move segment statistics update to datanode tt loop ( #13233 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-12-15 10:53:16 +08:00
23c7691d59
[skip ci]Update license for dc policy ( #10590 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-25 19:48:23 +08:00
1cb2e2153f
[skip ci] Fix AllocationPolicy golint ( #8883 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-09-29 20:52:09 +08:00
49b6c3101d
Remove datacoord useless code ( #7673 )
...
Issue: #7624
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-09-09 19:08:03 +08:00
5b22697ec4
Add segment manager unit tests ( #7525 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-09-07 13:59:58 +08:00
be5c492e5a
Add datacoord unit tests ( #7197 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-08-20 17:50:12 +08:00
6e34f4c7f1
Add a seal policy which restrict the lifetime of a segment ( #7172 )
...
issue: #7164
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-08-20 15:42:12 +08:00
3cc210da26
Add flush interval ( #6996 )
...
Time tick message is sent about per 0.2 seconds. DataCoord will call
`FlushSegment` too frequently. We add a min interval for segment flush. A
segment will called `FlushSegment` only once between the interval.
issue: #6995
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-08-19 14:08:10 +08:00
fc74a0f578
Auto create new segments when allocating rows more than ( #6665 )
...
max number of rows per segment
If user insert too much rows in a request. Now we will return a failed
response. Maybe auto creating new segments to hold that much rows is a
better way.
issue: #6664
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-07-23 21:58:33 +08:00
83be910baa
Refactor segment manager ( #6441 )
...
issue: #6440
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-07-12 17:24:25 +08:00
f6761bbc33
Refactor datacoord segment status ( #6039 )
...
* Refactor datacoord segment status
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Remove lastExpireTime
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-24 14:20:10 +08:00
0978b93a21
Refactor data coordinator ( #5982 )
...
Rename variable name and make error msg more clear
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-22 18:24:08 +08:00
d2c8462dac
Rename params in data_coord.yaml ( #5968 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-22 16:18:21 +08:00
c863a193de
Rename filename from dataservice to datacoordinator ( #5946 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-22 10:42:07 +08:00
congqixia
XuanYang-cn
Zhen Ye
yihao.dai
cai.zhang
aoiasd
SimFG
cui fliter
jaime
Enwei Jiao
Xiaofan
zhagnlu
Jiquan Long
Cai Yudong
Bingyi Sun
sunby