a4439cc911
enhance: Implement flusher in streamingNode ( #34942 )
...
- Implement flusher to:
- Manage the pipelines (creation, deletion, etc.)
- Manage the segment write buffer
- Manage sync operation (including receive flushMsg and execute flush)
- Add a new `GetChannelRecoveryInfo` RPC in DataCoord.
- Reorganize packages: `flushcommon` and `datanode`.
issue: https://github.com/milvus-io/milvus/issues/33285
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-02 18:30:23 +08:00
95462668ca
enhance: unify time in clustering compaction task to unix ( #35167 )
...
#34495
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-02 10:30:19 +08:00
aa247f192d
enhance: remove unused code for StorageV2 ( #35132 )
...
issue: https://github.com/milvus-io/milvus/issues/34168
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-08-01 12:08:13 +08:00
9412002d7d
fix: Fix data race for clustering buffer writer ( #35145 )
...
issue: #34495
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-01 11:20:13 +08:00
de8a266d8a
enhance: Enable linux code checker ( #35084 )
...
See also #34483
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 15:53:51 +08:00
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 ( #34394 )
...
issue: #34252
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
2372452fac
enhance: Optimized the GC logic to ensure that memory is released in time ( #34949 )
...
issue: #34703
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-28 23:53:47 +08:00
ea13b97b6a
fix: Fix bug for block clustering compaction ( #35019 )
...
issue: #34703
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-26 11:33:58 +08:00
260a6e2ba9
enhance: Send flush signal when the water level reaches the high watermark ( #34907 )
...
issue: #30633
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:29:42 +08:00
4c45bc412f
enhance: Add integration test for clustering compaction ( #34881 )
...
issue: #34792
---------
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:13:43 +08:00
643b9d521c
fix: Fix the issue of concurrent packing of the same segment ( #34840 )
...
issue: #34703
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:11:49 +08:00
8aab6cbfac
enhance: Organize the common modules of streamingNode and dataNode ( #34773 )
...
1. Move the common modules of streamingNode and dataNode to flushcommon
2. Add new GetVChannels interface for rootcoord
issue: https://github.com/milvus-io/milvus/issues/33285
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-22 11:33:51 +08:00
d294fdd060
enhance: add ut for clustering_compactor ( #34852 )
...
issue: #34792
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-21 19:29:42 +08:00
16c002b51b
enhance: unified channel name generation and conversion ( #34805 )
...
/kind improvement
- issue: #34804
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-20 23:11:39 +08:00
c79d1af390
enhance: Add compaction task slot usage logic ( #34581 )
...
#34544
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-18 10:27:41 +08:00
0c0ca4cf0e
fix: Fix bug where binlogs already flushed with new segment during pack ( #34762 )
...
issue: #34703
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-18 09:15:48 +08:00
ca758c36cc
enhance: Pre-allocate ids for compaction ( #34187 )
...
This PR removes the dependency of compaction on the ID allocator by
pre-allocating the logID and segmentID.
issue: https://github.com/milvus-io/milvus/issues/33957
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-17 13:23:42 +08:00
17c96e1bde
fix: Fix the bug that caused small segment flush frequently ( #34725 )
...
issue: #34703
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-17 12:15:40 +08:00
fbc10ac695
fix:Reset flushed row num after pack segment for clustering compaction ( #34702 )
...
issue: #34703
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-16 15:37:38 +08:00
5016038781
enhance: release the record in delete codec and add some log for compaction ( #34454 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-09 15:40:17 +08:00
21fc5f5d46
enhance: Remove datanode reporting TT based on MQ implementation ( #34421 )
...
issue: #34420
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-05 15:48:09 +08:00
8a2be8a457
fix: DataNode might OOM by estimating based on MemorySize ( #34201 )
...
See also: #34136
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-04 10:16:17 +08:00
8165961223
enhance: Optimeize clustering compaction ( #34313 )
...
issue: #30633
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-03 15:14:08 +08:00
d6afb31b94
enhance: make subfunctions of datanode component modular ( #33992 )
...
issue: #33994
also remove deprecated channel manager based on the etcd implementation
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-01 14:46:07 +08:00
e5d691d854
Use new stream segment reader in clustering compaction ( #34232 )
...
#32939
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-30 20:26:07 +08:00
fbe3231b1f
fix: fix error ignore in compactor ( #34169 )
...
#34170
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-26 10:24:03 +08:00
380d3f4469
fix: Fix memory buffer error & some renaming ( #33850 )
...
#30633
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-21 17:30:01 +08:00
04edb07d82
enhance: Add deltaRowCount in l0 compaction ( #33997 )
...
See also: #33998
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-20 17:46:01 +08:00
31ef0a1fe8
enhance: Add trace for bf cost in l0 compactor ( #33860 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-20 10:10:05 +08:00
5cb0760187
fix: Small fixs of major compaction ( #33929 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-18 16:53:58 +08:00
1a9ab52f66
enhance: Ensure the idempotency of compaction task ( #33872 )
...
/kind enhancement
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-16 22:09:57 +08:00
86a36b105a
enhance: Tidy compaction executor ( #33778 )
...
Move compaction executor to compaction pacakge.
issue: https://github.com/milvus-io/milvus/issues/32451
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-14 14:34:01 +08:00
4987067375
enhance: Execute bloom filter apply in parallel to speed up segment predict ( #33792 )
...
issue: #33610
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-14 11:37:56 +08:00
ab93d9c23d
enhance: Use BatchPkExist to reduce bloom filter func call cost ( #33611 )
...
issue:#33610
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-13 17:57:56 +08:00
a1232fafda
feat: Major compaction ( #33620 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
4dd0c54ca0
fix: Fix l0 compactor may cause DN from OOM ( #33554 )
...
See also: #33547
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-06 14:33:52 +08:00
77637180fa
enhance: Periodically synchronize segments to datanode watcher ( #33420 )
...
issue: #32809
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-05-30 13:37:44 +08:00
7730b910b9
enhance: Decouple compaction from shard ( #33138 )
...
Decouple compaction from shard, remove dependencies on shards (e.g.
SyncSegments, injection).
issue: https://github.com/milvus-io/milvus/issues/32809
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-24 09:07:41 +08:00
22bddde5ff
enhance: Tidy compactor and remove dup codes ( #32198 )
...
See also: #32451
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-23 09:53:40 +08:00
yihao.dai
wayblink
zhenshan.cao
cai.zhang
congqixia
wei liu
SimFG
jaime
XuanYang-cn