See also #27675#30469
For a sync task, the segment could be compacted during sync task. In
previous implementation, this sync task will hold only the old segment
id as KeyLock, in which case compaction on compacted to segment may run
in parallel with delta sync of this sync task.
This PR introduces sync target segment verification logic. It shall
check target segment lock it's holding beforing actually syncing logic.
If this check failed, sync task shall return`errTargetSegementNotMatch`
error and make manager re-fetch the current target segment id.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #29223
This PR make `conc.Pool` resizable by adding `Resize` method for it.
Also make newly added datanode `MaxParallelSyncMgrTasks` config
refreshable
---------
Signed-off-by: Congqi.Xia <congqi.xia@zilliz.com>
See also #27675
Compacted segment info shall be removed after all buffer belongs to it
is sync-ed.
This PR add the cleanup function after triggerSyncTask logic:
- The buffer is stable and protected by mutex
- Cleanup fetches compacted & non-sync segment
- Remove segment info only there is no buffered maintained in manager
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
fix#28629
orignal unit test close channel before setting the segment id, so there
is a chance that test read segment id before setting it change unit test
behavior to wait future return now
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #27675
- Fix LevelZero segment cannot be flushed
- Add level option for syncTask
- Invoke `AddSegment` when new LevelZero segment is allocated
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #27675
This PR make previously merged refactory of datanode go online
- Use write node to replace insert/delete node
- Use write buffer manager to control all buffers
- Use sync manager to control sync tasks instead of flush manager
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
congqixia
yiwangdr