milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
milvus/internal/datacoord
History
wei liu 75463725b3
fix: skip loading non-existent L0 segments to prevent load blocking (#43576) 
issue: #43557
In 2.5 branch, L0 segments must be loaded before other segments. If an
L0 segment has been garbage collected but is still in the target list,
the load operation would keep failing, preventing other segments from
being loaded.

This patch adds a segment existence check for L0 segments in
getSealedSegmentDiff. Only L0 segments that actually exist will be
included in the load list.

Changes:
- Add checkSegmentExist function parameter to SegmentChecker constructor
- Filter L0 segments by existence check in getSealedSegmentDiff
- Add unit tests using mockey to verify the fix behavior

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 		2025-07-31 14:33:38 +08:00
..
allocator fix: [2.5] mockery too unavailable after upgrade golang version (#41522) 2025-04-25 14:40:40 +08:00
broker fix: [2.5] mockery too unavailable after upgrade golang version (#41522) 2025-04-25 14:40:40 +08:00
session fix: [2.5] mockery too unavailable after upgrade golang version (#41522) 2025-04-25 14:40:40 +08:00
.mockery.yaml enhance: [2.5] Support detailed manual compaction criterion (#40892) (#40924) 2025-03-27 10:36:22 +08:00
OWNERS [skip ci]Update OWNERS files (#11898) 2021-11-16 15:41:11 +08:00
README.md [skip ci]Change etcd to lowercase (#9983) 2021-10-15 18:58:37 +08:00
analyze_meta.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
analyze_meta_test.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
build_index_policy.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
channel.go enhance: [2.5] support run analyzer by loaded collection field (#42119) 2025-05-29 10:26:30 +08:00
channel_manager.go fix: ChannelManager double assignment (#41877) 2025-05-16 14:58:23 +08:00
channel_manager_factory.go fix: drop collection failed if enable streaming service (#37444) 2024-11-07 10:26:26 +08:00
channel_manager_test.go fix: ChannelManager double assignment (#41877) 2025-05-16 14:58:23 +08:00
channel_store.go fix: ChannelManager double assignment (#41877) 2025-05-16 14:58:23 +08:00
channel_store_test.go fix: ChannelManager double assignment (#41877) 2025-05-16 14:58:23 +08:00
cluster.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
cluster_test.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
compaction.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
compaction_l0_view.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
compaction_l0_view_test.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
compaction_policy_clustering.go fix: [2.5] Only mark segment compacting for sort stats task (#42517) 2025-06-04 21:40:32 +08:00
compaction_policy_clustering_test.go fix: [cp25]Use diskSegmentMaxSize for coll with sparse and dense vectors (#43195) 2025-07-18 11:16:52 +08:00
compaction_policy_l0.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
compaction_policy_l0_test.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
compaction_policy_single.go fix: [2.5]datacoord stop get stuck After upgrading from 2.5 to 2.6 (#42669) 2025-06-16 19:56:38 +08:00
compaction_policy_single_test.go enhance: [2.5] Optimize datacoord meta mutex (#40753) 2025-03-25 13:50:21 +08:00
compaction_queue.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
compaction_queue_test.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
compaction_task.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
compaction_task_clustering.go fix: [2.5] Fix regeneratePartitionStats failed after restore clusteringCompactionTask (#43206) 2025-07-10 15:30:48 +08:00
compaction_task_clustering_test.go feat: cherry pick json path index (#40313) 2025-03-10 22:14:05 +08:00
compaction_task_l0.go enhance: [2.5] Reduce the allocation of ID (#41005) 2025-03-31 17:58:21 +08:00
compaction_task_l0_test.go fix: [2.5] Fix delete data loss due to duplicate binlogID (#40976) 2025-03-28 14:34:21 +08:00
compaction_task_meta.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
compaction_task_meta_test.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
compaction_task_mix.go enhance: [2.5] Reduce the allocation of ID (#41005) 2025-03-31 17:58:21 +08:00
compaction_task_mix_test.go fix: [2.5] Fix delete data loss due to duplicate binlogID (#40976) 2025-03-28 14:34:21 +08:00
compaction_test.go enhance: [cp25]Replace currRows with NumOfRows (#40074) (#40681) 2025-03-20 11:24:13 +08:00
compaction_trigger.go fix: [cp25]Use diskSegmentMaxSize for coll with sparse and dense vectors (#43195) 2025-07-18 11:16:52 +08:00
compaction_trigger_test.go enhance: refine expiring compaction(#41336) (#42052) 2025-05-25 16:44:27 +08:00
compaction_trigger_v2.go fix: [cp25]Use diskSegmentMaxSize for coll with sparse and dense vectors (#43195) 2025-07-18 11:16:52 +08:00
compaction_trigger_v2_test.go fix: [cp25]Use diskSegmentMaxSize for coll with sparse and dense vectors (#43195) 2025-07-18 11:16:52 +08:00
compaction_util.go fix: [2.5] Fix delete data loss due to duplicate binlogID (#40976) 2025-03-28 14:34:21 +08:00
compaction_view.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
const.go enhance: pass partition key scalar info if enabled when build vector index (#29931) 2024-01-24 00:04:55 +08:00
errors.go Refine DataCoord status (#27262) 2023-09-26 17:15:27 +08:00
errors_test.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
garbage_collector.go fix: [2.5]datacoord stop get stuck After upgrading from 2.5 to 2.6 (#42669) 2025-06-16 19:56:38 +08:00
garbage_collector_test.go fix: [2.5] Use locking to ensure the atomicity of dropping segment indexes (#42076) 2025-05-26 15:40:28 +08:00
go_channel_singleton.go fix: Fix improper use of offset in HybridSearch (#36244) 2024-09-13 22:05:15 +08:00
handler.go fix: [2.5]datacoord stop get stuck After upgrading from 2.5 to 2.6 (#42669) 2025-06-16 19:56:38 +08:00
handler_test.go feat: cherry pick json path index (#40313) 2025-03-10 22:14:05 +08:00
import_checker.go fix: [cp25]Use diskSegmentMaxSize for coll with sparse and dense vectors (#43195) 2025-07-18 11:16:52 +08:00
import_checker_test.go enhance: [2.5] Enhance import integration tests and logs (#42696) 2025-06-16 20:06:38 +08:00
import_job.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
import_meta.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
import_meta_test.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
import_scheduler.go enhance: [2.5] Enhance import context (#42051) 2025-05-24 18:40:26 +08:00
import_scheduler_test.go enhance: [2.5] Enhance import context (#42051) 2025-05-24 18:40:26 +08:00
import_task.go fix: [2.5] Fix import slot assignment (#41982) 2025-05-23 01:36:30 +08:00
import_util.go fix: [cp25]Use diskSegmentMaxSize for coll with sparse and dense vectors (#43195) 2025-07-18 11:16:52 +08:00
import_util_test.go fix: [cp25]Use diskSegmentMaxSize for coll with sparse and dense vectors (#43195) 2025-07-18 11:16:52 +08:00
index_engine_version_manager.go fix: solve incompitable problem for none-encoding index (#41346) 2025-04-16 21:04:52 +08:00
index_engine_version_manager_test.go fix: solve incompitable problem for none-encoding index (#41346) 2025-04-16 21:04:52 +08:00
index_meta.go fix: [cp25]Use diskSegmentMaxSize for coll with sparse and dense vectors (#43195) 2025-07-18 11:16:52 +08:00
index_meta_test.go fix: [2.5] Use locking to ensure the atomicity of dropping segment indexes (#42076) 2025-05-26 15:40:28 +08:00
index_service.go fix: [2.5] Don't create index for unsorted importing segment when enable stats (#41865) 2025-05-16 10:18:35 +08:00
index_service_test.go enhance: [2.5] Output index version information in the DescribeIndex interface (#41841) 2025-05-15 14:34:23 +08:00
job_manager.go fix: [2.5] Fix index blocked by failed sort stats (#43061) 2025-07-02 22:32:45 +08:00
job_manager_test.go fix: [2.5] Ensure task execution order by using a priority queue (#43272) 2025-07-23 10:16:54 +08:00
knapsack.go fix: [2.5] Pass Knapsnak ptr to avoid compact multiple times (#40401) 2025-03-06 14:16:03 +08:00
knapsack_test.go enhance: try compact small segments first if they may compose a full segment (#37709) 2024-12-02 16:12:38 +08:00
meta.go fix: [2.5] Only mark segment compacting for sort stats task (#42517) 2025-06-04 21:40:32 +08:00
meta_test.go feat: refine drop parition through the new interface notifydroppartition in datacoord (#41543) 2025-04-27 17:42:40 +08:00
meta_util.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
metrics_info.go feat: [2.5]Ignore reporting index metrics for non-existent indexes (#41296) 2025-04-17 16:28:34 +08:00
metrics_info_test.go enhance: [2.5] Refine task meta with key lock (#40353) 2025-03-13 19:18:08 +08:00
mock_channel_manager.go fix: [2.5] mockery too unavailable after upgrade golang version (#41522) 2025-04-25 14:40:40 +08:00
mock_channel_store.go fix: ChannelManager double assignment (#41877) 2025-05-16 14:58:23 +08:00
mock_cluster.go fix: [2.5] mockery too unavailable after upgrade golang version (#41522) 2025-04-25 14:40:40 +08:00
mock_compaction_meta.go fix: [2.5] mockery too unavailable after upgrade golang version (#41522) 2025-04-25 14:40:40 +08:00
mock_compaction_plan_context.go fix: [2.5] mockery too unavailable after upgrade golang version (#41522) 2025-04-25 14:40:40 +08:00
mock_handler.go fix: [2.5] mockery too unavailable after upgrade golang version (#41522) 2025-04-25 14:40:40 +08:00
mock_index_engine_version_manager.go fix: [2.5] mockery too unavailable after upgrade golang version (#41522) 2025-04-25 14:40:40 +08:00
mock_segment_manager.go feat: refine drop parition through the new interface notifydroppartition in datacoord (#41543) 2025-04-27 17:42:40 +08:00
mock_stats_job_manager.go fix: [2.5] mockery too unavailable after upgrade golang version (#41522) 2025-04-25 14:40:40 +08:00
mock_sub_cluster.go fix: [2.5] mockery too unavailable after upgrade golang version (#41522) 2025-04-25 14:40:40 +08:00
mock_test.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
mock_trigger.go fix: [2.5] mockery too unavailable after upgrade golang version (#41522) 2025-04-25 14:40:40 +08:00
mock_trigger_manager.go fix: [2.5] mockery too unavailable after upgrade golang version (#41522) 2025-04-25 14:40:40 +08:00
partition_stats_meta.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
partition_stats_meta_test.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
policy.go fix: [2.5] Fix channel not balance on datanodes (#40854) 2025-03-25 10:06:24 +08:00
policy_test.go fix: [2.5] Fix channel not balance on datanodes (#40854) 2025-03-25 10:06:24 +08:00
segment_allocation_policy.go fix: [2.5] Ignore growing segment without start pos for seal policy (#41131) 2025-04-07 23:42:27 +08:00
segment_allocation_policy_test.go fix: [2.5] Ignore growing segment without start pos for seal policy (#41131) 2025-04-07 23:42:27 +08:00
segment_info.go fix: [2.5] Only mark segment compacting for sort stats task (#42517) 2025-06-04 21:40:32 +08:00
segment_info_test.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
segment_manager.go feat: refine drop parition through the new interface notifydroppartition in datacoord (#41543) 2025-04-27 17:42:40 +08:00
segment_manager_test.go feat: refine drop parition through the new interface notifydroppartition in datacoord (#41543) 2025-04-27 17:42:40 +08:00
segment_operator.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
segment_operator_test.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
server.go enhance: [2.5] Enhance import context (#42051) 2025-05-24 18:40:26 +08:00
server_test.go enhance: [2.5] Support detailed manual compaction criterion (#40892) (#40924) 2025-03-27 10:36:22 +08:00
services.go enhance: [2.5] Enhance import context (#42051) 2025-05-24 18:40:26 +08:00
services_test.go enhance: [2.5] Optimize datacoord meta mutex (#40753) 2025-03-25 13:50:21 +08:00
stats_task_meta.go fix: [2.5] Update logging context and upgrade dependencies (#41319) 2025-04-24 23:50:40 +08:00
stats_task_meta_test.go fix: [2.5] Update logging context and upgrade dependencies (#41319) 2025-04-24 23:50:40 +08:00
sync_segments_scheduler.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00
sync_segments_scheduler_test.go enhance: [2.5] Optimize datacoord meta mutex (#40753) 2025-03-25 13:50:21 +08:00
task_analyze.go enhance: [2.5] Support slot for index task and stats task (#40288) 2025-03-19 16:50:11 +08:00
task_index.go fix: [2.5] Use locking to ensure the atomicity of dropping segment indexes (#42076) 2025-05-26 15:40:28 +08:00
task_queue.go fix: skip loading non-existent L0 segments to prevent load blocking (#43576) 2025-07-31 14:33:38 +08:00
task_scheduler.go fix: [2.5] Ensure task execution order by using a priority queue (#43272) 2025-07-23 10:16:54 +08:00
task_scheduler_test.go fix: [2.5] Ensure task execution order by using a priority queue (#43272) 2025-07-23 10:16:54 +08:00
task_stats.go fix: [2.5] Fix index blocked by failed sort stats (#43061) 2025-07-02 22:32:45 +08:00
task_stats_test.go fix: [2.5] Update logging context and upgrade dependencies (#41319) 2025-04-24 23:50:40 +08:00
types.go enhance: [2.5] Support slot for index task and stats task (#40288) 2025-03-19 16:50:11 +08:00
util.go fix: [2.5]datacoord stop get stuck After upgrading from 2.5 to 2.6 (#42669) 2025-06-16 19:56:38 +08:00
util_test.go enhance: [2.5] Use v2 package name for pkg module (#40117) 2025-02-23 00:46:01 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

  • KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
  • Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
  • Root Coordinator: timestamp, id and meta source.
  • Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.