History

yihao.dai b6612e02b4 enhance: Reduce GetIndexInfos calls (#37695 ) Batch `GetIndexInfos` calls for segments to reduce RPC calls. issue: https://github.com/milvus-io/milvus/issues/37634 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>		2024-11-19 14:24:31 +08:00
..
allocator	…
broker	…
session	…
OWNERS	…
README.md	…
analyze_meta.go	…
analyze_meta_test.go	…
build_index_policy.go	…
channel.go	…
channel_manager.go	fix: Limit the concurrency of channel tasks (#37740 )	2024-11-18 16:26:30 +08:00
channel_manager_factory.go	fix: drop collection failed if enable streaming service (#37444 )	2024-11-07 10:26:26 +08:00
channel_manager_test.go	feat: add segment,pipeline, replica and resourcegroup api for WebUI (#37344 )	2024-11-07 11:52:25 +08:00
channel_store.go	fix: drop collection failed if enable streaming service (#37444 )	2024-11-07 10:26:26 +08:00
channel_store_test.go	fix: drop collection failed if enable streaming service (#37444 )	2024-11-07 10:26:26 +08:00
cluster.go	…
cluster_test.go	…
compaction.go	fix: refuse schedule compaction tasks if there is no slot (#37589 )	2024-11-13 15:12:30 +08:00
compaction_l0_view.go	…
compaction_l0_view_test.go	…
compaction_policy_clustering.go	enhance: Remove pre-marking segments as L2 during clustering compaction (#36799 )	2024-10-23 17:15:28 +08:00
compaction_policy_clustering_test.go	…
compaction_policy_l0.go	enhance: tidy compaction logs (#37595 )	2024-11-12 21:18:29 +08:00
compaction_policy_l0_test.go	…
compaction_policy_single.go	enhance: tidy compaction logs (#37595 )	2024-11-12 21:18:29 +08:00
compaction_policy_single_test.go	…
compaction_queue.go	fix: Refine compactionTask to avoid data race (#36936 )	2024-10-24 09:55:28 +08:00
compaction_queue_test.go	fix: Refine compactionTask to avoid data race (#36936 )	2024-10-24 09:55:28 +08:00
compaction_task.go	fix: Refine compactionTask to avoid data race (#36936 )	2024-10-24 09:55:28 +08:00
compaction_task_clustering.go	enhance: Handoff growing segment after sorted (#37385 )	2024-11-07 16:08:24 +08:00
compaction_task_clustering_test.go	fix: Refine compactionTask to avoid data race (#36936 )	2024-10-24 09:55:28 +08:00
compaction_task_l0.go	fix: Exlude L0 compaction when clustering is executing (#37141 )	2024-10-29 06:28:24 +08:00
compaction_task_l0_test.go	fix: Exlude L0 compaction when clustering is executing (#37141 )	2024-10-29 06:28:24 +08:00
compaction_task_meta.go	enhance: Use internal json(sonic) to replace std json lib (#37708 )	2024-11-18 10:46:31 +08:00
compaction_task_meta_test.go	enhance: Use internal json(sonic) to replace std json lib (#37708 )	2024-11-18 10:46:31 +08:00
compaction_task_mix.go	fix: refuse schedule compaction tasks if there is no slot (#37589 )	2024-11-13 15:12:30 +08:00
compaction_task_mix_test.go	fix: Skip mark compaction timeout for mix and l0 compaction (#37118 )	2024-10-28 14:33:29 +08:00
compaction_test.go	fix: refuse schedule compaction tasks if there is no slot (#37589 )	2024-11-13 15:12:30 +08:00
compaction_trigger.go	fix: Separate L0 and Mix trigger interval (#37190 )	2024-11-12 10:56:37 +08:00
compaction_trigger_test.go	fix: Int64 overflow in JSON encoding (#37657 )	2024-11-14 22:52:30 +08:00
compaction_trigger_v2.go	fix: Separate L0 and Mix trigger interval (#37190 )	2024-11-12 10:56:37 +08:00
compaction_trigger_v2_test.go	…
compaction_view.go	…
const.go	…
errors.go	…
errors_test.go	…
garbage_collector.go	enhance: Support gc with bm25 stats logs (#37408 )	2024-11-12 11:30:28 +08:00
garbage_collector_test.go	fix: Int64 overflow in JSON encoding (#37657 )	2024-11-14 22:52:30 +08:00
go_channel_singleton.go	…
handler.go	enhance: Handoff growing segment after sorted (#37385 )	2024-11-07 16:08:24 +08:00
handler_test.go	enhance: Handoff growing segment after sorted (#37385 )	2024-11-07 16:08:24 +08:00
import_checker.go	…
import_checker_test.go	…
import_job.go	fix: Revert "enhance: Support db for bulkinsert (#37012 )" (#37420 )	2024-11-07 17:02:25 +08:00
import_meta.go	enhance: Use internal json(sonic) to replace std json lib (#37708 )	2024-11-18 10:46:31 +08:00
import_meta_test.go	enhance: Use internal json(sonic) to replace std json lib (#37708 )	2024-11-18 10:46:31 +08:00
import_scheduler.go	…
import_scheduler_test.go	…
import_task.go	enhance: Use internal json(sonic) to replace std json lib (#37708 )	2024-11-18 10:46:31 +08:00
import_util.go	feat: add tasks page into management WebUI (#37002 )	2024-10-28 10:13:29 +08:00
import_util_test.go	enhance: Use internal json(sonic) to replace std json lib (#37708 )	2024-11-18 10:46:31 +08:00
index_engine_version_manager.go	…
index_engine_version_manager_test.go	…
index_meta.go	enhance: Reduce GetIndexInfos calls (#37695 )	2024-11-19 14:24:31 +08:00
index_meta_test.go	enhance: Reduce GetIndexInfos calls (#37695 )	2024-11-19 14:24:31 +08:00
index_service.go	enhance: Reduce GetIndexInfos calls (#37695 )	2024-11-19 14:24:31 +08:00
index_service_test.go	fix: Int64 overflow in JSON encoding (#37657 )	2024-11-14 22:52:30 +08:00
job_manager.go	…
job_manager_test.go	feat: add segment/channel/task/slow query render (#37561 )	2024-11-12 17:44:29 +08:00
meta.go	enhance: Remove segment-level tag from monitoring metrics (#37696 )	2024-11-16 10:58:31 +08:00
meta_test.go	fix: l0RowCount metrics value always empty (#37306 )	2024-11-11 17:00:27 +08:00
meta_util.go	…
metrics_info.go	enhance: Use internal json(sonic) to replace std json lib (#37708 )	2024-11-18 10:46:31 +08:00
metrics_info_test.go	enhance: Use internal json(sonic) to replace std json lib (#37708 )	2024-11-18 10:46:31 +08:00
mock_channel_store.go	enhance: regenerate datacoord mockery (#37501 )	2024-11-07 17:38:24 +08:00
mock_channelmanager.go	feat: add segment,pipeline, replica and resourcegroup api for WebUI (#37344 )	2024-11-07 11:52:25 +08:00
mock_cluster.go	…
mock_compaction_meta.go	…
mock_compaction_plan_context.go	…
mock_handler.go	…
mock_index_engine_version_manager.go	…
mock_job_manager.go	…
mock_segment_manager.go	…
mock_subcluster.go	…
mock_test.go	enhance: Support custom privilege group for RBAC (#37087 )	2024-11-09 08:44:28 +08:00
mock_trigger_manager.go	…
partition_stats_meta.go	fix: Set current partition stats version to 0 by default when not present (#37299 )	2024-10-31 12:48:21 +08:00
partition_stats_meta_test.go	fix: Set current partition stats version to 0 by default when not present (#37299 )	2024-10-31 12:48:21 +08:00
policy.go	fix: drop collection failed if enable streaming service (#37444 )	2024-11-07 10:26:26 +08:00
policy_test.go	…
segment_allocation_policy.go	fix: Fix large growing segment (#37388 )	2024-11-07 17:12:25 +08:00
segment_allocation_policy_test.go	…
segment_info.go	enhance: Remove unnecessary clone in SetState (#37697 )	2024-11-15 15:06:30 +08:00
segment_info_test.go	…
segment_manager.go	…
segment_manager_test.go	fix: use binlog counter to trigger flush but not stats log (#37037 )	2024-10-23 15:07:29 +08:00
segment_operator.go	…
segment_operator_test.go	…
server.go	enhance: Handoff growing segment after sorted (#37385 )	2024-11-07 16:08:24 +08:00
server_test.go	fix: Int64 overflow in JSON encoding (#37657 )	2024-11-14 22:52:30 +08:00
services.go	fix: Change memoryCheck write lock to read lock (#37525 )	2024-11-15 10:44:31 +08:00
services_test.go	fix: Int64 overflow in JSON encoding (#37657 )	2024-11-14 22:52:30 +08:00
stats_task_meta.go	feat: add segment,pipeline, replica and resourcegroup api for WebUI (#37344 )	2024-11-07 11:52:25 +08:00
stats_task_meta_test.go	…
sync_segments_scheduler.go	…
sync_segments_scheduler_test.go	…
task_analyze.go	…
task_index.go	enhance: add unify vector index config management (#36846 )	2024-11-01 06:18:21 +08:00
task_scheduler.go	enhance: Reduce GetIndexInfos calls (#37695 )	2024-11-19 14:24:31 +08:00
task_scheduler_test.go	fix: Int64 overflow in JSON encoding (#37657 )	2024-11-14 22:52:30 +08:00
task_stats.go	…
task_stats_test.go	enhance: Remove unnecessary clone in SetState (#37697 )	2024-11-15 15:06:30 +08:00
types.go	…
util.go	enhance: add vector index mgr to remove vector index type dependency (#36843 )	2024-10-17 22:15:25 +08:00
util_test.go	…

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
Root Coordinator: timestamp, id and meta source.
Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.