milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
milvus/internal/datacoord
History
Zhen Ye 6f1febe881
enhance: move streaming coord from datacoord to rootcoord (#39009) 
issue: #38399
pr: #39007

We want to support broadcast operation for both streaming and msgstream.
But msgstream can be only sent message from rootcoord and proxy.
So this pr move the streamingcoord to rootcoord to make easier
implementation.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 		2025-01-07 17:56:56 +08:00
..
allocator fix: fix `make generate-mockery` panic with go1.22 (#36830) 2024-10-17 12:11:31 +08:00
broker fix: fix `make generate-mockery` panic with go1.22 (#36830) 2024-10-17 12:11:31 +08:00
session fix: SyncSegments rpc always failed (#38578) 2024-12-19 15:40:45 +08:00
OWNERS [skip ci]Update OWNERS files (#11898) 2021-11-16 15:41:11 +08:00
README.md [skip ci]Change etcd to lowercase (#9983) 2021-10-15 18:58:37 +08:00
analyze_meta.go enhance: refine the datacoord meta related interfaces (#37957) 2024-11-26 19:46:34 +08:00
analyze_meta_test.go enhance: refine the datacoord meta related interfaces (#37957) 2024-11-26 19:46:34 +08:00
build_index_policy.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
channel.go fix: ChannelManager concurret Release and Watch bug (#38590) 2024-12-19 22:50:47 +08:00
channel_manager.go fix: ChannelManager concurret Release and Watch bug (#38590) 2024-12-19 22:50:47 +08:00
channel_manager_factory.go fix: drop collection failed if enable streaming service (#37444) 2024-11-07 10:26:26 +08:00
channel_manager_test.go fix: ChannelManager concurret Release and Watch bug (#38590) 2024-12-19 22:50:47 +08:00
channel_store.go fix: ChannelManager concurret Release and Watch bug (#38590) 2024-12-19 22:50:47 +08:00
channel_store_test.go fix: ChannelManager concurret Release and Watch bug (#38590) 2024-12-19 22:50:47 +08:00
cluster.go enhance: refine exists log print with ctx (#38080) 2024-12-14 22:36:44 +08:00
cluster_test.go enhance: Move datanode/indexnode manager to session pkg (#35634) 2024-08-22 16:02:56 +08:00
compaction.go fix: [2.5] Release lock when return function (#38863) 2024-12-30 22:44:39 +08:00
compaction_l0_view.go enhance: Add deltaRowCount in l0 compaction (#33997) 2024-06-20 17:46:01 +08:00
compaction_l0_view_test.go enhance: Add deltaRowCount in l0 compaction (#33997) 2024-06-20 17:46:01 +08:00
compaction_policy_clustering.go enhance: refine the datacoord meta related interfaces (#37957) 2024-11-26 19:46:34 +08:00
compaction_policy_clustering_test.go enhance: refine the datacoord meta related interfaces (#37957) 2024-11-26 19:46:34 +08:00
compaction_policy_l0.go enhance: tidy compaction logs (#37595) 2024-11-12 21:18:29 +08:00
compaction_policy_l0_test.go enhance: Separate allocator pkg in datacoord (#35622) 2024-08-22 10:06:56 +08:00
compaction_policy_single.go enhance: tidy compaction logs (#37595) 2024-11-12 21:18:29 +08:00
compaction_policy_single_test.go enhance: refine the datacoord meta related interfaces (#37957) 2024-11-26 19:46:34 +08:00
compaction_queue.go fix: Refine compactionTask to avoid data race (#36936) 2024-10-24 09:55:28 +08:00
compaction_queue_test.go fix: Refine compactionTask to avoid data race (#36936) 2024-10-24 09:55:28 +08:00
compaction_task.go enhance: clean compaction task in compactionHandler (#38170) 2024-12-19 12:36:47 +08:00
compaction_task_clustering.go enhance: clean compaction task in compactionHandler (#38170) 2024-12-19 12:36:47 +08:00
compaction_task_clustering_test.go enhance: clean compaction task in compactionHandler (#38170) 2024-12-19 12:36:47 +08:00
compaction_task_l0.go enhance: clean compaction task in compactionHandler (#38170) 2024-12-19 12:36:47 +08:00
compaction_task_l0_test.go enhance: clean compaction task in compactionHandler (#38170) 2024-12-19 12:36:47 +08:00
compaction_task_meta.go enhance: refine the datacoord meta related interfaces (#37957) 2024-11-26 19:46:34 +08:00
compaction_task_meta_test.go enhance: refine the datacoord meta related interfaces (#37957) 2024-11-26 19:46:34 +08:00
compaction_task_mix.go enhance: clean compaction task in compactionHandler (#38170) 2024-12-19 12:36:47 +08:00
compaction_task_mix_test.go enhance: refine the datacoord meta related interfaces (#37957) 2024-11-26 19:46:34 +08:00
compaction_test.go enhance: clean compaction task in compactionHandler (#38170) 2024-12-19 12:36:47 +08:00
compaction_trigger.go enhance: refine exists log print with ctx (#38080) 2024-12-14 22:36:44 +08:00
compaction_trigger_test.go enhance: improve mix compaction performance by removing max segment limitations (#38344) 2024-12-11 20:38:42 +08:00
compaction_trigger_v2.go enhance: refine exists log print with ctx (#38080) 2024-12-14 22:36:44 +08:00
compaction_trigger_v2_test.go enhance: Separate allocator pkg in datacoord (#35622) 2024-08-22 10:06:56 +08:00
compaction_view.go enhance: Add deltaRowCount in l0 compaction (#33997) 2024-06-20 17:46:01 +08:00
const.go enhance: pass partition key scalar info if enabled when build vector index (#29931) 2024-01-24 00:04:55 +08:00
errors.go Refine DataCoord status (#27262) 2023-09-26 17:15:27 +08:00
errors_test.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
garbage_collector.go enhance: refine exists log print with ctx (#38080) 2024-12-14 22:36:44 +08:00
garbage_collector_test.go enhance: refine the datacoord meta related interfaces (#37957) 2024-11-26 19:46:34 +08:00
go_channel_singleton.go fix: Fix improper use of offset in HybridSearch (#36244) 2024-09-13 22:05:15 +08:00
handler.go fix: Support get segments from current segments view (#38512) 2024-12-18 18:00:54 +08:00
handler_test.go fix: Support get segments from current segments view (#38512) 2024-12-18 18:00:54 +08:00
import_checker.go fix: Use the correct RootPath when decompressing binlog in stats task (#38341) 2024-12-11 16:16:42 +08:00
import_checker_test.go fix: Use the correct RootPath when decompressing binlog in stats task (#38341) 2024-12-11 16:16:42 +08:00
import_job.go fix: Revert "enhance: Support db for bulkinsert (#37012)" (#37420) 2024-11-07 17:02:25 +08:00
import_meta.go enhance: refine the datacoord meta related interfaces (#37957) 2024-11-26 19:46:34 +08:00
import_meta_test.go enhance: refine the datacoord meta related interfaces (#37957) 2024-11-26 19:46:34 +08:00
import_scheduler.go fix:[Cherry-pick] bm25 import segment loss stats (#38881) 2024-12-31 19:24:54 +08:00
import_scheduler_test.go enhance: refine the datacoord meta related interfaces (#37957) 2024-11-26 19:46:34 +08:00
import_task.go enhance: Use internal json(sonic) to replace std json lib (#37708) 2024-11-18 10:46:31 +08:00
import_util.go fix: Use the correct RootPath when decompressing binlog in stats task (#38341) 2024-12-11 16:16:42 +08:00
import_util_test.go fix: Use the correct RootPath when decompressing binlog in stats task (#38341) 2024-12-11 16:16:42 +08:00
index_engine_version_manager.go enhance: Add `go-deadlock` as unittest only dependency (#33063) 2024-05-15 16:33:34 +08:00
index_engine_version_manager_test.go Optimize the codec code of session (#27360) 2023-10-01 10:33:30 +08:00
index_meta.go enhance: refine exists log print with ctx (#38080) 2024-12-14 22:36:44 +08:00
index_meta_test.go enhance: Add ctx param to KV operation interfaces (#38154) 2024-12-05 15:16:41 +08:00
index_service.go enhance: refine exists log print with ctx (#38080) 2024-12-14 22:36:44 +08:00
index_service_test.go enhance: alterindex & altercollection supports altering properties (#37437) 2024-12-11 10:20:42 +08:00
job_manager.go fix: Use the correct RootPath when decompressing binlog in stats task (#38341) 2024-12-11 16:16:42 +08:00
job_manager_test.go feat: add segment/channel/task/slow query render (#37561) 2024-11-12 17:44:29 +08:00
knapsack.go enhance: improve mix compaction performance by removing max segment limitations (#38344) 2024-12-11 20:38:42 +08:00
knapsack_test.go enhance: try compact small segments first if they may compose a full segment (#37709) 2024-12-02 16:12:38 +08:00
meta.go fix:[Cherry-pick] bm25 import segment loss stats (#38881) 2024-12-31 19:24:54 +08:00
meta_test.go fix:[Cherry-pick] bm25 import segment loss stats (#38881) 2024-12-31 19:24:54 +08:00
meta_util.go Make VChannelInfo read compatible with 2.0.2 (#17995) 2022-07-07 21:58:22 +08:00
metrics_info.go enhance: add disk stats within system metrics (#38033) 2024-12-06 16:32:41 +08:00
metrics_info_test.go enhance: add list index and segment index retrieval API for WebUI (#37861) 2024-11-22 16:58:34 +08:00
mock_channel_store.go fix: ChannelManager concurret Release and Watch bug (#38590) 2024-12-19 22:50:47 +08:00
mock_channelmanager.go feat: add segment,pipeline, replica and resourcegroup api for WebUI (#37344) 2024-11-07 11:52:25 +08:00
mock_cluster.go fix: fix `make generate-mockery` panic with go1.22 (#36830) 2024-10-17 12:11:31 +08:00
mock_compaction_meta.go enhance: refine the datacoord meta related interfaces (#37957) 2024-11-26 19:46:34 +08:00
mock_compaction_plan_context.go enhance: refine the datacoord meta related interfaces (#37957) 2024-11-26 19:46:34 +08:00
mock_handler.go fix: Support get segments from current segments view (#38512) 2024-12-18 18:00:54 +08:00
mock_index_engine_version_manager.go fix: fix `make generate-mockery` panic with go1.22 (#36830) 2024-10-17 12:11:31 +08:00
mock_job_manager.go fix: Use the correct RootPath when decompressing binlog in stats task (#38341) 2024-12-11 16:16:42 +08:00
mock_segment_manager.go enhance: Reduce segmentManager lock granularity (#37836) 2024-12-17 14:12:52 +08:00
mock_subcluster.go fix: fix `make generate-mockery` panic with go1.22 (#36830) 2024-10-17 12:11:31 +08:00
mock_test.go fix: revert optimize CPU usage for CheckHealth requests (#35589) (#38555) 2024-12-19 00:38:45 +08:00
mock_trigger_manager.go fix: fix `make generate-mockery` panic with go1.22 (#36830) 2024-10-17 12:11:31 +08:00
partition_stats_meta.go enhance: clean compaction task in compactionHandler (#38170) 2024-12-19 12:36:47 +08:00
partition_stats_meta_test.go enhance: clean compaction task in compactionHandler (#38170) 2024-12-19 12:36:47 +08:00
policy.go fix: drop collection failed if enable streaming service (#37444) 2024-11-07 10:26:26 +08:00
policy_test.go fix: Fill nil schema so that Milvus can watch channel for those upgraded from 2.2 to 2.4 #35695 (#35694) 2024-08-27 10:36:59 +08:00
segment_allocation_policy.go fix: Fix large growing segment (#37388) 2024-11-07 17:12:25 +08:00
segment_allocation_policy_test.go enhance: Change the fixed value to a ratio for clustering segment size (#35076) 2024-08-01 22:04:14 +08:00
segment_info.go enhance: Remove unnecessary clone in SetState (#37697) 2024-11-15 15:06:30 +08:00
segment_info_test.go fix: Return all compactTo segments after support split (#36361) 2024-09-20 14:11:11 +08:00
segment_manager.go enhance: Reduce segmentManager lock granularity (#37836) 2024-12-17 14:12:52 +08:00
segment_manager_test.go enhance: Reduce segmentManager lock granularity (#37836) 2024-12-17 14:12:52 +08:00
segment_operator.go enhance: Update stats task to optional (#35947) 2024-09-12 20:37:08 +08:00
segment_operator_test.go fix: Prevent clone when selecting segments from meta (#30928) 2024-03-01 13:59:02 +08:00
server.go enhance: move streaming coord from datacoord to rootcoord (#39009) 2025-01-07 17:56:56 +08:00
server_test.go fix: revert optimize CPU usage for CheckHealth requests (#35589) (#38555) 2024-12-19 00:38:45 +08:00
services.go fix: Revert "Expose metrics of stanby coordinators (#27698)" (#38621) 2024-12-20 18:04:47 +08:00
services_test.go enhance: Reduce segmentManager lock granularity (#37836) 2024-12-17 14:12:52 +08:00
stats_task_meta.go feat: add segment,pipeline, replica and resourcegroup api for WebUI (#37344) 2024-11-07 11:52:25 +08:00
stats_task_meta_test.go fix: Record the nodeID before assigning tasks (#36371) 2024-09-28 17:21:15 +08:00
sync_segments_scheduler.go fix: Support get segments from current segments view (#38512) 2024-12-18 18:00:54 +08:00
sync_segments_scheduler_test.go fix: datacoord stuck at stopping progress (#36852) 2024-10-17 12:13:37 +08:00
task_analyze.go enhance: refine the datacoord meta related interfaces (#37957) 2024-11-26 19:46:34 +08:00
task_index.go fix:remove unnecessary error logs (#38245) 2024-12-08 17:20:40 +08:00
task_scheduler.go enhance: Refine task scheduler logs (#38334) 2024-12-11 15:00:44 +08:00
task_scheduler_test.go enhance: refine the datacoord meta related interfaces (#37957) 2024-11-26 19:46:34 +08:00
task_stats.go enhance: refine the datacoord meta related interfaces (#37957) 2024-11-26 19:46:34 +08:00
task_stats_test.go enhance: refine the datacoord meta related interfaces (#37957) 2024-11-26 19:46:34 +08:00
types.go fix: Record the nodeID before assigning tasks (#36371) 2024-09-28 17:21:15 +08:00
util.go fix: revert optimize CPU usage for CheckHealth requests (#35589) (#38555) 2024-12-19 00:38:45 +08:00
util_test.go enhance: Separate allocator pkg in datacoord (#35622) 2024-08-22 10:06:56 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

  • KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
  • Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
  • Root Coordinator: timestamp, id and meta source.
  • Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.