milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
milvus/internal/datacoord
History
chyezh 0c7474d7e8
enhance: add graceful stop timeout to avoid node stop hang under extreme cases (#30317) 
1. add coordinator graceful stop timeout to 5s
2. change the order of datacoord component while stop
3. change querynode grace stop timeout to 900s, and we should
potentially change this to 600s when graceful stop is smooth

issue: #30310
also see pr: #30306

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 		2024-02-29 17:01:50 +08:00
..
broker enhance: Move datacoord broker into separate package (#28876) 2023-12-01 10:22:34 +08:00
OWNERS [skip ci]Update OWNERS files (#11898) 2021-11-16 15:41:11 +08:00
README.md [skip ci]Change etcd to lowercase (#9983) 2021-10-15 18:58:37 +08:00
allocator.go enhance:Remove msgbase unnecessary assignments (#28511) 2023-11-24 15:02:39 +08:00
allocator_test.go Decoupling client and server API in types interface (#27186) 2023-09-26 09:57:25 +08:00
build_index_policy.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
channel.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
channel_checker.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
channel_checker_test.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
channel_manager.go enhance:change some logs (#29579) 2024-01-05 16:12:48 +08:00
channel_manager_factory.go enhance: Remove not in use policies (#29448) 2023-12-28 10:38:46 +08:00
channel_manager_test.go enhance: Change ChannelManager to interface (#29300) 2023-12-25 19:24:46 +08:00
channel_store.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
channel_store_test.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
cluster.go enhance: Change ChannelManager to interface (#29300) 2023-12-25 19:24:46 +08:00
cluster_test.go enhance: Change ChannelManager to interface (#29300) 2023-12-25 19:24:46 +08:00
compaction.go enhance: Refine compaction interfaces to support major compaction (#30632) 2024-02-19 20:52:50 +08:00
compaction_l0_view.go enhance: Add force trigger (#30641) 2024-02-19 18:40:50 +08:00
compaction_l0_view_test.go enhance: Add force trigger (#30641) 2024-02-19 18:40:50 +08:00
compaction_scheduler.go enhance: Add CompactionTaskNum metrics (#29518) 2023-12-28 15:46:55 +08:00
compaction_scheduler_test.go enhance: Add CompactionTaskNum metrics (#29518) 2023-12-28 15:46:55 +08:00
compaction_test.go fix: Donot set LogPath when executing compaction (#30537) 2024-02-28 19:03:01 +08:00
compaction_trigger.go fix: donot set l0 segment as growing when savebinlogs (#29194) 2024-02-04 10:21:06 +08:00
compaction_trigger_test.go fix: donot set l0 segment as growing when savebinlogs (#29194) 2024-02-04 10:21:06 +08:00
compaction_trigger_v2.go fix: Fill in info in CompactionSegmentBinlogs (#30279) 2024-01-26 10:51:02 +08:00
compaction_trigger_v2_test.go fix: Fix L0 compaction in datacoord (#28814) 2023-12-05 18:44:37 +08:00
compaction_view.go fix: Fix some left comments (#29027) 2023-12-13 18:48:39 +08:00
compaction_view_manager.go fix: Fix L0 compaction in datacoord (#28814) 2023-12-05 18:44:37 +08:00
compaction_view_manager_test.go fix: Limit L0 Compaction segment size and count (#30374) 2024-02-01 20:39:03 +08:00
const.go enhance: pass partition key scalar info if enabled when build vector index (#29931) 2024-01-24 00:04:55 +08:00
errors.go Refine DataCoord status (#27262) 2023-09-26 17:15:27 +08:00
errors_test.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
garbage_collector.go enhance: Add log when garbage collection resumed (#30535) 2024-02-05 17:09:15 +08:00
garbage_collector_test.go enhance:don't store logPath in meta to reduce memory (#28873) 2024-01-18 22:06:31 +08:00
handler.go enhance: Explicitly pass LevelZero segment ids in vchan info (#29612) 2024-01-04 16:46:45 +08:00
index_builder.go fix: compatibility for diskann cache param (#30119) 2024-02-26 16:54:55 +08:00
index_builder_test.go enhance: pass partition key scalar info if enabled when build vector index (#29931) 2024-01-24 00:04:55 +08:00
index_engine_version_manager.go feat: trigger compaction to handle index version (#28442) 2023-11-21 09:26:22 +08:00
index_engine_version_manager_test.go Optimize the codec code of session (#27360) 2023-10-01 10:33:30 +08:00
index_meta.go feat: Support multiple vector indexes in a collection (#27700) 2023-12-29 11:44:45 +08:00
index_meta_test.go feat: Support multiple vector indexes in a collection (#27700) 2023-12-29 11:44:45 +08:00
index_service.go fix: alter_index should return error if index not found (#30786) 2024-02-29 15:55:01 +08:00
index_service_test.go fix: report error if the altering index doesn't support mmap (#29832) 2024-01-17 16:40:54 +08:00
indexnode_manager.go Decoupling client and server API in types interface (#27186) 2023-09-26 09:57:25 +08:00
indexnode_manager_test.go Refine state check (#27541) 2023-10-11 21:01:35 +08:00
meta.go fix: Donot set LogPath when executing compaction (#30537) 2024-02-28 19:03:01 +08:00
meta_test.go fix: Donot set LogPath when executing compaction (#30537) 2024-02-28 19:03:01 +08:00
meta_util.go Make VChannelInfo read compatible with 2.0.2 (#17995) 2022-07-07 21:58:22 +08:00
metrics_info.go disable auto balance when old node exists (#28191) 2023-11-07 14:02:20 +08:00
metrics_info_test.go Refine state check (#27541) 2023-10-11 21:01:35 +08:00
mock_allocator_test.go Organize all mockery generation commands in Makefile (#26826) 2023-09-04 21:19:48 +08:00
mock_channel_store.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
mock_channelmanager.go enhance: Add CompactionTaskNum metrics (#29518) 2023-12-28 15:46:55 +08:00
mock_cluster.go enhance: pack datacoord Cluster and SessionManager with interface and mock them (#28869) 2023-12-11 17:52:37 +08:00
mock_compaction_meta.go enhance: Refine compaction interfaces to support major compaction (#30632) 2024-02-19 20:52:50 +08:00
mock_compaction_plan_context.go fix: Separate schedule and check results loop (#28692) 2023-11-29 10:50:29 +08:00
mock_handler.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
mock_index_engine_version_manager.go feat: trigger compaction to handle index version (#28442) 2023-11-21 09:26:22 +08:00
mock_scheduler.go enhance: Add CompactionTaskNum metrics (#29518) 2023-12-28 15:46:55 +08:00
mock_session_manager.go enhance: Add open telemetry tracing for compaction (#30168) 2024-01-23 10:37:00 +08:00
mock_test.go feat: Implement DescribeAlias and ListAliases interfaces (#29641) 2024-01-11 19:12:51 +08:00
mock_trigger_manager.go feat: Add Compaction views and triggers (#27906) 2023-11-23 17:30:25 +08:00
policy.go enhance: Remove not in use policies (#29448) 2023-12-28 10:38:46 +08:00
policy_test.go enhance: Remove not in use policies (#29448) 2023-12-28 10:38:46 +08:00
segment_allocation_policy.go fix: Decoupling importing segment from flush process (#30402) 2024-02-03 13:01:12 +08:00
segment_allocation_policy_test.go Adjust datacoord for L0 Delta (#28021) 2023-11-06 15:26:16 +08:00
segment_info.go enhance:don't store logPath in meta to reduce memory (#28873) 2024-01-18 22:06:31 +08:00
segment_manager.go fix: Decoupling importing segment from flush process (#30402) 2024-02-03 13:01:12 +08:00
segment_manager_test.go fix: Decoupling importing segment from flush process (#30402) 2024-02-03 13:01:12 +08:00
server.go enhance: add graceful stop timeout to avoid node stop hang under extreme cases (#30317) 2024-02-29 17:01:50 +08:00
server_test.go fix: Decoupling importing segment from flush process (#30402) 2024-02-03 13:01:12 +08:00
services.go fix: Donot set LogPath when executing compaction (#30537) 2024-02-28 19:03:01 +08:00
services_test.go fix: [skip-e2e] Fix unstable SaveBinlogPath ut (#30508) 2024-02-05 16:00:59 +08:00
session.go Decoupling client and server API in types interface (#27186) 2023-09-26 09:57:25 +08:00
session_manager.go enhance: Add open telemetry tracing for compaction (#30168) 2024-01-23 10:37:00 +08:00
session_manager_test.go enhance: Change ChannelManager to interface (#29300) 2023-12-25 19:24:46 +08:00
util.go fix: compatibility for diskann cache param (#30119) 2024-02-26 16:54:55 +08:00
util_test.go enhance: add l0 metric and fix datacoord no need drop l0 segment when flush (#28373) 2023-11-24 15:58:24 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

  • KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
  • Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
  • Root Coordinator: timestamp, id and meta source.
  • Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.