History

cai.zhang 2c9bb4dfa3 feat: Support stats task to sort segment by PK (#35054 ) issue: #33744 This PR includes the following changes: 1. Added a new task type to the task scheduler in datacoord: stats task, which sorts segments by primary key. 2. Implemented segment sorting in indexnode. 3. Added a new field `FieldStatsLog` to SegmentInfo to store token index information. --------- Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>		2024-09-02 14:19:03 +08:00
..
allocator	enhance: Separate allocator pkg in datacoord (#35622 )	2024-08-22 10:06:56 +08:00
broker	…
session	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
OWNERS	…
README.md	…
analyze_meta.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
analyze_meta_test.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
build_index_policy.go	…
channel.go	fix: Fill nil schema so that Milvus can watch channel for those upgraded from 2.2 to 2.4 #35695 (#35694 )	2024-08-27 10:36:59 +08:00
channel_manager.go	enhance: using streaming service in insert/upsert/flush/delete/querynode (#35406 )	2024-08-29 10:03:08 +08:00
channel_manager_factory.go	…
channel_manager_test.go	fix: Fill nil schema so that Milvus can watch channel for those upgraded from 2.2 to 2.4 #35695 (#35694 )	2024-08-27 10:36:59 +08:00
channel_store.go	…
channel_store_test.go	fix: Fill nil schema so that Milvus can watch channel for those upgraded from 2.2 to 2.4 #35695 (#35694 )	2024-08-27 10:36:59 +08:00
cluster.go	enhance: Move datanode/indexnode manager to session pkg (#35634 )	2024-08-22 16:02:56 +08:00
cluster_test.go	enhance: Move datanode/indexnode manager to session pkg (#35634 )	2024-08-22 16:02:56 +08:00
compaction.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
compaction_l0_view.go	…
compaction_l0_view_test.go	…
compaction_policy_clustering.go	enhance: Separate allocator pkg in datacoord (#35622 )	2024-08-22 10:06:56 +08:00
compaction_policy_clustering_test.go	enhance: Separate allocator pkg in datacoord (#35622 )	2024-08-22 10:06:56 +08:00
compaction_policy_l0.go	…
compaction_policy_l0_test.go	enhance: Separate allocator pkg in datacoord (#35622 )	2024-08-22 10:06:56 +08:00
compaction_policy_single.go	enhance: Separate allocator pkg in datacoord (#35622 )	2024-08-22 10:06:56 +08:00
compaction_policy_single_test.go	enhance: Separate allocator pkg in datacoord (#35622 )	2024-08-22 10:06:56 +08:00
compaction_task.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
compaction_task_clustering.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
compaction_task_clustering_test.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
compaction_task_l0.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
compaction_task_l0_test.go	enhance: Move datanode/indexnode manager to session pkg (#35634 )	2024-08-22 16:02:56 +08:00
compaction_task_meta.go	…
compaction_task_meta_test.go	…
compaction_task_mix.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
compaction_task_mix_test.go	enhance: Enable to write multiple segments in mix compactor (#35705 )	2024-08-30 11:29:01 +08:00
compaction_task_test.go	enhance: Move datanode/indexnode manager to session pkg (#35634 )	2024-08-22 16:02:56 +08:00
compaction_test.go	enhance: Move datanode/indexnode manager to session pkg (#35634 )	2024-08-22 16:02:56 +08:00
compaction_trigger.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
compaction_trigger_test.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
compaction_trigger_v2.go	enhance: Enable to write multiple segments in mix compactor (#35705 )	2024-08-30 11:29:01 +08:00
compaction_trigger_v2_test.go	enhance: Separate allocator pkg in datacoord (#35622 )	2024-08-22 10:06:56 +08:00
compaction_view.go	…
const.go	…
errors.go	…
errors_test.go	…
garbage_collector.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
garbage_collector_test.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
handler.go	fix: Clear channelcp meta and metrics ASAP (#35658 )	2024-08-26 12:22:57 +08:00
import_checker.go	enhance: Separate allocator pkg in datacoord (#35622 )	2024-08-22 10:06:56 +08:00
import_checker_test.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
import_job.go	…
import_meta.go	…
import_meta_test.go	…
import_scheduler.go	enhance: Move datanode/indexnode manager to session pkg (#35634 )	2024-08-22 16:02:56 +08:00
import_scheduler_test.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
import_task.go	…
import_util.go	enhance: Separate allocator pkg in datacoord (#35622 )	2024-08-22 10:06:56 +08:00
import_util_test.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
index_engine_version_manager.go	…
index_engine_version_manager_test.go	…
index_meta.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
index_meta_test.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
index_service.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
index_service_test.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
meta.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
meta_test.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
meta_util.go	…
metrics_info.go	enhance: Move datanode/indexnode manager to session pkg (#35634 )	2024-08-22 16:02:56 +08:00
metrics_info_test.go	enhance: Move datanode/indexnode manager to session pkg (#35634 )	2024-08-22 16:02:56 +08:00
mock_channel_store.go	…
mock_channelmanager.go	…
mock_cluster.go	enhance: Move datanode/indexnode manager to session pkg (#35634 )	2024-08-22 16:02:56 +08:00
mock_compaction_meta.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
mock_compaction_plan_context.go	…
mock_handler.go	…
mock_index_engine_version_manager.go	…
mock_segment_manager.go	…
mock_subcluster.go	…
mock_test.go	enhance: Separate allocator pkg in datacoord (#35622 )	2024-08-22 10:06:56 +08:00
mock_trigger_manager.go	…
partition_stats_meta.go	…
partition_stats_meta_test.go	…
policy.go	…
policy_test.go	fix: Fill nil schema so that Milvus can watch channel for those upgraded from 2.2 to 2.4 #35695 (#35694 )	2024-08-27 10:36:59 +08:00
segment_allocation_policy.go	…
segment_allocation_policy_test.go	…
segment_info.go	…
segment_info_test.go	…
segment_manager.go	enhance: Separate allocator pkg in datacoord (#35622 )	2024-08-22 10:06:56 +08:00
segment_manager_test.go	enhance: Separate allocator pkg in datacoord (#35622 )	2024-08-22 10:06:56 +08:00
segment_operator.go	…
segment_operator_test.go	…
server.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
server_test.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
services.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
services_test.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
stats_task_meta.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
stats_task_meta_test.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
sync_segments_scheduler.go	enhance: Move datanode/indexnode manager to session pkg (#35634 )	2024-08-22 16:02:56 +08:00
sync_segments_scheduler_test.go	enhance: Move datanode/indexnode manager to session pkg (#35634 )	2024-08-22 16:02:56 +08:00
task_analyze.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
task_index.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
task_scheduler.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
task_scheduler_test.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
task_stats.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
task_stats_test.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
types.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
util.go	feat: Support stats task to sort segment by PK (#35054 )	2024-09-02 14:19:03 +08:00
util_test.go	enhance: Separate allocator pkg in datacoord (#35622 )	2024-08-22 10:06:56 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
Root Coordinator: timestamp, id and meta source.
Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.