milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
milvus/internal/datacoord
History
congqixia 6f7b2b4e75
enhance: [2.5] Refine error msg for schema & index checking (#39533) (#39565) 
Cherry-pick from master
pr: #39533

The error message was malformated or missing some meta info, say field
name. This PR recitfies some message format and add field name in error
message when type param check fails.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 		2025-01-24 13:43:06 +08:00
..
allocator enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
broker enhance: [2.5] Read metadata concurrently to accelerate recovery (#38900) 2025-01-16 17:53:01 +08:00
session enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
OWNERS [skip ci]Update OWNERS files (#11898) 2021-11-16 15:41:11 +08:00
README.md [skip ci]Change etcd to lowercase (#9983) 2021-10-15 18:58:37 +08:00
analyze_meta.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
analyze_meta_test.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
build_index_policy.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
channel.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
channel_manager.go fix: [2.5] Fix consume blocked due to too many consumers (#38915) 2025-01-16 15:19:03 +08:00
channel_manager_factory.go fix: drop collection failed if enable streaming service (#37444) 2024-11-07 10:26:26 +08:00
channel_manager_test.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
channel_store.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
channel_store_test.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
cluster.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
cluster_test.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
compaction.go fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490) 2025-01-22 10:27:05 +08:00
compaction_l0_view.go enhance: Add deltaRowCount in l0 compaction (#33997) 2024-06-20 17:46:01 +08:00
compaction_l0_view_test.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
compaction_policy_clustering.go enhance: [2.5] Reduce mutex contention in datacoord meta (#38904) 2025-01-15 00:55:00 +08:00
compaction_policy_clustering_test.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
compaction_policy_l0.go fix: [cp25]Record active collections for l0Policy (#39217) (#39383) 2025-01-20 10:47:03 +08:00
compaction_policy_l0_test.go fix: [cp25]Record active collections for l0Policy (#39217) (#39383) 2025-01-20 10:47:03 +08:00
compaction_policy_single.go enhance: [2.5] Reduce mutex contention in datacoord meta (#38904) 2025-01-15 00:55:00 +08:00
compaction_policy_single_test.go enhance: [2.5] Reduce mutex contention in datacoord meta (#38904) 2025-01-15 00:55:00 +08:00
compaction_queue.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
compaction_queue_test.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
compaction_task.go fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490) 2025-01-22 10:27:05 +08:00
compaction_task_clustering.go fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490) 2025-01-22 10:27:05 +08:00
compaction_task_clustering_test.go fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490) 2025-01-22 10:27:05 +08:00
compaction_task_l0.go fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490) 2025-01-22 10:27:05 +08:00
compaction_task_l0_test.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
compaction_task_meta.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
compaction_task_meta_test.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
compaction_task_mix.go fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490) 2025-01-22 10:27:05 +08:00
compaction_task_mix_test.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
compaction_test.go fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490) 2025-01-22 10:27:05 +08:00
compaction_trigger.go enhance: [2.5] Reduce mutex contention in datacoord meta (#38904) 2025-01-15 00:55:00 +08:00
compaction_trigger_test.go fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490) 2025-01-22 10:27:05 +08:00
compaction_trigger_v2.go fix: [cp25]Record active collections for l0Policy (#39217) (#39383) 2025-01-20 10:47:03 +08:00
compaction_trigger_v2_test.go fix: [cp25]Record active collections for l0Policy (#39217) (#39383) 2025-01-20 10:47:03 +08:00
compaction_view.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
const.go enhance: pass partition key scalar info if enabled when build vector index (#29931) 2024-01-24 00:04:55 +08:00
errors.go Refine DataCoord status (#27262) 2023-09-26 17:15:27 +08:00
errors_test.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
garbage_collector.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
garbage_collector_test.go enhance: [2.5] Read metadata concurrently to accelerate recovery (#38900) 2025-01-16 17:53:01 +08:00
go_channel_singleton.go fix: Fix improper use of offset in HybridSearch (#36244) 2024-09-13 22:05:15 +08:00
handler.go fix: [2.5] Record a map to avoid repeatedly traversing the CompactionFrom (#38926) 2025-01-15 10:31:00 +08:00
handler_test.go fix: [2.5] Record a map to avoid repeatedly traversing the CompactionFrom (#38926) 2025-01-15 10:31:00 +08:00
import_checker.go enhance: [2.5] Limit the maximum number of segments restored and fail the job if saving the binlog fails (#39359) 2025-01-17 10:27:04 +08:00
import_checker_test.go enhance: [2.5] Limit the maximum number of segments restored and fail the job if saving the binlog fails (#39359) 2025-01-17 10:27:04 +08:00
import_job.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
import_meta.go enhance: refine the datacoord meta related interfaces (#37957) 2024-11-26 19:46:34 +08:00
import_meta_test.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
import_scheduler.go enhance: [2.5] Limit the maximum number of segments restored and fail the job if saving the binlog fails (#39359) 2025-01-17 10:27:04 +08:00
import_scheduler_test.go enhance: [2.5] Read metadata concurrently to accelerate recovery (#38900) 2025-01-16 17:53:01 +08:00
import_task.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
import_util.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
import_util_test.go enhance: [2.5] Read metadata concurrently to accelerate recovery (#38900) 2025-01-16 17:53:01 +08:00
index_engine_version_manager.go fix: [2.5] Add scalar index engine version for compatibility (#39236) 2025-01-14 21:01:01 +08:00
index_engine_version_manager_test.go fix: [2.5] Add scalar index engine version for compatibility (#39236) 2025-01-14 21:01:01 +08:00
index_meta.go fix: [2.5] Remove frequently updating metric to avoid mutex contention (#38778) 2025-01-16 11:51:02 +08:00
index_meta_test.go enhance: record memory size (uncompressed) item for index (#38844) 2025-01-14 10:33:06 +08:00
index_service.go enhance: [2.5] Refine error msg for schema & index checking (#39533) (#39565) 2025-01-24 13:43:06 +08:00
index_service_test.go fix: [2.5] deny to set the mmap param for the alter index api (#39520) 2025-01-22 23:55:06 +08:00
job_manager.go fix: [2.5]Restore the compacting state for stats task during recovery (#39460) 2025-01-21 01:03:05 +08:00
job_manager_test.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
knapsack.go enhance: improve mix compaction performance by removing max segment limitations (#38344) 2024-12-11 20:38:42 +08:00
knapsack_test.go enhance: try compact small segments first if they may compose a full segment (#37709) 2024-12-02 16:12:38 +08:00
meta.go fix: [2.5] Set the stating state correctly (#39514) 2025-01-22 18:38:29 +08:00
meta_test.go enhance: [2.5] Read metadata concurrently to accelerate recovery (#38900) 2025-01-16 17:53:01 +08:00
meta_util.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
metrics_info.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
metrics_info_test.go enhance: record memory size (uncompressed) item for index (#38844) 2025-01-14 10:33:06 +08:00
mock_channel_store.go fix: ChannelManager concurret Release and Watch bug (#38590) 2024-12-19 22:50:47 +08:00
mock_channelmanager.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
mock_cluster.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
mock_compaction_meta.go fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490) 2025-01-22 10:27:05 +08:00
mock_compaction_plan_context.go fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490) 2025-01-22 10:27:05 +08:00
mock_handler.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
mock_index_engine_version_manager.go fix: [2.5] Add scalar index engine version for compatibility (#39236) 2025-01-14 21:01:01 +08:00
mock_job_manager.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
mock_segment_manager.go fix: deleted the sealed segment data accidentally (#39422) 2025-01-20 17:49:03 +08:00
mock_subcluster.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
mock_test.go enhance: [2.5] Read metadata concurrently to accelerate recovery (#38900) 2025-01-16 17:53:01 +08:00
mock_trigger_manager.go fix: deleted the sealed segment data accidentally (#39422) 2025-01-20 17:49:03 +08:00
partition_stats_meta.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
partition_stats_meta_test.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
policy.go fix: drop collection failed if enable streaming service (#37444) 2024-11-07 10:26:26 +08:00
policy_test.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
segment_allocation_policy.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
segment_allocation_policy_test.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
segment_info.go fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490) 2025-01-22 10:27:05 +08:00
segment_info_test.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
segment_manager.go fix: deleted the sealed segment data accidentally (#39422) 2025-01-20 17:49:03 +08:00
segment_manager_test.go fix: deleted the sealed segment data accidentally (#39422) 2025-01-20 17:49:03 +08:00
segment_operator.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
segment_operator_test.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
server.go fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490) 2025-01-22 10:27:05 +08:00
server_test.go fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490) 2025-01-22 10:27:05 +08:00
services.go fix: deleted the sealed segment data accidentally (#39422) 2025-01-20 17:49:03 +08:00
services_test.go enhance: record memory size (uncompressed) item for index (#38844) 2025-01-14 10:33:06 +08:00
stats_task_meta.go fix: [2.5]Restore the compacting state for stats task during recovery (#39460) 2025-01-21 01:03:05 +08:00
stats_task_meta_test.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
sync_segments_scheduler.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
sync_segments_scheduler_test.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
task_analyze.go fix: [2.5] Set deltalogs for stats task after set segment stating (#39502) 2025-01-22 16:29:06 +08:00
task_index.go fix: [2.5] Set deltalogs for stats task after set segment stating (#39502) 2025-01-22 16:29:06 +08:00
task_scheduler.go fix: [2.5] Set deltalogs for stats task after set segment stating (#39502) 2025-01-22 16:29:06 +08:00
task_scheduler_test.go fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490) 2025-01-22 10:27:05 +08:00
task_stats.go fix: [2.5] Set deltalogs for stats task after set segment stating (#39502) 2025-01-22 16:29:06 +08:00
task_stats_test.go fix: [2.5] Set deltalogs for stats task after set segment stating (#39502) 2025-01-22 16:29:06 +08:00
types.go fix: [2.5] Set deltalogs for stats task after set segment stating (#39502) 2025-01-22 16:29:06 +08:00
util.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00
util_test.go enhance: make new go package to manage proto (#39128) 2025-01-10 10:53:01 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

  • KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
  • Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
  • Root Coordinator: timestamp, id and meta source.
  • Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.