milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
milvus/internal/datacoord
History
Patrick Weizhi Xu 52ae47c850
enhance: gather materialized view search info once per request (#31996) 
issue: #29892 

This PR:
1. Move the process of gathering materialized search info to when the
search plan is created, before it goes to each segment, to avoid
repeated work and access the plan node under multi-threaded
circumstances.
2. Enforce the supported MV type to `VARCHAR`
3. Add integration test

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 		2024-04-11 15:21:19 +08:00
..
broker enhance: index meta use independent rather than global meta lock (#30869) 2024-03-04 16:56:59 +08:00
OWNERS [skip ci]Update OWNERS files (#11898) 2021-11-16 15:41:11 +08:00
README.md [skip ci]Change etcd to lowercase (#9983) 2021-10-15 18:58:37 +08:00
allocator.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
allocator_test.go Decoupling client and server API in types interface (#27186) 2023-09-26 09:57:25 +08:00
build_index_policy.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
channel.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
channel_checker.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
channel_checker_test.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
channel_manager.go enhance:change some logs (#29579) 2024-01-05 16:12:48 +08:00
channel_manager_factory.go enhance: Remove not in use policies (#29448) 2023-12-28 10:38:46 +08:00
channel_manager_test.go enhance: Change ChannelManager to interface (#29300) 2023-12-25 19:24:46 +08:00
channel_store.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
channel_store_test.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
cluster.go enhance: Remove import v1 (#31403) 2024-03-22 15:29:09 +08:00
cluster_test.go enhance: Remove import v1 (#31403) 2024-03-22 15:29:09 +08:00
compaction.go fix: [cherry-pick]Skip changing meta if nodeID not match with channel (#31672) 2024-04-10 15:09:18 +08:00
compaction_l0_view.go fix: Trigger l0 compaction when l0 views don't change (#30729) 2024-03-05 16:37:00 +08:00
compaction_l0_view_test.go enhance: Add force trigger (#30641) 2024-02-19 18:40:50 +08:00
compaction_scheduler.go fix: Trigger l0 compaction when l0 views don't change (#30729) 2024-03-05 16:37:00 +08:00
compaction_scheduler_test.go enhance: Add CompactionTaskNum metrics (#29518) 2023-12-28 15:46:55 +08:00
compaction_test.go fix: [cherry-pick]Skip changing meta if nodeID not match with channel (#31672) 2024-04-10 15:09:18 +08:00
compaction_trigger.go feat: Add global mmap enable configuration (#31267) 2024-03-18 15:17:10 +08:00
compaction_trigger_test.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
compaction_trigger_v2.go fix: Skip to submit l0 tasks when scheduler full (#31270) 2024-03-15 10:21:12 +08:00
compaction_trigger_v2_test.go fix: Skip to submit l0 tasks when scheduler full (#31270) 2024-03-15 10:21:12 +08:00
compaction_view.go fix: Trigger l0 compaction when l0 views don't change (#30729) 2024-03-05 16:37:00 +08:00
compaction_view_manager.go fix: Correct the last empty l0 views (#31198) 2024-03-14 10:31:04 +08:00
compaction_view_manager_test.go fix: Correct the last empty l0 views (#31198) 2024-03-14 10:31:04 +08:00
const.go enhance: pass partition key scalar info if enabled when build vector index (#29931) 2024-01-24 00:04:55 +08:00
errors.go Refine DataCoord status (#27262) 2023-09-26 17:15:27 +08:00
errors_test.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
garbage_collector.go fix: Use correct ticker for gc work defer (#31456) 2024-03-20 22:17:06 +08:00
garbage_collector_test.go enhance: Remove import v1 (#31403) 2024-03-22 15:29:09 +08:00
handler.go enhance: Add AllPartitionsID const to replace InvalidPartitionID (#31438) 2024-03-20 19:01:05 +08:00
import_checker.go enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629) 2024-04-01 20:09:13 +08:00
import_checker_test.go enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629) 2024-04-01 20:09:13 +08:00
import_job.go enhance: Return more fields in import progress response (#31539) 2024-03-24 21:57:06 +08:00
import_meta.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
import_meta_test.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
import_scheduler.go enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629) 2024-04-01 20:09:13 +08:00
import_scheduler_test.go enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629) 2024-04-01 20:09:13 +08:00
import_task.go enhance: Enhance and modify the return content of ImportV2 (#31192) 2024-03-13 19:51:03 +08:00
import_util.go enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629) 2024-04-01 20:09:13 +08:00
import_util_test.go enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629) 2024-04-01 20:09:13 +08:00
index_builder.go enhance: gather materialized view search info once per request (#31996) 2024-04-11 15:21:19 +08:00
index_builder_test.go enhance: gather materialized view search info once per request (#31996) 2024-04-11 15:21:19 +08:00
index_engine_version_manager.go feat: trigger compaction to handle index version (#28442) 2023-11-21 09:26:22 +08:00
index_engine_version_manager_test.go Optimize the codec code of session (#27360) 2023-10-01 10:33:30 +08:00
index_meta.go enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629) 2024-04-01 20:09:13 +08:00
index_meta_test.go enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629) 2024-04-01 20:09:13 +08:00
index_service.go feat: Add global mmap enable configuration (#31267) 2024-03-18 15:17:10 +08:00
index_service_test.go enhance: Add `ListIndexes` API from datacoord (#31104) 2024-03-07 17:37:01 +08:00
indexnode_manager.go Decoupling client and server API in types interface (#27186) 2023-09-26 09:57:25 +08:00
indexnode_manager_test.go Refine state check (#27541) 2023-10-11 21:01:35 +08:00
meta.go enhance: add db label on binlog size metrics (#32003) 2024-04-10 21:01:20 +08:00
meta_test.go enhance: add db label on binlog size metrics (#32003) 2024-04-10 21:01:20 +08:00
meta_util.go Make VChannelInfo read compatible with 2.0.2 (#17995) 2022-07-07 21:58:22 +08:00
metrics_info.go enhance: add the related data size for the read apis (#31816) 2024-04-10 15:07:17 +08:00
metrics_info_test.go Refine state check (#27541) 2023-10-11 21:01:35 +08:00
mock_allocator_test.go enhance: index meta use independent rather than global meta lock (#30869) 2024-03-04 16:56:59 +08:00
mock_channel_store.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
mock_channelmanager.go enhance: Add CompactionTaskNum metrics (#29518) 2023-12-28 15:46:55 +08:00
mock_cluster.go fix: add lazy load field to mark segment load type (#31591) 2024-03-28 11:23:10 +08:00
mock_compaction_meta.go enhance: Refine compaction interfaces to support major compaction (#30632) 2024-02-19 20:52:50 +08:00
mock_compaction_plan_context.go fix: Separate schedule and check results loop (#28692) 2023-11-29 10:50:29 +08:00
mock_handler.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
mock_index_engine_version_manager.go feat: trigger compaction to handle index version (#28442) 2023-11-21 09:26:22 +08:00
mock_scheduler.go enhance: Add CompactionTaskNum metrics (#29518) 2023-12-28 15:46:55 +08:00
mock_segment_manager.go enhance: index meta use independent rather than global meta lock (#30869) 2024-03-04 16:56:59 +08:00
mock_session_manager.go fix: add lazy load field to mark segment load type (#31591) 2024-03-28 11:23:10 +08:00
mock_test.go enhance: Remove import v1 (#31403) 2024-03-22 15:29:09 +08:00
mock_trigger_manager.go feat: Add Compaction views and triggers (#27906) 2023-11-23 17:30:25 +08:00
policy.go enhance: Remove not in use policies (#29448) 2023-12-28 10:38:46 +08:00
policy_test.go enhance: Remove not in use policies (#29448) 2023-12-28 10:38:46 +08:00
segment_allocation_policy.go fix: Decoupling importing segment from flush process (#30402) 2024-02-03 13:01:12 +08:00
segment_allocation_policy_test.go Adjust datacoord for L0 Delta (#28021) 2023-11-06 15:26:16 +08:00
segment_info.go enhance: Remove import v1 (#31403) 2024-03-22 15:29:09 +08:00
segment_info_test.go fix: starve lock caused by slow GetCompactionTo method when too much segments (#30963) 2024-03-05 10:04:59 +08:00
segment_manager.go enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629) 2024-04-01 20:09:13 +08:00
segment_manager_test.go feat: add more operation detail info for better allocation (#30438) 2024-03-28 06:33:11 +08:00
segment_operator.go fix: Prevent clone when selecting segments from meta (#30928) 2024-03-01 13:59:02 +08:00
segment_operator_test.go fix: Prevent clone when selecting segments from meta (#30928) 2024-03-01 13:59:02 +08:00
server.go fix: Make coordinator `Register` not blocked on ProcessActiveStandby (#32069) 2024-04-10 18:49:18 +08:00
server_test.go fix: Make coordinator `Register` not blocked on ProcessActiveStandby (#32069) 2024-04-10 18:49:18 +08:00
services.go enhance: Remove the storage info report (#31772) 2024-04-01 20:50:59 -07:00
services_test.go enhance: Remove import v1 (#31403) 2024-03-22 15:29:09 +08:00
session.go Decoupling client and server API in types interface (#27186) 2023-09-26 09:57:25 +08:00
session_manager.go feat: add more operation detail info for better allocation (#30438) 2024-03-28 06:33:11 +08:00
session_manager_test.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
util.go enhance: gather materialized view search info once per request (#31996) 2024-04-11 15:21:19 +08:00
util_test.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

  • KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
  • Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
  • Root Coordinator: timestamp, id and meta source.
  • Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.