design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260105-external_table.md
issue: #45881
This change introduces manual refresh capability for external
collections, allowing users to trigger on-demand data synchronization
from external sources. It replaces the legacy update mechanism with a
more robust job-task hierarchy and persistent state management.
Key changes:
- Add RefreshExternalCollection, GetRefreshExternalCollectionProgress,
and ListRefreshExternalCollectionJobs APIs across Client, Proxy,
and DataCoord
- Implement ExternalCollectionRefreshManager to manage refresh jobs
with a 1:N Job-Task hierarchy
- Add ExternalCollectionRefreshMeta for persistent storage of jobs and
tasks in the metastore
- Add ExternalCollectionRefreshChecker for task state management and
worker assignment
- Implement ExternalCollectionRefreshInspector for periodic job
cleanup
- Use WAL Broadcast mechanism for distributed consistency and
idempotency
- Replace legacy external_collection_inspector and update tasks with
the new refresh-based implementation
- Add comprehensive unit tests for refresh job lifecycle and state
transitions
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260105-external_table.md
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
wei liu