Cherry-pick from master
pr: #31363
See also #31362
This PR make datacoord garbage collection scan operation using differet
interval than other opeartion.
This interval is a newly added param item, which default value is 7*24
hours.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry-pick from master
pr: #32069
See also #32066
This PR make coordinator register successful and let
`ProcessActiveStandBy` run async. And roles may receive stop signal and
notify servers.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #31662#31409
pr: #31723
during FilterIndexedSegment in GetRecoveryInfo, it try to acquire index
meta's read lock for every segment. when a collection has thousands of
segments, which may blocked for more than 10 seconds and even longer.
cause `AddSegmentIndex` may also triggered frequently, which try to get
the write lock.
This PR avoid acquire index meta's lock for each segment
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Cherry-pick from master
pr: #31473
See also #31470#31506
This PR adds nodeID assignment verification before updating channel
checkpoints.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
/kind improvement
pr: #31271
fix: https://github.com/milvus-io/milvus/issues/31272
This pr add more metrics, which are:
Slow query count, which the duration considered as slow can be
configurable;
Number of deleted entities;
Number of entities per collection;
Number of loaded entities per collection;
Number of indexed entities;
Number of indexed entities, per collection, per index and whether it's a
vetor index;
Quota states (LongTimeTickDelay, MemoryExhuasted, DiskQuotaExhuasted)
per database;
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Cherry-pick from master
pr: #31104
See also #31103
This PR add `listIndexes` API for datacoor server to list all indexes
for provided collection.
Comparing to the existing `DescribeIndex` API, the new one does NOT
check the segment index building progress to ease the burden when
invoking it
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
This PR includes the following adjustments:
1. To prevent channelCP update task backlog, only one task with the same
vchannel is retained in the updater. Additionally, the lastUpdateTime is
refreshed after the flowgraph submits the update task, rather than in
the callBack function.
2. Batch updates of multiple vchannel checkpoints are performed in the
UpdateChannelCheckpoint RPC (default batch size is 128). Additionally,
the lock for channelCPs in DataCoord meta has been switched from key
lock to global lock.
3. The concurrency of UpdateChannelCheckpoint RPCs in the datanode has
been reduced from 1000 to 10.
issue: https://github.com/milvus-io/milvus/issues/30004
pr: https://github.com/milvus-io/milvus/pull/30941
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
If DC restarted, those unkonwn compaction tasks
will never get call back in DN, so that the segments in the compaction
task will be locked, unable to sync and compaction again, blocking cp
advance and compaction executing.
See also: #30137
pr: #30850
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
This pr decoups importing segment from flush process by:
1. Exclude the importing segment from the flush policy, this approch
avoids notifying the datanode to flush the importing segment, which may
not exist.
2. When RootCoord call Flush, DataCoord directly set the importing
segment state to `Flushed`.
issue: https://github.com/milvus-io/milvus/issues/30359
pr: https://github.com/milvus-io/milvus/pull/30402
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
1. add coordinator and proxy graceful stop timeout to 5s.
3. add other work node graceful stop timeout to 900s, and we should
potentially change this to 600s when graceful stop is smooth
4. change the order of datacoord component while stop.
5. `LivenessCheck` do not perform graceful shutdown now.
issue: https://github.com/milvus-io/milvus/issues/30310
pr: #30317
also see: https://github.com/milvus-io/milvus/pull/30306
---------
Signed-off-by: chyezh <chyezh@outlook.com>
patch search cache param from index configs when index meta could not
get the search cache size key
issue: #30113
pr: #30119
Signed-off-by: xianliang <xianliang.li@zilliz.com>
pr: #29856
If segment has more than 128 log fils, drop segment will exceed etcd txn
ops limit, which will failed the drop segment request
This PR drop segment meta info with prefix, to avoid drop segment meta
failed
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Cherry-pick from master
pr: #29273
See also: #29272
This PR add `getDeltaLogID` to safely return logID when Binlog struct
has zero value logID. It parses logID from logPath if the format is
valid. Otherwise, this function shall return error.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry pick from master
pr: #29328
See also #29327
Change channel checkpoint metrics to unix seconds instead of checkpoint
timestamp lag value
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Gao
chyezh
cai.zhang
xige-16
wayblink
Jiquan Long
congqixia
wei liu
cqy123456
XuanYang-cn
groot
jaime
zhagnlu
yihao.dai
foxspy
chyezh
SimFG
aoiasd