996b79c76c
enhance: Add channelCPs in FlushResponse ( #32683 )
...
#32609
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-05-10 09:55:31 +08:00
d9ac8e9e36
fix: [2.3] Mark channel checkpoint dropped prevent cp lag metrics leakage ( #32454 )
...
See also #31506 #31508
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-19 14:55:22 +08:00
ab059bb064
enhance: add more metrics ( #31271 ) ( #31511 )
...
/kind improvement
pr: #31271
fix: https://github.com/milvus-io/milvus/issues/31272
This pr add more metrics, which are:
Slow query count, which the duration considered as slow can be
configurable;
Number of deleted entities;
Number of entities per collection;
Number of loaded entities per collection;
Number of indexed entities;
Number of indexed entities, per collection, per index and whether it's a
vetor index;
Quota states (LongTimeTickDelay, MemoryExhuasted, DiskQuotaExhuasted)
per database;
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-22 16:11:07 +08:00
095c94305c
fix: add GetSegments optimization to avoid meta mutex competition ( #31026 )
...
pr: #31025
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-03-05 14:49:01 +08:00
91d17870d6
enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs ( #30941 ) ( #31024 )
...
This PR includes the following adjustments:
1. To prevent channelCP update task backlog, only one task with the same
vchannel is retained in the updater. Additionally, the lastUpdateTime is
refreshed after the flowgraph submits the update task, rather than in
the callBack function.
2. Batch updates of multiple vchannel checkpoints are performed in the
UpdateChannelCheckpoint RPC (default batch size is 128). Additionally,
the lock for channelCPs in DataCoord meta has been switched from key
lock to global lock.
3. The concurrency of UpdateChannelCheckpoint RPCs in the datanode has
been reduced from 1000 to 10.
issue: https://github.com/milvus-io/milvus/issues/30004
pr: https://github.com/milvus-io/milvus/pull/30941
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-05 14:27:01 +08:00
336e0ae45e
enhance: index meta use independent rather than global meta lock ( #30986 )
...
issue: https://github.com/milvus-io/milvus/issues/30837
pr: #30869
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-03-05 08:48:59 +08:00
df09222029
fix: starve lock caused by slow GetCompactionTo method when too much segments ( #30965 )
...
issue: #30823
pr: #30963
Signed-off-by: chyezh <chyezh@outlook.com>
2024-03-04 20:51:00 +08:00
38e3d6af3e
enhance: Optimize DescribeIndex to reduce lock contention ( #30975 )
...
issue: #29313
issue: #30443
master pr: #30939
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-04 11:30:59 +08:00
ef086dc0ca
fix: [Pick] Skip filling segmentID in indexBuildCh to prevent flush blocked ( #30749 )
...
issue: #30580
master pr: #30747
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-02-22 20:42:56 +08:00
98aae10273
fix: compact operation on datacoord meta should preform as a transcation ( #29776 )
...
issue: #29691
pr: #29775
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2024-01-12 14:54:52 +08:00
5ba0f476d5
fix: [2.3]parse logID from logPath if copyDeltalog find logID not provided ( #29276 )
...
Cherry-pick from master
pr: #29273
See also: #29272
This PR add `getDeltaLogID` to safely return logID when Binlog struct
has zero value logID. It parses logID from logPath if the format is
valid. Otherwise, this function shall return error.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-27 14:42:46 +08:00
9acf32a0b7
enhance: [cherry-pick] change cp metric to absolute unix ts ( #29328 ) ( #29337 )
...
Cherry pick from master
pr: #29328
See also #29327
Change channel checkpoint metrics to unix seconds instead of checkpoint
timestamp lag value
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-20 15:04:42 +08:00
0688ec4d83
enhance: [cherry-pick] Add channel checkpoint ts lag metric ( #28767 ) ( #28776 )
...
Cherry-pick from master
pr: #28767
See also #28765
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-28 19:28:28 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
4f6122dd4a
enable ctx traceId for assignsegment on dc( #26972 ) ( #27108 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-09-18 09:53:22 +08:00
d7cd1f2a6d
Fix bug for get index state with compaction segment ( #26822 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-09-04 17:29:48 +08:00
bed034a44b
Protect DataCoord from calculating segment lines by stale log entries num ( #26523 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-08-22 14:16:22 +08:00
3c62acde05
Add coll and segState lebels to DataCoordNumStoredRows monitoring ( #26299 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-08-17 15:52:17 +08:00
7619db1edc
Fast GC compacted dropped segment ( #26207 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-08-09 19:39:15 +08:00
b533c68632
Forbid update checkpoint without msgID ( #25694 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-07-25 10:43:04 +08:00
f4e72cb170
remove sync segmentLastExpire every time when assigning( #25271 ) ( #25316 ) ( #25557 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-07-24 14:11:07 +08:00
95f91e6e35
Fix segments meta not consistent between catalog and in-memory ( #25799 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: czs007 <zhenshan.cao@zilliz.com>
2023-07-21 14:21:05 +08:00
bccdef1ad7
Fix segment meta only record last flushed binlog ( #25707 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-07-19 15:28:57 +08:00
66fdc71479
Refactor logs in DataCoord & DataNode ( #25574 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-07-14 15:56:31 +08:00
b22d6d2a5d
Avoid update all binlogs every a segment is updated ( #25526 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-13 14:08:29 +08:00
54acb5b725
Remove alter segment interface ( #25382 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-06 20:46:25 +08:00
fe22720ff3
Avoid rewriting all binlogs every time a segment is updated ( #25339 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-06 18:20:26 +08:00
33c2012675
Add more metrics ( #25081 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-26 17:52:44 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
19164ea753
Make binlog size metric to collect healthy binlog only ( #24557 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-31 16:45:29 +08:00
fc010ee86b
Improve the metric of stored binlog size to the level of collection and segment ( #23859 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-07 20:52:38 +08:00
b517bc9e6a
refine balance mechanism including:( #23454 ) ( #23763 ) ( #23791 )
...
1. balance granuity to replica to avoid influence unrelated replicas
2. avoid balance back and forth
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
2023-05-04 12:22:40 +08:00
ed8836cd15
Add disk quota at the collection level ( #23704 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-04-26 21:52:36 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
1287ca699a
Refine usage of TimeRecorder.Record ( #23142 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-03-30 18:56:22 +08:00
4a90490a67
Fix the `segment not found` error ( #22772 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-03-17 17:27:56 +08:00
aa6212d5b0
[Cherry-pick] Fix channel checkpoint issues and fix GC ( #22559 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: Ten Thousand Leaves <69466447+soothing-rain@users.noreply.github.com>
2023-03-09 14:13:52 +08:00
efdc77bce6
Fix not fill timestamp when use collection created ts ( #22629 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-08 18:53:51 +08:00
d126f06946
Decouple mq module from internal proto definition ( #22536 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-03-04 23:21:50 +08:00
319f1773af
Fix segment not found if forward delete to empty segment ( #22528 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-03 14:13:49 +08:00
3d6bded115
Fix Row Number Mismatch ( #22307 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-02-27 10:41:46 +08:00
fef12b53ef
Fix 2.2.2 possible broken channel cp ( #22205 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-02-16 16:46:35 +08:00
f9495fef00
Minor fix on segment metric ( #21976 )
...
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2023-02-03 19:33:52 +08:00
f1daef22a1
Fix failed bulkload segment marked as sealed ( #21799 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-31 12:41:53 +08:00
9fd9bed2b9
Remove collection meta after GC finished ( #21595 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-01-12 09:55:42 +08:00
8c2c04517e
Fix an issue where flushed segments can turn into sealed ( #21566 )
...
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2023-01-11 18:57:43 +08:00
73d8509600
Improve ListSegments performance for datacoord catalog ( #21477 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-01-06 14:33:36 +08:00
e5f408dceb
Merge IndexCoord and DataCoord ( #21267 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-04 19:37:36 +08:00
5d044c4f29
Add check for stats log and bin log number ( #21264 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-12-26 16:35:30 +08:00
89b810a4db
Refactor all params into ParamItem ( #20987 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
wayblink
congqixia
Jiquan Long
zhagnlu
yihao.dai
jaime
chyezh
cai.zhang
chyezh
SimFG
MrPresent-Han
yah01
zhenshan.cao
aoiasd
Enwei Jiao
Bingyi Sun
xige-16
Xiaofan
Ten Thousand Leaves