3b5cbed86a
enhance: add db label for deletion and bulkinsert operation ( #32611 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-26 15:53:29 +08:00
2586c2f1b3
enhance: use WalkWithPrefix api for oss, enable piplined file gc ( #31740 )
...
issue: #19095,#29655,#31718
- Change `ListWithPrefix` to `WalkWithPrefix` of OOS into a pipeline
mode.
- File garbage collection is performed in other goroutine.
- Segment Index Recycle clean index file too.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-25 20:41:27 +08:00
72c172a7d7
enhance: Remove duplicated collectionID label for task latency ( #32308 )
...
`CollectionID` already exists in channel name, so remove it to save
metrics traffic.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-16 18:55:19 +08:00
c012e6786f
feat: support rate limiter based on db and partition levels ( #31070 )
...
issue: https://github.com/milvus-io/milvus/issues/30577
co-author: @jaime0815
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-04-12 16:01:19 +08:00
d4fd6c7283
enhance: add db label on binlog size metrics ( #32003 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-10 21:01:20 +08:00
90bed1caf9
enhance: add the related data size for the read apis ( #31816 )
...
issue: #30436
origin pr: #30438
related pr: #31772
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-10 15:07:17 +08:00
c9faa6d936
enhance: add more metrics for cache and search ( #31777 )
...
issue: #30931
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-10 10:55:17 +08:00
23ba2a5388
enhance: Remove deprecated DataCoordNumStoredRowsCounter metric ( #31778 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-07 00:25:15 +08:00
bd853be8c7
enhance: Add db label for some usual metrics ( #30956 )
...
issue: #31782
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-02 14:27:13 +08:00
1ad5ccc50f
enhance: add rg and db interface for segment and db/rg metric label ( #31715 )
...
issue: #30931
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-01 10:21:21 +08:00
7c7375031d
enhance: Add metrics for task latency in querycoord scheduler ( #31405 )
...
This PR add metrics for task latency in querycoord scheduler, so if any
kind of task stuck, it's easy to figure out by metrics
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-20 19:29:06 +08:00
dc2cdbe387
enhance: add more metrics ( #31271 )
...
/kind improvement
fix : #31272
This pr add more metrics, which are:
- Slow query count, which the duration considered as slow can be
configurable;
- Number of deleted entities;
- Number of entities imported;
- Number of entities per collection;
- Number of loaded entities per collection;
- Number of indexed entities;
- Number of indexed entities, per collection, per index and whether it's
a vetor index;
- Quota states (LongTimeTickDelay, MemoryExhuasted, DiskQuotaExhuasted)
per database;
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-19 15:23:06 +08:00
194a611814
enhance: Add metrics for querycoord current target cp lag ( #31391 )
...
See also #31390
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-19 14:07:05 +08:00
5016031dc7
enhance: add metrics for load segment progress ( #31005 )
...
This PR add metrics for load segment progress:
1. add metrics for load segment/index concurrency
2. add metrics for load index latency
3. change load segment latency's time unit to ms
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-08 15:39:02 +08:00
a434d33e75
feat: Add import scheduler and manager ( #29367 )
...
This PR introduces novel managerial roles for importv2:
1. ImportMeta: To manage all the import tasks;
2. ImportScheduler: To process tasks and modify their states;
3. ImportChecker: To ascertain the completion of all tasks and instigate
relevant operations.
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-01 18:31:02 +08:00
211143c5e6
enhance: add basic information of milvus into metrics ( #29665 )
...
add basic build information and runtime component dependency into
metrics.
issue: #29664
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2024-01-29 15:47:02 +08:00
fd19e419f9
fix: Use size bucket for compacted segment size metric ( #30028 )
...
See also: #29204
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-26 10:53:02 +08:00
0ac37a4a0c
fix: Use longTime bucket for longer latency ( #30175 )
...
See also: #30174
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-22 19:18:55 +08:00
c8a78465b4
enhance: Add a counter monitoring for the rate-limit requests ( #30109 )
...
Add a counter monitoring metric for the ratelimited rpc requests with
labels: proxy nodeID, rpc request type, and state.
issue: https://github.com/milvus-io/milvus/issues/30052
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-21 14:44:59 +08:00
86f48861c1
fix: Add more throughput in related metrics ( #30038 )
...
This PR also fixes bugs in l0 compactor where
l0 results would never be removed from datanode
See also: #30099
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-19 11:34:54 +08:00
9702cef2b5
feat: Support multiple vector search ( #29433 )
...
issue #25639
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-08 15:34:48 +08:00
bebe1baf18
fix: Reigster CompactionTaskNum metrics ( #29640 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-04 17:16:45 +08:00
b12c90af72
enhance:Add upsert vector metrics ( #29226 )
...
add some metrics.
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-03 10:06:48 +08:00
4b406e5973
enhance: Add CompactionTaskNum metrics ( #29518 )
...
See also: #27606
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-28 15:46:55 +08:00
fe04598900
enhance: Add compaction type label to metrics ( #29485 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-27 15:56:48 +08:00
7a6aa8552a
fix: add back existing datanode metrics ( #29360 )
...
See also: #29204
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-22 14:20:43 +08:00
cb0676153f
enhance: add metrics for stopping querynode balance progress ( #29201 )
...
This PR add three metrics to track the stopping balance progress.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-21 10:14:52 +08:00
010d8362ad
fix: datanode L0 segment num wrong ( #29050 )
...
relate: https://github.com/milvus-io/milvus/issues/27675
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-20 11:20:44 +08:00
cbf0f9c527
enhance: change cp metric to absolute unix ts ( #29328 )
...
See also #29327
Change channel checkpoint metrics to unix seconds instead of checkpoint
timestamp lag value
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-20 11:04:45 +08:00
43abe9ce31
enhance: Delete datanode metric after flowgraph released ( #29003 )
...
`DataNodeFlowGraphBufferDataSize` monitoring metric is leaked. This PR
resolve it.
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-12-07 17:00:35 +08:00
9b3e06ae86
enhance: add more metrics for level zero segments ( #29029 )
...
- Add SegmentNum metric for level zero segments
- Add level zero segments size metirc
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-07 14:34:35 +08:00
dd682658da
enhance: Add channel checkpoint ts lag metric ( #28767 )
...
See also #28765
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-28 07:00:28 +08:00
8a4cfb7d6a
enhance: add l0 metric and fix datacoord no need drop l0 segment when flush ( #28373 )
...
relate: https://github.com/milvus-io/milvus/issues/27675
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-24 15:58:24 +08:00
1d76565894
Add metrics for garbage collection ( #27303 )
...
Also fix second metrics usage in compaction
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-22 18:47:25 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
ea2b909c6d
Fix historm metrics buckets range for index ( #26976 )
...
See also: #26743
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-13 17:21:18 +08:00
791e6ef6c6
[Improvement] add pulsar metrics and fix timeout ( #26907 )
...
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2023-09-13 12:03:19 +08:00
c6024a32f5
Add rate limit ratio monitoring ( #26709 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-03 20:47:02 +08:00
ef75784715
Fix LoadSegmentLatency metric p99 ( #26761 )
...
See also: #26743
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-01 11:07:07 +08:00
8d54509e54
Fix CompactionLatency metrics ( #26747 )
...
- Refine compactor logs
See also: #26743
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-08-31 17:35:03 +08:00
d343888f3c
Fix pkg codecov & static-check script ( #26720 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-31 09:59:01 +08:00
3c62acde05
Add coll and segState lebels to DataCoordNumStoredRows monitoring ( #26299 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-08-17 15:52:17 +08:00
8b54914287
Add go.mod file for the milvus/pkg package ( #26359 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-08-16 14:49:32 +08:00
1971d98897
Add disk metric info ( #25675 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-08-11 10:35:42 +08:00
05c87af146
fix unexpected score in look aside balancer ( #26213 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-10 19:45:16 +08:00
47392b0a0f
support metrics mutex to monitor cost of locks( #26102 ) ( #26103 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-08-03 15:03:06 +08:00
b5e79e7f34
add metrics for assign-segment on proxy( #24570 ) ( #25907 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-07-26 14:15:01 +08:00
302897f866
refine look aside balance logic ( #25837 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-25 18:51:01 +08:00
2ff469771c
fix delegator insert/delete cost metrics ( #25871 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-25 10:01:00 +08:00
32827f538a
add metrics for delegator insert/delete cost ( #25733 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-21 15:30:59 +08:00
jaime
chyezh
congqixia
SimFG
yihao.dai
wei liu
Jiquan Long
XuanYang-cn
xige-16
smellthemoon
aoiasd
yah01
chyezh
MrPresent-Han