3ed3bf92e5
fix: [2.5] Ensure task execution order by using a priority queue ( #43272 )
...
issue: #43260
master pr: #43271
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-07-23 10:16:54 +08:00
a0a6510db9
fix: [2.5] Only mark segment compacting for sort stats task ( #42517 )
...
issue: #42506
master pr: #42516
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-06-04 21:40:32 +08:00
97fc68bf08
fix: [2.5] Set task queue time before enqueue ( #40815 ) ( #40853 )
...
Cherry-pick from master
pr: #40815
Related to #40814
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-25 14:02:22 +08:00
d703d8dac8
fix: [2.5] Set correct default configuration for task slots ( #40821 )
...
issue: #39101
master pr: #39084
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-03-23 21:28:19 +08:00
0cfe835cac
enhance: [2.5] Support slot for index task and stats task ( #40288 )
...
issue: #39101
master pr: #39084
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-03-19 16:50:11 +08:00
f6c4ad52f0
enhance: [2.5] Refine task meta with key lock ( #40353 )
...
issue: #39101
master pr: #39084
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-03-13 19:18:08 +08:00
4a75fdac03
enhance: Remove task meta when task is no need to do ( #40146 )
...
issue: #39911
master pr: #39084
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-02-25 10:35:57 +08:00
6c81463263
fix: [2.5] Fix task scheduler dead lock ( #40121 )
...
issue: #39101
master pr: #39084
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2025-02-23 18:29:55 +08:00
9cbab98382
enhance: [2.5] Export request timeout interval in config ( #40118 )
2025-02-23 13:47:12 +08:00
709594f158
enhance: [2.5] Use v2 package name for pkg module ( #40117 )
...
Cherry-pick from master
pr: #39990
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-23 00:46:01 +08:00
be30f5301d
enhance: Refine index task scheduler policy ( #40104 )
...
master pr: #39084
issue: #39101
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-02-22 11:25:54 +08:00
52434ccc78
enhance: [2.5] Limit the speed of the generating stats task ( #39645 )
...
master pr: #39644
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-02-17 16:06:17 +08:00
ddc5b299ad
enhance: expose more metrics data ( #39466 )
...
issue: #36621 #39417
pr: #39456
1. Adjust the server-side cache size.
2. Add source information for configurations.
3. Add node ID for compaction and indexing tasks.
4. Resolve localhost access issues to fix health check failures for
etcd.
Signed-off-by: jaime <yun.zhang@zilliz.com>
2025-02-07 11:48:45 +08:00
22a69b5399
enhance: [2.5]Only check L0 compaction with same channel when stating ( #39543 )
...
issue: #39333
master pr: #39459
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-02-05 17:17:11 +08:00
cbf1161177
fix: [2.5] Set deltalogs for stats task after set segment stating ( #39502 )
...
issue: #39333
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-01-22 16:29:06 +08:00
e46c8ba7fb
fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats ( #39490 )
...
issue: #39333
master pr: #39489
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2025-01-22 10:27:05 +08:00
817b616eb4
fix: [2.5]Restore the compacting state for stats task during recovery ( #39460 )
...
issue: #39333
master pr: #39459
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-01-21 01:03:05 +08:00
95809ca767
enhance: make new go package to manage proto ( #39128 )
...
issue: #39095
pr: #39114
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:53:01 +08:00
9be106dedf
enhance: Refine task scheduler logs ( #38334 )
...
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-12-11 15:00:44 +08:00
b6612e02b4
enhance: Reduce GetIndexInfos calls ( #37695 )
...
Batch `GetIndexInfos` calls for segments to reduce RPC calls.
issue: https://github.com/milvus-io/milvus/issues/37634
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-19 14:24:31 +08:00
1e8ea4a7e7
feat: add segment/channel/task/slow query render ( #37561 )
...
issue: #36621
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-12 17:44:29 +08:00
9d16b972ea
feat: add tasks page into management WebUI ( #37002 )
...
issue: #36621
1. Add API to access task runtime metrics, including:
- build index task
- compaction task
- import task
- balance (including load/release of segments/channels and some leader
tasks on querycoord)
- sync task
2. Add a debug model to the webpage by using debug=true or debug=false
in the URL query parameters to enable or disable debug mode.
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-28 10:13:29 +08:00
7bf40694fd
fix: Record the nodeID before assigning tasks ( #36371 )
...
issue: #33744
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-28 17:21:15 +08:00
8395c8a8db
enhance: Update stats task to optional ( #35947 )
...
issue: #33744
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-12 20:37:08 +08:00
2c9bb4dfa3
feat: Support stats task to sort segment by PK ( #35054 )
...
issue: #33744
This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-02 14:19:03 +08:00
582d2eec79
enhance: Move datanode/indexnode manager to session pkg ( #35634 )
...
Related to #28861
Move session manager, worker manager to session package. Also renaming
each manager to corresponding node name(datanode, indexnode).
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-22 16:02:56 +08:00
6542c1ab0e
enhance: Add monitoring metrics for task execution time in datacoord ( #35139 )
...
issue: #35138
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-05 16:26:17 +08:00
feb13cdf07
enhance: Skip pick worker when task doesn't need to execute actually ( #34348 )
...
issue: #34347
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-03 15:52:09 +08:00
27cc9f2630
enhance: Support analyze data ( #33651 )
...
issue: #30633
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
2024-06-06 17:37:51 +08:00
cai.zhang
congqixia
jaime
Zhen Ye
yihao.dai