3ed3bf92e5
fix: [2.5] Ensure task execution order by using a priority queue ( #43272 )
...
issue: #43260
master pr: #43271
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-07-23 10:16:54 +08:00
e02a3daff5
enhance: [2.5] Optimize datacoord meta mutex ( #40753 )
...
Use a separate collection mutex.
issue: https://github.com/milvus-io/milvus/issues/40551
pr: https://github.com/milvus-io/milvus/pull/40552
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-25 13:50:21 +08:00
0cfe835cac
enhance: [2.5] Support slot for index task and stats task ( #40288 )
...
issue: #39101
master pr: #39084
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-03-19 16:50:11 +08:00
33e9db1539
fix: cherry pick json prs ( #40621 )
...
issue: #35528
pr: #40604
this pr includes some prs:
1. #40604
2. #40611
3. #40619
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-03-13 20:36:10 +08:00
f6c4ad52f0
enhance: [2.5] Refine task meta with key lock ( #40353 )
...
issue: #39101
master pr: #39084
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-03-13 19:18:08 +08:00
2846cc5810
fix: [2.5] Skip executing stats for zero segment ( #40449 )
...
issue : #40241
master pr: #40448
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-03-07 16:16:03 +08:00
f02e549840
fix: compaction task cannot be genereted if size greater than max size ( #40350 )
...
See: #40343
pr: #40348
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-05 00:40:01 +08:00
4a75fdac03
enhance: Remove task meta when task is no need to do ( #40146 )
...
issue: #39911
master pr: #39084
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-02-25 10:35:57 +08:00
709594f158
enhance: [2.5] Use v2 package name for pkg module ( #40117 )
...
Cherry-pick from master
pr: #39990
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-23 00:46:01 +08:00
be30f5301d
enhance: Refine index task scheduler policy ( #40104 )
...
master pr: #39084
issue: #39101
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-02-22 11:25:54 +08:00
22a69b5399
enhance: [2.5]Only check L0 compaction with same channel when stating ( #39543 )
...
issue: #39333
master pr: #39459
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-02-05 17:17:11 +08:00
e46c8ba7fb
fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats ( #39490 )
...
issue: #39333
master pr: #39489
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2025-01-22 10:27:05 +08:00
817b616eb4
fix: [2.5]Restore the compacting state for stats task during recovery ( #39460 )
...
issue: #39333
master pr: #39459
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-01-21 01:03:05 +08:00
adfc3f945e
enhance: record memory size (uncompressed) item for index ( #38844 )
...
issue: #38715
pr: #38770
- Current milvus use a serialized index size(compressed) for estimate
resource for loading.
- Add a new field MemSize (before compressing) for index to estimate
resource.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-14 10:33:06 +08:00
95809ca767
enhance: make new go package to manage proto ( #39128 )
...
issue: #39095
pr: #39114
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:53:01 +08:00
1dbb6cd7cb
enhance: refine the datacoord meta related interfaces ( #37957 )
...
issue: #35917
This PR refines the meta-related APIs in datacoord to allow the ctx to
be passed down to the catalog operation interfaces
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-11-26 19:46:34 +08:00
1d06d4324b
fix: Int64 overflow in JSON encoding ( #37657 )
...
issue: ##36621
- For simple types in a struct, add "string" to the JSON tag for
automatic string conversion during JSON encoding.
- For complex types in a struct, replace "int64" with "string."
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-14 22:52:30 +08:00
9d16b972ea
feat: add tasks page into management WebUI ( #37002 )
...
issue: #36621
1. Add API to access task runtime metrics, including:
- build index task
- compaction task
- import task
- balance (including load/release of segments/channels and some leader
tasks on querycoord)
- sync task
2. Add a debug model to the webpage by using debug=true or debug=false
in the URL query parameters to enable or disable debug mode.
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-28 10:13:29 +08:00
3de57ec4fa
enhance: add vector index mgr to remove vector index type dependency ( #36843 )
...
issue: #34298
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-10-17 22:15:25 +08:00
7bf40694fd
fix: Record the nodeID before assigning tasks ( #36371 )
...
issue: #33744
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-28 17:21:15 +08:00
a61668c77e
feat: Introduce stats task for import ( #35868 )
...
This PR introduce stats task for import:
1. Define new `Stats` and `IndexBuilding` states for importJob
2. Add new stats step to the import process: trigger the stats task and
wait for its completion
3. Abort stats task if import job failed
issue: https://github.com/milvus-io/milvus/issues/33744
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-15 15:17:08 +08:00
8395c8a8db
enhance: Update stats task to optional ( #35947 )
...
issue: #33744
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-12 20:37:08 +08:00
e480b103bd
feat: supporing hybrid search group_by ( #35982 )
...
related: #35096
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-08 17:09:04 +08:00
2c9bb4dfa3
feat: Support stats task to sort segment by PK ( #35054 )
...
issue: #33744
This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-02 14:19:03 +08:00
4236c3bafb
enhance: enable MV on binary vectors ( #35668 )
...
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-08-25 16:27:09 +08:00
582d2eec79
enhance: Move datanode/indexnode manager to session pkg ( #35634 )
...
Related to #28861
Move session manager, worker manager to session package. Also renaming
each manager to corresponding node name(datanode, indexnode).
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-22 16:02:56 +08:00
6542c1ab0e
enhance: Add monitoring metrics for task execution time in datacoord ( #35139 )
...
issue: #35138
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-05 16:26:17 +08:00
aa247f192d
enhance: remove unused code for StorageV2 ( #35132 )
...
issue: https://github.com/milvus-io/milvus/issues/34168
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-08-01 12:08:13 +08:00
acaa78db58
fix: disable sparse and binary in MV ( #35126 )
...
issue: #35120
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 0d340ebe7412cd85963825bc8abd3c874e339679)
2024-07-31 15:02:01 +08:00
104d0966b7
feat: support partition key isolation ( #34336 )
...
issue: #34332
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-07-11 19:01:35 +08:00
be92147723
enhance: Print reason when seal segments ( #34575 )
...
See also #34574
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-11 11:40:51 +08:00
feb13cdf07
enhance: Skip pick worker when task doesn't need to execute actually ( #34348 )
...
issue: #34347
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-03 15:52:09 +08:00
b961767005
enhance: support integral type for MV and skip MV if there is only one category ( #33161 )
...
issue: #29892
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-06-24 10:20:01 +08:00
27cc9f2630
enhance: Support analyze data ( #33651 )
...
issue: #30633
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
2024-06-06 17:37:51 +08:00
cai.zhang
yihao.dai
Bingyi Sun
Ted Xu
congqixia
Zhen Ye
tinswzy
jaime
foxspy
Chun Han
Patrick Weizhi Xu
zhenshan.cao