Commit Graph

57 Commits (eb046863485fdf3e130fc60484485c901b81276b)

Author SHA1 Message Date
wei liu 69b8b89369
enhance: Remove QueryCoord's scheduling of L0 segments (#39552)
issue: #39551
This PR remove querycoord's scheduling of l0 segments:
  - only load l0 segment when watch channel
- only release l0 segment when release channel or sync data distribution

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-02-26 21:38:00 +08:00
congqixia cb7f2fa6fd
enhance: Use v2 package name for pkg module (#39990)
Related to #39095

https://go.dev/doc/modules/version-numbers

Update pkg version according to golang dep version convention

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
Zhen Ye bb8d1ab3bf
enhance: make new go package to manage proto (#39114)
issue: #39095

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:49:01 +08:00
tinswzy 27229f7907
enhance: refine exists log print with ctx (#38080)
issue: #35917 
Refines exists log print with ctx

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-14 22:36:44 +08:00
wei liu 108434969f
enhance: Add collection id to search request count metrics (#38069)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-29 17:28:38 +08:00
cai.zhang 14e007d6fb
fix: Fix the bug that retrieved from wrong field for L0 segments (#37598)
issue: #37574

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-12 20:58:29 +08:00
congqixia caeab0cc1f
enhance: Fill start pos & level for growing segment (#36888)
Start position & level info is missing for growing segment loaded in
watch dml channel operation.

Level is important for metrics and start position is crucial for growing
exclude logic.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-16 14:03:31 +08:00
aoiasd db34572c56
feat: support load and query with bm25 metric (#36071)
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-11 10:23:20 +08:00
aoiasd c22a2cebb6
fix: split stream query result to avoid grpc response too large error (#36090)
relate: https://github.com/milvus-io/milvus/issues/36089

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-13 15:07:09 +08:00
Chun Han e480b103bd
feat: supporing hybrid search group_by (#35982)
related: #35096

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-08 17:09:04 +08:00
Chun Han bfd9d86fe9
feat: support groupby size on go-layer(#33544) (#33845)
related: #33544

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-08-27 14:21:00 +08:00
aoiasd 59a7a46904
enhance: Merge query stream result for reduce delete task (#32855)
relate: https://github.com/milvus-io/milvus/issues/32854

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-27 18:15:43 +08:00
Bingyi Sun fecd9c21ba
feat: LRU cache implementation (#32567)
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.

Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2024-05-06 20:29:30 +08:00
wei liu 1a98ce39f5
enhance: Remove useless logic about FromShardLeader (#32029)
issue: #32047

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-10 20:11:19 +08:00
zhenshan.cao 089c805e0a
enhance:Refactor hybrid search (#32020)
issue: https://github.com/milvus-io/milvus/issues/25639
https://github.com/milvus-io/milvus/issues/31368

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-04-09 14:21:18 +08:00
wei liu c311932d5f
fix: Update segment's version in leader task (#31643)
issue: #31468

1. when segment's version in leader view doesn't match segment's version
in dist, should update leader view
2. after call loadDeltalog, should update segment's load version with
latest ts
3. change leader task's priority from high to low, to avoid leader task
replace segment task and balance task

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-01 10:37:21 +08:00
yiwangdr c6665c2a4c
test: support multiple data/querynodes in integration test (#30618)
issue: https://github.com/milvus-io/milvus/issues/29507

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-02-21 11:54:53 +08:00
xige-16 060c8603a3
fix: Support mvcc with hybrid serach (#30114)
issue: https://github.com/milvus-io/milvus/issues/29656
/kind bug

Signed-off-by: xige-16 <xi.ge@zilliz.com>

---------

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-02-01 16:03:03 +08:00
zhenshan.cao 60e88fb833
fix: Restore the MVCC functionality. (#29749)
When the TimeTravel functionality was previously removed, it
inadvertently affected the MVCC functionality within the system. This PR
aims to reintroduce the internal MVCC functionality as follows:

1. Add MvccTimestamp to the requests of Search/Query and the results of
Search internally.
2. When the delegator receives a Query/Search request and there is no
MVCC timestamp set in the request, set the delegator's current tsafe as
the MVCC timestamp of the request. If the request already has an MVCC
timestamp, do not modify it.
3. When the Proxy handles Search and triggers the second phase ReQuery,
divide the ReQuery into different shards and pass the MVCC timestamp to
the corresponding Query requests.

issue: #29656

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-01-09 11:38:48 +08:00
yah01 c4dda3c50f
fix: panic caused by type assert LocalSegment on Segment (#29018)
related #29017

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-08 01:50:38 +08:00
yihao.dai 4b8bc2798e
enhance: Print nq (#28507)
Log nq in search path.

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-11-30 10:42:27 +08:00
Gao 3e77365de5
fix: correct autoindex segment num (#28387)
Fix #28386 
Current code snippet 
```
// get delegator
sd, ok := node.delegators.Get(channel)
if !ok {
err := merr.WrapErrChannelNotFound(channel)
log.Warn("Query failed, failed to get shard delegator for search", zap.Error(err))
return nil, err
}
req, err = node.optimizeSearchParams(ctx, req, sd)
if err != nil {
log.Warn("failed to optimize search params", zap.Error(err))
return nil, err
}
// do search
results, err := sd.Search(searchCtx, req)
```

We could move these into `ShardDelegator`, and directly use sealed
segment num in `Search` methods, also segment num got outside could be
wrong when we specify partitions.

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-11-22 11:12:22 +08:00
yah01 ece592a42f
Deliver L0 segments delete records (#27722)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-07 01:44:18 +08:00
wei liu 9b737b77a7
fix fromShardLeader typo (#28061)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-02 15:50:16 +08:00
aoiasd 9ba79a97e4
Add some log for delete task when it need query (#27317)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-10-24 14:54:12 +08:00
yah01 be980fbc38
Refine state check (#27541)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-11 21:01:35 +08:00
yah01 6539a5ae2c
Refine DataCoord status (#27262)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-26 17:15:27 +08:00
yah01 00c65fa0d7
Refine QueryNode errors (#27013)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-12 16:07:18 +08:00
aoiasd e107d0794c
support complex delete expression (#25752)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-09-12 10:19:17 +08:00
SimFG 28681276e2
Improve the retry of the rpc client (#26795)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-06 17:43:14 +08:00
yah01 3349db4aa7
Refine errors to remove changes breaking design (#26521)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-04 09:57:09 +08:00
yah01 5dae6a654a
Protect segment from being released while query/search (#26322)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-16 18:38:17 +08:00
wei liu b47a72bfcf
fix set dirty segment distribution to leader view (#26180)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-11 11:21:32 +08:00
smellthemoon 3fc18c53a3
Fix wrong log (#26022)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-08-02 11:01:07 +08:00
wei liu 302897f866
refine look aside balance logic (#25837)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-25 18:51:01 +08:00
congqixia cc71e065e3
Support load index after segment loaded (#25567)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-18 10:51:19 +08:00
Enwei Jiao 66fdc71479
Refactor logs in DataCoord & DataNode (#25574)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-07-14 15:56:31 +08:00
yah01 205a7c430a
Fix QueryNode panic while inserting to released segment (#25495)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-12 19:48:29 +08:00
wei liu b7740835fe
fix metric label for sq count metrics (#25399)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-07 14:06:25 +08:00
wei liu d5ec7dda80
fix metric label in search/query count (#25377)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-06 17:48:26 +08:00
wei liu c84496a1ba
fix sq count metrics (#25287)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-06 08:46:26 +08:00
chyezh d7d61f529c
[Feature|Pick] enable scheduler policy and add user-task-polling policy (#24839)
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2023-07-03 18:24:25 +08:00
Gao de6e4817a2
Support dynamic tuning config (#25152)
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-07-03 15:18:24 +08:00
wei liu 68ae199a9f
load segment with target version, avoid read redundant segment (#24929)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-06-27 11:48:45 +08:00
Gao 1f8af0596e
Fix autoindex filter bug (#24775)
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-06-09 18:18:36 +08:00
congqixia 41af0a98fa
Use go-api/v2 for milvus-proto (#24770)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
aoiasd 44e5daae3a
Fix querynode read quota (#24412)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-06-08 17:24:36 +08:00
Gao ea62922829
Disable search params rewrite while searching with filter (#24625)
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-06-06 14:28:35 +08:00
yihao.dai 014387fd94
Forbid to get quantized vector from ChunkManager (#24334)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-24 23:03:27 +08:00
wei liu ce03248b1a
simplify logic circle in search (#24110)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-05-23 16:01:26 +08:00