milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
21,884 Commits
171 Branches
147 Tags
598 MiB
Commit Graph

153 Commits (eb046863485fdf3e130fc60484485c901b81276b)

Author SHA1 Message Date
congqixia d90e01532f
enhance: Release level zero segments when channel unsub (#31486) 
Related to #27349
See also #30816

Level zero is not allowed to balance among delegators, they shall always
serve current delegator. This PR releases all level zero segments after
channel is unsubscribed and preventing level zero segment blocking
graceful stop.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-03-22 10:27:17 +08:00
wei liu 9cfe183253
enhance: remove duplicated target node id check (#31087) 
issue: #31109
This PR remove duplicate target node id check, due to server id has
already been checked in rpc's interceptor

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-03-11 15:31:02 +08:00
wei liu 382b7485a6
enhance: Correct misleading nodeID in GetComponentStates's log (#30731) 
This PR corrects the misleading nodeId in GetComponentStates's log

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-02-28 13:13:12 +08:00
congqixia 536e082f5a
enhance: Use defer to clean legacy growing segment (#30770) 
See also #30735 #30735

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-02-23 10:02:55 +08:00
congqixia f3b7a8892f
fix: Release loaded growing if WatchDmlChannel fail (#30735) 
See also #30734

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-02-21 22:12:53 +08:00
yiwangdr c6665c2a4c
test: support multiple data/querynodes in integration test (#30618) 
issue: https://github.com/milvus-io/milvus/issues/29507

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2024-02-21 11:54:53 +08:00
zhagnlu e8a6f1ea2b
fix: erase pk empty check when pk index replace raw data (#30432) 
#30350

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-02-07 14:56:47 +08:00
xige-16 060c8603a3
fix: Support mvcc with hybrid serach (#30114) 
issue: https://github.com/milvus-io/milvus/issues/29656
/kind bug

Signed-off-by: xige-16 <xi.ge@zilliz.com>

---------

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2024-02-01 16:03:03 +08:00
SimFG ddccccbcab
enhance: add the bytes data type for merge data and format some code (#30105) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-01-18 22:18:55 +08:00
congqixia 10622698df
enhance: reduce delete detail log to delete range (#29916) 
Delete detail log will be large and hard to read when log level is
debug. This PR change the log to stringer and print only pk range,
number.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-12 14:40:59 +08:00
congqixia 93f87417fd
enhance: remove .git folder for unit test workflow (#29833) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-10 16:46:49 +08:00
zhenshan.cao 60e88fb833
fix: Restore the MVCC functionality. (#29749) 
When the TimeTravel functionality was previously removed, it
inadvertently affected the MVCC functionality within the system. This PR
aims to reintroduce the internal MVCC functionality as follows:

1. Add MvccTimestamp to the requests of Search/Query and the results of
Search internally.
2. When the delegator receives a Query/Search request and there is no
MVCC timestamp set in the request, set the delegator's current tsafe as
the MVCC timestamp of the request. If the request already has an MVCC
timestamp, do not modify it.
3. When the Proxy handles Search and triggers the second phase ReQuery,
divide the ReQuery into different shards and pass the MVCC timestamp to
the corresponding Query requests.

issue: #29656

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-01-09 11:38:48 +08:00
xige-16 9702cef2b5
feat: Support multiple vector search (#29433) 
issue #25639 

Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2024-01-08 15:34:48 +08:00
congqixia da7c3cbd88
enhance: make delegator delete buffer holding all delete from cp (#29626) 
See also #29625

This PR:
- Add a new implemention of `DeleteBuffer`: listDeleteBuffer
  - holds cacheBlock slice
  - `Put` method append new delete data into last block
  - when a block is full, append a new block into the list
- Add `TryDiscard` method for `DeleteBuffer` interface
  - For doubleCacheBuffer, do nothing
- For listDeleteBuffer, try to evict "old" blocks, which are blocks
before the first block whose start ts is behind provided ts
- Add checkpoint field for `UpdateVersion` sync action, which shall be
used to discard old cache delete block

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-04 17:02:46 +08:00
congqixia a8b7629315
fix: exclude insertData before growing checkpoint (#29558) 
Resolves: #29556
Refine exclude segment function signature
Add exclude growing before checkpoint logic

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-28 18:18:54 +08:00
congqixia 6597c72992
fix: compose exclude info from flushed segment id (#29548) 
See also #29526

Previous PR removed flushed segment info from request, which causes
pipeline failing to exclude flushed segment info

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-28 14:02:54 +08:00
congqixia b251c3a682
enhance: add ctx for HandleCStatus and callers (#29517) 
See also #29516

Make `HandleCStatus` print trace id for better logging

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-27 16:10:47 +08:00
congqixia 02bc0d0dd5
fix: Add scope limit for querynode DeleteRequest (#29474) 
See also #27515

When Delegator processes delete data, it forwards delete data with only
segment id specified. When two segments has same segment id but one is
growing and the other is sealed, the delete will be applied to both
segments which causes delete data out of order when concurrent load
segment occurs.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-26 14:28:47 +08:00
aoiasd b4af6f8c40
fix: sync action load segment with lack collection index info list (#28788) 
relate: https://github.com/milvus-io/milvus/issues/28779
https://github.com/milvus-io/milvus/issues/28637

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-12-04 18:14:34 +08:00
yah01 bf633bb5d7
enhance: refine the retry error (#28573) 
return the last error but not combining all errors, to improve
readability and erorr handling

resolve: #28572

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-30 18:34:32 +08:00
yihao.dai 4b8bc2798e
enhance: Print nq (#28507) 
Log nq in search path.

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-11-30 10:42:27 +08:00
wei liu 911a915798
feat: enable balance based on growing segment row count (#28623) 
issue: #28622 

query node with delegator will has more rows than other query node due
to delgator loads all growing rows.
This PR enable the balance segment which based on the num of growing
rows in leader view.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-27 14:58:26 +08:00
congqixia 098c1c926d
fix: Add IndexList check for load segment request (#28601) 
See also #28022 #28034
The load segment may reaches before watch dml channel, so the index meta
may be empty as well

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-23 19:42:30 +08:00
Gao 3e77365de5
fix: correct autoindex segment num (#28387) 
Fix #28386 
Current code snippet 
```
// get delegator
sd, ok := node.delegators.Get(channel)
if !ok {
err := merr.WrapErrChannelNotFound(channel)
log.Warn("Query failed, failed to get shard delegator for search", zap.Error(err))
return nil, err
}
req, err = node.optimizeSearchParams(ctx, req, sd)
if err != nil {
log.Warn("failed to optimize search params", zap.Error(err))
return nil, err
}
// do search
results, err := sd.Search(searchCtx, req)
```

We could move these into `ShardDelegator`, and directly use sealed
segment num in `Search` methods, also segment num got outside could be
wrong when we specify partitions.

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-11-22 11:12:22 +08:00
smellthemoon 73f2bab454
enhance:add some log when create client and get component states (#28160) 
/kind improvement

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-11-22 09:12:22 +08:00
yah01 d20ea061d6
Fix panic while forwarding empty deletions to growing segment (#28213) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-08 16:42:21 +08:00
yah01 ece592a42f
Deliver L0 segments delete records (#27722) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-07 01:44:18 +08:00
wei liu 7485eeb689
fix sync distribution with wrong version (#28130) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-03 19:02:18 +08:00
yah01 dc89730a50
Support collection-level mmap control (#26901) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-02 23:52:16 +08:00
wei liu 9b737b77a7
fix fromShardLeader typo (#28061) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-02 15:50:16 +08:00
yah01 1b9b8a31a8
Fix QueryNode panic while upgrading (#28034) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-31 15:50:13 +08:00
congqixia 13877a07ff
Add ctx parameter for tsafe pkg & NewDelegator method (#27877) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-26 19:14:10 +08:00
yah01 be980fbc38
Refine state check (#27541) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-11 21:01:35 +08:00
wei liu 0695c88844
refine_log_of_update_target_version (#27572) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-10-11 06:51:33 +08:00
yah01 2d6a968233
Refine QueryNode errors (#27380) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-26 22:09:28 +08:00
yah01 6539a5ae2c
Refine DataCoord status (#27262) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-26 17:15:27 +08:00
jaime 7f7c71ea7d
Decoupling client and server API in types interface (#27186) 
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-09-26 09:57:25 +08:00
SimFG 26f06dd732
Format the code (#27275) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-21 09:45:27 +08:00
yah01 168e82ee10
Fix panic while handling with the nil status (#27040) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-15 10:09:21 +08:00
congqixia 394c898b4c
Discard SyncDistribution set action from legacy querycoord (#27027) 
Since Milvus in lower version (< 2.3.0), there is no load info in set action
which may corrupt data integrity and cause panicking

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-13 09:35:18 +08:00
yah01 00c65fa0d7
Refine QueryNode errors (#27013) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-12 16:07:18 +08:00
aoiasd e107d0794c
support complex delete expression (#25752) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-09-12 10:19:17 +08:00
MrPresent-Han 528948559f
fix false load failure for long unserviable period(#26813) (#26818) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-06 12:57:15 +08:00
yah01 3349db4aa7
Refine errors to remove changes breaking design (#26521) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-04 09:57:09 +08:00
congqixia e8f1b1736e
Remove log.Error(err.error())-style log (#26783) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-01 13:09:01 +08:00
MrPresent-Han 8330c18dc9
add log for loading segment(#26564) (#26640) 
/kind improvement

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-31 12:03:00 +08:00
wei liu 7af0f7d90c
avoid concurrent sub/unsub on same channel (#26454) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-23 10:10:22 +08:00
congqixia 123ad921e0
Fix querynodev2 search/query segments return unexpected grpc err (#26341) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-17 18:24:18 +08:00
Enwei Jiao 7d61355ab0
Refactor log for Query (#26310) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-14 18:57:32 +08:00
Gao 9ca8f28005
Fix missing totalChannelNum bug (#26295) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-08-11 18:35:29 +08:00
Enwei Jiao ca1349708b
Remove time travel ralted testcase (#26119) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-10 18:53:17 +08:00
wei liu 302897f866
refine look aside balance logic (#25837) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-25 18:51:01 +08:00
wei liu fc19b85a40
fix count(*）retrieve redundant growing segment (#25825) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-24 14:09:00 +08:00
congqixia cc71e065e3
Support load index after segment loaded (#25567) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-18 10:51:19 +08:00
MrPresent-Han 49655d2f13
fix panic due to lack of indexInfo on querynode(#25362) (#25613) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-07-17 19:23:19 +08:00
yah01 d216f9abda
Clear collection meta after all channels/segments released (#25486) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-14 10:28:30 +08:00
wei liu b72809db15
fix release sealed segment on unwatch channel (#25555) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-13 14:30:29 +08:00
yah01 205a7c430a
Fix QueryNode panic while inserting to released segment (#25495) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-12 19:48:29 +08:00
congqixia bcd3abf644
Separate segments/shards reduce latency metrics (#25475) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-11 18:18:28 +08:00
congqixia 5aec6036dc
Add index info in GetDataDistribution response (#25444) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-11 11:22:29 +08:00
wei liu cc730c1ee5
fix consume growing from dml after release (#25421) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-10 17:36:28 +08:00
wei liu a47fc28023
fix querynode panic in get distribution (#25405) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-07 18:26:25 +08:00
wei liu d5ec7dda80
fix metric label in search/query count (#25377) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-06 17:48:26 +08:00
congqixia 3f9384b515
Fix exclude segment info dml position for flushed segments (#25200) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-06 17:30:26 +08:00
yah01 3d2283415f
Fix delegator leaked (#25347) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-06 14:18:26 +08:00
wei liu c84496a1ba
fix sq count metrics (#25287) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-06 08:46:26 +08:00
yihao.dai c56a26a556
Add SetMetricType and set metric only for shard leader (#25285) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-07-04 10:20:25 +08:00
chyezh d7d61f529c
[Feature|Pick] enable scheduler policy and add user-task-polling policy (#24839) 
Signed-off-by: chyezh <ye.zhen@zilliz.com>
 2023-07-03 18:24:25 +08:00
Gao 8750be1e97
Set receive searchRequest log to debug level (#25223) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-06-29 14:04:22 +08:00
wei liu 9b91519c88
fix search cost metrics on replica selection (#25059) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-06-27 19:02:44 +08:00
wei liu 68ae199a9f
load segment with target version, avoid read redundant segment (#24929) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-06-27 11:48:45 +08:00
congqixia d51e1379fe
Move loadDelta operation after needTransfer check (#24922) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-19 17:14:41 +08:00
wei liu 46f7d903a3
enable look aside balancer on replica selection (#24791) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-06-16 18:38:39 +08:00
yah01 a413842e38
Fix deleted data is still visible (#24849) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-16 17:16:41 +08:00
yihao.dai c7dc1c067a
Assign metric type in load segment request (#24917) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-06-16 16:50:41 +08:00
wei liu e40d95e093
add search/query request cost metrics on qn (#24413) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-06-15 14:24:38 +08:00
yihao.dai e7125672d4
Check metric type at watchChannelTask (#24860) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-06-13 22:08:38 +08:00
congqixia 41af0a98fa
Use go-api/v2 for milvus-proto (#24770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
yihao.dai 711bbd6373
Check metric type before search (#24652) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-06-07 15:58:36 +08:00
yihao.dai 5f7099a9bd
Fix search failed due to metric type mismatch (#24691) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-06-06 21:40:35 +08:00
yah01 fb66924c90
Set metric type while init the collection (#24665) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-06 13:24:36 +08:00
yihao.dai b0ce4cff83
Assign metric type in search request when user has not explicitly assigned (#24591) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-06-01 18:10:32 +08:00
congqixia 48dff300db
Fix Get segment returns ambiguous result (#24505) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-30 12:25:28 +08:00
yihao.dai e49df11942
Fix QueryNode is not able to recovery (#24335) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-05-24 14:29:29 +08:00
yah01 abd250da09
Fix QueryNode is not able to recovery (#24300) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-05-23 17:17:26 +08:00
wei liu ce03248b1a
simplify logic circle in search (#24110) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-05-23 16:01:26 +08:00
yihao.dai 1a3dca9b5e
Fix dynamic partitions loading (#24112) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-05-18 09:17:23 +08:00
yah01 400364483d
Improve predict the resource usage (#23968) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-05-11 15:33:24 +08:00
smellthemoon 146050db82
Fix some wrong ut (#23990) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-05-10 09:31:19 +08:00
yihao.dai f0eb5e8563
Fix partition released but can be searched (#23872) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-05-06 17:42:39 +08:00
yihao.dai 4b26c0afb3
Add collection in querynode when sync new partitions (#23825) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-05-01 16:36:38 +08:00
smellthemoon f0ababb420
Use merr in querynodev2 (#23652) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-04-26 17:06:34 +08:00
foxspy 6f4ed517de
add growing segment index (#23615) 
Signed-off-by: xianliang <xianliang.li@zilliz.com>
 2023-04-26 10:14:41 +08:00
yah01 71364056f9
Fix QueryNode may report succeeded to unsubscribe channel early (#23580) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-04-20 19:30:33 +08:00
Jiquan Long ad7f3d4c3e
Support Count(*) on querynodev2 (#23321) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-04-11 15:40:31 +08:00
MrPresent-Han ba02e70301
fix bug when syncing distribution without schema and remove unused logic(#23085) (#23216) 
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2023-04-07 19:32:29 +08:00
jaime c9d0c157ec
Move some modules from internal to public package (#22572) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-04-06 19:14:32 +08:00
congqixia 8db24e0a83
Support queryHook in querynodev2 (#23140) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-06 14:32:29 +08:00
congqixia e3e3ac1525
Use peek segment info in querynodev2 (#23183) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-03 16:10:24 +08:00
yah01 2a760c108c
Add metrics for QueryNodeV2 (#23163) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-04-03 15:24:24 +08:00