milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
19,229 Commits
127 Branches
113 Tags
553 MiB
Commit Graph

410 Commits (2a047103d6989b978913cff4d176f507263e1386)

Author SHA1 Message Date
yah01 a0e1a1eb31
feat: support enable/disable mmap for index (#29005) 
support enable/disable mmap for index, the user could alter the index's
mode by `AlterIndex` method
related: https://github.com/milvus-io/milvus/issues/21866

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-21 18:07:24 +08:00
wei liu 820ee692fc
enhance: Add config for querycoord auto balance channel (#29231) 
issue: #23726
This PR add control config to querycoord's background auto balance
channel operation

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-18 10:00:40 +08:00
yah01 13beb5ccc0
fix: load gets stuck probably (#29191) 
we found the load got stuck probably, and reviewed the logs.

the target observer seems not working, the reason is the taskDispatcher
removes the task in a goroutine, and modifies the task status after
committing the task into the goroutine pool, but this may happen after
the task removed, which leads to the task will never be removed

related #29086

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-14 18:28:38 +08:00
wei liu 008bae675d
enhance: Skip balance segment when channel need be balanced (#29116) 
issue: #28622
After we support balance segment with growing segment count #28623, if
we balance segment and channel at same time, some segments need to be
rebalanced after balance channel finish.

This PR skip balance segment when channel need be balanced.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-14 16:44:43 +08:00
yah01 58dbb7872a
fix: forbid balancing level zero segments (#29168) 
related #29128
Signed-off-by: yah01 <yah2er0ne@outlook.com>

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-14 14:30:39 +08:00
yah01 2f0c7a6544
fix: forbid balancing level zero segments (#29130) 
we can't balance the L0 segments
related #29128

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-12 20:38:38 +08:00
yah01 9819090247
enhance: add more logs for target updating (#29090) 
- add more logs about the condition satisfying

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-12 14:06:43 +08:00
yah01 0a87724f18
enhance: remove merger for load segments (#29062) 
remove merger as now QueryNode could load segments concurrently
fix #29063

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-12 10:48:37 +08:00
congqixia a67fc08865
fix: balance_unstable_view unit test (#29127) 
fix: #29126
Allow unstable output channel balance plan

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-12 10:02:39 +08:00
wei liu 42e538b683
enhance: enable balance channel in querycoord (#28469) 
issue: #23726

/kind improvement

1. enable auto balance channel between nodes in querycoord
2. make `genSegmentPlan` reuse the `AssignSegment` logic
3. make `genChannelPlan` reuse the `AssignChannel` logic

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-11 14:18:37 +08:00
yah01 fab52d167b
fix: may miss stream delta while loading (#28871) 
we consume the delta data from the lastest channel checkpoint while
loading segment,

this works well without level 0 segments, but now it may lead to miss
some delta data,

so we have to consume from the current target's channel checkpoint

related: #27349

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-05 17:34:45 +08:00
Bingyi Sun 10bb2431d8
test: add checker unittests (#28954) 
issue: https://github.com/milvus-io/milvus/issues/28610

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-12-05 10:56:33 +08:00
aoiasd b4af6f8c40
fix: sync action load segment with lack collection index info list (#28788) 
relate: https://github.com/milvus-io/milvus/issues/28779
https://github.com/milvus-io/milvus/issues/28637

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-12-04 18:14:34 +08:00
Bingyi Sun 45e6801ce4
feat: Add checker activation service interfaces (#28850) 
issue: #28610

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-12-04 17:38:37 +08:00
wei liu d081fd5481
enhance: Change some frequency log to rated level (#28897) 
This pr change some frequency log's level to rated.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-04 10:38:35 +08:00
wei liu 043ac87be0
fix: Balance channel may cause channel not availble error (#28829) 
issue: #28831
release old delegator before new delegator update it's distribution may
cause `channel not availble` error

This PR will block release old delgator before new delegator finish
`syncDistribution`

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-01 10:08:34 +08:00
congqixia f9bb8e9648
enhance: Change const magic number in querycoord to param (#28819) 
See also #28817

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-30 09:06:28 +08:00
yah01 c0f6eccb7a
fix: No LevelZero segment in target (#28803) 
the incorrect filter causes all LevelZero segment filtered, so the
deleted entities may be still visible
related: #27349

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-29 11:48:27 +08:00
jaime b1e0a27f31
enhance: Add logs for each step during service initialization (#28624) 
/kind improvement

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-11-27 16:30:26 +08:00
wei liu 911a915798
feat: enable balance based on growing segment row count (#28623) 
issue: #28622 

query node with delegator will has more rows than other query node due
to delgator loads all growing rows.
This PR enable the balance segment which based on the num of growing
rows in leader view.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-27 14:58:26 +08:00
Bingyi Sun 8514a39d1a
feat: Add checker activation (#28611) 
issue: https://github.com/milvus-io/milvus/issues/28610

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-11-24 18:08:24 +08:00
congqixia a2fe9dad49
enhance: Make etcd kv request timeout configurable (#28661) 
See also #28660
This pr add request timeout config item for etcd kv request timeout
 Sync the default timeout value to same value for etcdKV & tikv config

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-23 19:34:23 +08:00
smellthemoon 73f2bab454
enhance:add some log when create client and get component states (#28160) 
/kind improvement

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-11-22 09:12:22 +08:00
yah01 bfccfcd0ca
enhance: refine error messages (#28424) 
- Split the simple reason and full detail
- Refine existing error messages
related: #28422

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-21 17:02:24 +08:00
aoiasd 13a5b9f64a
fix: query l0 segment bugs (#28558) 
relate: https://github.com/milvus-io/milvus/issues/27675

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-11-20 17:26:23 +08:00
wei liu 7895ac96b5
enhance: Remove rpc during querycoord start (#28396) 
issue: #28332

during querycoord's recover, it try to call `DescribeCollection` and
`ShowPartitions` to root coord, to checker whether collection or
partition has been released in rootcoord. but if rootcoord isn't not
ready yet, the rpc will fail, the querycoord panic.

to fix this, we remove rpc call during querycoord's start

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-17 11:48:19 +08:00
congqixia 81caf02554
fix: make qcv2 observer dispatcher execute exactly once (#28472) 
See also #28466

In `taskDispatcher.schedule`, same task may be resubmitted if the
previous round did not finish
In this case, TaskObserver.check may set current target by mistake,
which may cause the random search/query failure

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-16 10:24:19 +08:00
yah01 70995383bf
enhance: modify log to avoid ambiguity and improve readability (#28331) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-10 14:32:20 +08:00
wei liu b9bf910039
fix unstable auto balance config ut (#28288) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-09 10:00:22 +08:00
wei liu 7f78e1dd46
fix datacoord unstable ut (#28281) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-08 18:43:31 +08:00
yah01 ecb3f585c3
Fix passing the wrong dropped list from current target (#28265) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-08 17:02:18 +08:00
yah01 1b90630633
Fix the target updated before version updated to cause data missing (#28250) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-08 11:36:22 +08:00
wei liu 5b45a138b1
disable auto balance when old node exists (#28191) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-07 14:02:20 +08:00
yah01 ece592a42f
Deliver L0 segments delete records (#27722) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-07 01:44:18 +08:00
yah01 90e2c63d9e
Fix getting incorrect CPU num (#28146) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-11-06 06:02:16 +08:00
wei liu 7485eeb689
fix sync distribution with wrong version (#28130) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-03 19:02:18 +08:00
wei liu 86ec6f4832
fix load index for stopping node (#28047) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-03 07:58:18 +08:00
yah01 dc89730a50
Support collection-level mmap control (#26901) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-02 23:52:16 +08:00
congqixia 5d2eba2c2f
Set qcv2 index task priority to Low (#28117) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-02 23:22:17 +08:00
Filip Haltmayer 6b1a106a31
Moving etcd client into session (#27069) 
Signed-off-by: Filip Haltmayer <filip.haltmayer@zilliz.com>
 2023-10-27 07:36:12 +08:00
congqixia 852be152de
Change task sourceID to stringer interface (#27965) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-27 01:08:12 +08:00
wei liu e0222b2ce3
refine target manager code style (#27883) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-10-25 00:44:12 +08:00
congqixia 323fc107a7
Fix taskDispatcher add multiple tasks will ignore following ones (#27885) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-24 17:18:13 +08:00
wei liu 178db7b0f0
check stopping node during start qc (#27859) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-10-24 12:20:11 +08:00
congqixia 93a877f55e
Make qcv2 target&leader observer execute in parallel (#27844) 
- Add `taskDispatcher` to submit and run task async safely
- Change `LeaderObeserver` and `TargetObserver` schedule and manual check action to submitting task into dispatcher
- Fix logic problem in collection observer when manual check return false

See also #27494

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-24 10:14:11 +08:00
Xiaofan 2ea7579dbb
Reduce rpc size for GetRecoveryInfoV2 (#27483) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-10-23 21:44:09 +08:00
wei liu 55e5f80e24
update collection target after observer start (#27774) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-10-19 21:52:10 +08:00
zhenshan.cao 020ad9a6bc
Rectify wrong exception messages associated with Array datatype (#27769) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-10-19 17:24:07 +08:00
yah01 635efdf170
Schedule loading L0 segments first (#27593) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-19 11:14:06 +08:00
yihao.dai 49b3a12804
Return newly defined merr instead of grpc unimplemented err (#27751) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-10-18 15:32:11 +08:00
wayblink e3f2122618
Expose metrics of stanby coordinators (#27698) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-10-16 15:04:09 +08:00
jaime ec1fe3549e
Add a stop hook to clean session (#27564) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-10-16 10:24:10 +08:00
yah01 be980fbc38
Refine state check (#27541) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-11 21:01:35 +08:00
smellthemoon a0ca982a52
Fix typo in priority name (#27558) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-10-11 14:19:35 +08:00
wei liu 42c475a0e0
remove useless log in querycoord (#27362) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-10-11 10:13:34 +08:00
congqixia b91a5ef42c
Refine log and err handling in querycoord broker (#27546) 
- Add log.Ctx(ctx) for all log occurences
- Use `merr.CheckRPCErr` for all grpc response error handling

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-10 11:49:32 +08:00
MrPresent-Han cb71a3e235
rm dependency to rc when getting recovery info(#25363) (#27405) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-10-09 18:51:32 +08:00
congqixia eca79d149c
Add ctx control for observer manual check methods (#27531) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-09 11:07:33 +08:00
yah01 a715165306
Set timeout for leader observer syncing (#27504) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-08 16:55:31 +08:00
Xiaofan 41124f281a
Remove parser dependency (#27514) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-10-08 15:05:31 +08:00
congqixia 5d558623fe
Add revive sub-lints and fix existing problems (#27495) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-07 20:53:38 +08:00
yah01 8394b3a1ec
Block creating new error from status reason (#27426) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-07 11:29:32 +08:00
congqixia cd5f03f80c
Add var-name sub linter in revive (#27424) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-07 10:09:31 +08:00
yah01 63ac43a3b8
Refine errors for import (#27379) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-30 10:31:28 +08:00
Jiquan Long 370fdaf50d
Record engine version for segment index (#27384) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-28 18:03:28 +08:00
yah01 a8ce1b6686
Refine QueryCoord stopping (#27371) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-27 16:27:27 +08:00
wei liu 4071132f6a
reload loading collection when qc recover (#27300) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-27 11:43:28 +08:00
yah01 6539a5ae2c
Refine DataCoord status (#27262) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-26 17:15:27 +08:00
jaime 7f7c71ea7d
Decoupling client and server API in types interface (#27186) 
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-09-26 09:57:25 +08:00
yah01 24354b166c
Fix unit test failed when run single test (#27348) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-26 09:23:25 +08:00
foxspy 5db4a0489e
dynamic index version control (#27335) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-25 21:39:27 +08:00
MrPresent-Han 4b12cb8847
fix unstable ut due to unstable sort of unique set (#27302) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-22 19:07:26 +08:00
foxspy 370b6fde58
milvus support multi index engine (#27178) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-22 09:59:26 +08:00
SimFG 26f06dd732
Format the code (#27275) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-21 09:45:27 +08:00
yah01 b4f86ea55e
Construct all success status with merr (#27226) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-20 10:57:23 +08:00
congqixia cc9974979f
Add staticcheck linter and fix existing problems (#27174) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-19 10:05:22 +08:00
yah01 168e82ee10
Fix panic while handling with the nil status (#27040) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-15 10:09:21 +08:00
congqixia edde3cf1c7
Add tracer for querycoord tasks (#27058) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-14 09:59:19 +08:00
PowderLi c033580af4
show index info while GetSegmentInfo (#26981) 
according to QueryNode::GetSegmentInfo

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-09-13 11:37:18 +08:00
aoiasd e107d0794c
support complex delete expression (#25752) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-09-12 10:19:17 +08:00
congqixia c45c32fad4
Set task reason for collection released (#26962) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-10 15:15:17 +08:00
MrPresent-Han 2101f2d289
fix unstable checker id due to go map iteration(#26943) (#26944) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-10 10:11:16 +08:00
congqixia 758aad705d
Fix checker using default interval after manual check (#26953) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-09 08:29:16 +08:00
SimFG 0901b76732
Avoid the panic when the status of rpc response is nil (#26910) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-07 19:23:15 +08:00
yiwangdr 337edc321b
tikv integration (#26246) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-09-07 07:25:14 +08:00
SimFG 28681276e2
Improve the retry of the rpc client (#26795) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-06 17:43:14 +08:00
Enwei Jiao fb0705df1b
Decouple basetable and componentparam (#26725) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-05 10:31:48 +08:00
congqixia 1a8cf5c415
Organize all mockery generation commands in Makefile (#26826) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-04 21:19:48 +08:00
yah01 3349db4aa7
Refine errors to remove changes breaking design (#26521) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-04 09:57:09 +08:00
yah01 941a383019
Fix failed to load collection with more than 128 partitions (#26763) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-02 00:09:01 +08:00
congqixia e8f1b1736e
Remove log.Error(err.error())-style log (#26783) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-01 13:09:01 +08:00
wei liu 5602b22531
refine checker code style (#26759) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-01 11:57:01 +08:00
wei liu 949c320185
remove pull target from qc recover (#26775) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-01 11:17:01 +08:00
yah01 213db490bd
Use pointer receiver for large struct (#26668) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-30 10:24:29 +08:00
congqixia 9364d0ea49
Remove etcd dependency for querycoord unit test (#26550) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-28 11:20:25 +08:00
yah01 f9c060e0d2
Treat balance task with released source segment as stale (#26453) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-22 17:12:22 +08:00
congqixia 065d1a962e
Add sourceID output for task.String and fill reduce channel reason (#26447) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-18 13:50:19 +08:00
yihao.dai 63b86b32a6
Add server id validation interceptor (#26395) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-08-17 20:20:20 +08:00
yah01 9723787141
Calculate memory usage without page cache (#26389) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-16 17:04:17 +08:00
Enwei Jiao 7d61355ab0
Refactor log for Query (#26310) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-14 18:57:32 +08:00