ed644983e2
enhance: add param for bloomfilter( #29388 ) ( #29490 )
...
related: #29388
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-28 18:10:46 +08:00
b251c3a682
enhance: add ctx for HandleCStatus and callers ( #29517 )
...
See also #29516
Make `HandleCStatus` print trace id for better logging
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-27 16:10:47 +08:00
02bc0d0dd5
fix: Add scope limit for querynode DeleteRequest ( #29474 )
...
See also #27515
When Delegator processes delete data, it forwards delete data with only
segment id specified. When two segments has same segment id but one is
growing and the other is sealed, the delete will be applied to both
segments which causes delete data out of order when concurrent load
segment occurs.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-26 14:28:47 +08:00
ac95c52171
enhance: change protection to RLock for loadStreamDelete ( #29450 )
...
See also: #29332
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-25 23:27:02 +08:00
1eacdc591b
fix: delegator may mark segment offline by mistake ( #29343 )
...
See also #29332
The segment may be released before or during the request when delegator
tries to forward delete request to yet. Currently, these two situation
returns different error code.
In this particular case, `ErrSegmentNotLoaded` and `ErrSegmentNotFound`
shall both be ignored preventing return search service unavailable by
mistake.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-20 21:22:43 +08:00
9b3e06ae86
enhance: add more metrics for level zero segments ( #29029 )
...
- Add SegmentNum metric for level zero segments
- Add level zero segments size metirc
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-07 14:34:35 +08:00
3e77365de5
fix: correct autoindex segment num ( #28387 )
...
Fix #28386
Current code snippet
```
// get delegator
sd, ok := node.delegators.Get(channel)
if !ok {
err := merr.WrapErrChannelNotFound(channel)
log.Warn("Query failed, failed to get shard delegator for search", zap.Error(err))
return nil, err
}
req, err = node.optimizeSearchParams(ctx, req, sd)
if err != nil {
log.Warn("failed to optimize search params", zap.Error(err))
return nil, err
}
// do search
results, err := sd.Search(searchCtx, req)
```
We could move these into `ShardDelegator`, and directly use sealed
segment num in `Search` methods, also segment num got outside could be
wrong when we specify partitions.
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-11-22 11:12:22 +08:00
cc952e0486
enhance: optimize forwarding level0 deletions by respecting partition ( #28456 )
...
- Cache the level 0 deletions after loading level0 segments
- Divide the level 0 deletions by partition
related: #27349
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 18:24:22 +08:00
d20ea061d6
Fix panic while forwarding empty deletions to growing segment ( #28213 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-08 16:42:21 +08:00
ece592a42f
Deliver L0 segments delete records ( #27722 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-07 01:44:18 +08:00
ecec5dfcfd
fix retry on offline node ( #28079 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-03 10:14:16 +08:00
e4fdf5e68e
Refine offline segments logic in shard delegator ( #28073 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-01 20:04:24 +08:00
13877a07ff
Add ctx parameter for tsafe pkg & NewDelegator method ( #27877 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-26 19:14:10 +08:00
8041fc3c75
avoid add empty growing segment to delegator distribution ( #27930 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-10-26 10:10:10 +08:00
9ba79a97e4
Add some log for delete task when it need query ( #27317 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-10-24 14:54:12 +08:00
d9431266bb
Fix panic at loadSegment during rolling upgrade ( #27671 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-10-13 16:31:35 +08:00
be980fbc38
Refine state check ( #27541 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-11 21:01:35 +08:00
0695c88844
refine_log_of_update_target_version ( #27572 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-10-11 06:51:33 +08:00
258e1ccd66
Refine querynode scheduler lifetime ( #26915 )
...
This PR refines scheduler lifetime control:
- Move private tri-state into lifetime package
- Make scheduler block incoming "Add" task
- Make scheduler Stop wait until all previously accepted task done
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-28 10:21:26 +08:00
8c59dba329
Refine queryHook mockery ( #27394 )
...
This PR move `QueryHook` interface to `optimizers` pkg
Update all mockery generated files to latest
Add makefile entry for `QueryHook`
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-28 10:01:26 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
cc9974979f
Add staticcheck linter and fix existing problems ( #27174 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-19 10:05:22 +08:00
e51cb739eb
Remove some logs in querynode delegator ( #27156 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-09-18 10:57:20 +08:00
e107d0794c
support complex delete expression ( #25752 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-09-12 10:19:17 +08:00
2a5d574a0d
Fix querynodev2 concurrent load logic ( #26959 )
...
Fix logic error from #26926
function `waitSegmentLoadDone` shall return error when context is done
Make delegator control concurrency for each same segment
Related to #26908
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-10 07:41:18 +08:00
7f1ae35e72
Add timeout in dispatcher, AsConsumer and Seek ( #26686 )
...
See also: #25309
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-08 09:51:17 +08:00
af5c01082b
Refine delegator lifetime control ( #26881 )
...
- Add SafeChan interface in lifetime package
- Embed SafeChan into interface
- Replace private lifetime struct in delegator package with
- Refine delegator on-going task lifetime control and wait all accepted task done
- Fix potential goroutine leakage from if delegator closed concurrently
/kind improvement
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-07 10:11:15 +08:00
528948559f
fix false load failure for long unserviable period( #26813 ) ( #26818 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-09-06 12:57:15 +08:00
4b58c71908
Add ctx parameter for organizeTask and GetWorker method ( #26835 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-05 10:05:48 +08:00
1a8cf5c415
Organize all mockery generation commands in Makefile ( #26826 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-04 21:19:48 +08:00
3349db4aa7
Refine errors to remove changes breaking design ( #26521 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-04 09:57:09 +08:00
89fc9aad82
Improve sync target version logic ( #26630 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-29 23:12:27 +08:00
7af0f7d90c
avoid concurrent sub/unsub on same channel ( #26454 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-23 10:10:22 +08:00
855a04539a
Set delegator lastUpdateTs to startTs ( #26416 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-17 12:24:18 +08:00
23baecd70f
set sealed segment to unreadable before sync target version ( #26338 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-15 17:27:35 +08:00
7d61355ab0
Refactor log for Query ( #26310 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-08-14 18:57:32 +08:00
b47a72bfcf
fix set dirty segment distribution to leader view ( #26180 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-11 11:21:32 +08:00
eea9197306
Perform alter distribution under mutex protection ( #26229 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-09 13:05:15 +08:00
99c633954a
Downgrade the log level in search path ( #26188 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-08 18:35:14 +08:00
54c0e64059
Fix search on empty segments set bug ( #26136 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-08-08 11:17:08 +08:00
a132122370
fix time unit in insert/delete cost metrics value ( #25893 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-25 15:07:01 +08:00
b533c68632
Forbid update checkpoint without msgID ( #25694 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-07-25 10:43:04 +08:00
0e26486abc
fix segment num in log ( #25876 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-25 09:13:00 +08:00
fc19b85a40
fix count(*）retrieve redundant growing segment ( #25825 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-24 14:09:00 +08:00
32827f538a
add metrics for delegator insert/delete cost ( #25733 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-21 15:30:59 +08:00
9a4761dcc7
Remove binary metrics TANIMOTO/SUPERSTRUCTURE/SUBSTRUCTURE ( #25708 )
...
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-07-19 16:16:58 +08:00
cc71e065e3
Support load index after segment loaded ( #25567 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-18 10:51:19 +08:00
a7023ab3cc
Narrow the scope of deleteMut lock in loadStreamDelete ( #25436 )
...
Signed-off-by: Ted Yu <yuzhihong@gmail.com>
2023-07-17 09:30:34 +08:00
b62c82af22
fix set target version in loading sealed segment ( #25603 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-14 20:00:31 +08:00
d216f9abda
Clear collection meta after all channels/segments released ( #25486 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-14 10:28:30 +08:00
MrPresent-Han
congqixia
yah01
Gao
wei liu
aoiasd
yihao.dai
SimFG
Xiaofan
XuanYang-cn
Enwei Jiao
Bingyi Sun
Cai Yudong
Zhihong Yu
yah01