Commit Graph

29 Commits (e6eb6f2c78e0ea4e7c7296b139767e5df8322800)

Author SHA1 Message Date
yah01 a0cec4047a
fix: make the entity num metric accurate (#29643)
fix #29642

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-05 18:24:47 +08:00
congqixia da7c3cbd88
enhance: make delegator delete buffer holding all delete from cp (#29626)
See also #29625

This PR:
- Add a new implemention of `DeleteBuffer`: listDeleteBuffer
  - holds cacheBlock slice
  - `Put` method append new delete data into last block
  - when a block is full, append a new block into the list
- Add `TryDiscard` method for `DeleteBuffer` interface
  - For doubleCacheBuffer, do nothing
- For listDeleteBuffer, try to evict "old" blocks, which are blocks
before the first block whose start ts is behind provided ts
- Add checkpoint field for `UpdateVersion` sync action, which shall be
used to discard old cache delete block

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-04 17:02:46 +08:00
MrPresent-Han ed644983e2
enhance: add param for bloomfilter(#29388) (#29490)
related: #29388

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-28 18:10:46 +08:00
congqixia b251c3a682
enhance: add ctx for HandleCStatus and callers (#29517)
See also #29516

Make `HandleCStatus` print trace id for better logging

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-27 16:10:47 +08:00
congqixia 1eacdc591b
fix: delegator may mark segment offline by mistake (#29343)
See also #29332

The segment may be released before or during the request when delegator
tries to forward delete request to yet. Currently, these two situation
returns different error code.

In this particular case, `ErrSegmentNotLoaded` and `ErrSegmentNotFound`
shall both be ignored preventing return search service unavailable by
mistake.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-20 21:22:43 +08:00
yah01 9b3e06ae86
enhance: add more metrics for level zero segments (#29029)
- Add SegmentNum metric for level zero segments
- Add level zero segments size metirc

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-07 14:34:35 +08:00
Gao 3e77365de5
fix: correct autoindex segment num (#28387)
Fix #28386 
Current code snippet 
```
// get delegator
sd, ok := node.delegators.Get(channel)
if !ok {
err := merr.WrapErrChannelNotFound(channel)
log.Warn("Query failed, failed to get shard delegator for search", zap.Error(err))
return nil, err
}
req, err = node.optimizeSearchParams(ctx, req, sd)
if err != nil {
log.Warn("failed to optimize search params", zap.Error(err))
return nil, err
}
// do search
results, err := sd.Search(searchCtx, req)
```

We could move these into `ShardDelegator`, and directly use sealed
segment num in `Search` methods, also segment num got outside could be
wrong when we specify partitions.

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-11-22 11:12:22 +08:00
yah01 cc952e0486
enhance: optimize forwarding level0 deletions by respecting partition (#28456)
- Cache the level 0 deletions after loading level0 segments
- Divide the level 0 deletions by partition
related: #27349

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 18:24:22 +08:00
yah01 d20ea061d6
Fix panic while forwarding empty deletions to growing segment (#28213)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-08 16:42:21 +08:00
yah01 ece592a42f
Deliver L0 segments delete records (#27722)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-07 01:44:18 +08:00
wei liu ecec5dfcfd
fix retry on offline node (#28079)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-03 10:14:16 +08:00
congqixia 13877a07ff
Add ctx parameter for tsafe pkg & NewDelegator method (#27877)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-26 19:14:10 +08:00
XuanYang-cn 7f1ae35e72
Add timeout in dispatcher, AsConsumer and Seek (#26686)
See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-08 09:51:17 +08:00
congqixia 4b58c71908
Add ctx parameter for organizeTask and GetWorker method (#26835)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-05 10:05:48 +08:00
congqixia 89fc9aad82
Improve sync target version logic (#26630)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-29 23:12:27 +08:00
wei liu 23baecd70f
set sealed segment to unreadable before sync target version (#26338)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-15 17:27:35 +08:00
wei liu fc19b85a40
fix count(*)retrieve redundant growing segment (#25825)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-24 14:09:00 +08:00
Cai Yudong 9a4761dcc7
Remove binary metrics TANIMOTO/SUPERSTRUCTURE/SUBSTRUCTURE (#25708)
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-07-19 16:16:58 +08:00
yah01 d216f9abda
Clear collection meta after all channels/segments released (#25486)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-14 10:28:30 +08:00
SimFG f9e2d00f91
Prevent `exclusive consumer` exception in pulsar (#25376)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-07-12 17:26:30 +08:00
wei liu 68ae199a9f
load segment with target version, avoid read redundant segment (#24929)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-06-27 11:48:45 +08:00
congqixia 41af0a98fa
Use go-api/v2 for milvus-proto (#24770)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
congqixia 73a181d226
Fix get vector it timeout and improve some string const usage (#24141)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-16 17:41:22 +08:00
yihao.dai 7e0d1492c7
Load delete from channel checkpoint (#23961)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-09 19:10:41 +08:00
foxspy 6f4ed517de
add growing segment index (#23615)
Signed-off-by: xianliang <xianliang.li@zilliz.com>
2023-04-26 10:14:41 +08:00
wei liu 3ad9ff7a9a
fix release segment failed (#23462)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-04-17 18:46:30 +08:00
jaime c9d0c157ec
Move some modules from internal to public package (#22572)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
congqixia e3e3ac1525
Use peek segment info in querynodev2 (#23183)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-04-03 16:10:24 +08:00
yah01 081572d31c
Refactor QueryNode (#21625)
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
2023-03-27 00:42:00 +08:00