See also #29526
Previous PR removed flushed segment info from request, which causes
pipeline failing to exclude flushed segment info
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #27515
When Delegator processes delete data, it forwards delete data with only
segment id specified. When two segments has same segment id but one is
growing and the other is sealed, the delete will be applied to both
segments which causes delete data out of order when concurrent load
segment occurs.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue:https://github.com/milvus-io/milvus/issues/29230
this pr do two things about cagra index:
a.milvus yaml config support gpu memory settings
b.add cagra-params check
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
Co-authored-by: yusheng.ma <yusheng.ma@zilliz.com>
the mmap mode isn't controlled by the config anymore, so we don't
require user to set it when they enabled mmap mode.
Signed-off-by: yah01 <yah2er0ne@outlook.com>
support enable/disable mmap for index, the user could alter the index's
mode by `AlterIndex` method
related: https://github.com/milvus-io/milvus/issues/21866
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
See also #29332
The segment may be released before or during the request when delegator
tries to forward delete request to yet. Currently, these two situation
returns different error code.
In this particular case, `ErrSegmentNotLoaded` and `ErrSegmentNotFound`
shall both be ignored preventing return search service unavailable by
mistake.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
many growing segments may be created in a short time and there is no
restriction to the process, the CGO call will leave many threads
related: #29282
Signed-off-by: yah01 <yah2er0ne@outlook.com>
See also #28795
Orignal `C.NewSegment` may panic if some condition is not met, this pr
changes response struct to `CNewSegmentResult`, which contains
`C.CStatus` and may return catched exception
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
return the last error but not combining all errors, to improve
readability and erorr handling
resolve: #28572
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
We have implemented the chunkcache (in cpp) to retrieve vectors, hence
rendering the vectorchunkcache (in golang) obsolete.
issue: https://github.com/milvus-io/milvus/issues/28568
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Now segcore load system field info as well, the growing segment
assertion shall not pass with "+ 2" value
This will cause all growing segments load failure
Fix#28801
Related to #28478
See also #28524
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #27678
when interimIndex = true, memory predict should be update with the
memory usage of binlog index build process.
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
issue: #28622
query node with delegator will has more rows than other query node due
to delgator loads all growing rows.
This PR enable the balance segment which based on the num of growing
rows in leader view.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
See also #28022#28034
The load segment may reaches before watch dml channel, so the index meta
may be empty as well
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Fix#28386
Current code snippet
```
// get delegator
sd, ok := node.delegators.Get(channel)
if !ok {
err := merr.WrapErrChannelNotFound(channel)
log.Warn("Query failed, failed to get shard delegator for search", zap.Error(err))
return nil, err
}
req, err = node.optimizeSearchParams(ctx, req, sd)
if err != nil {
log.Warn("failed to optimize search params", zap.Error(err))
return nil, err
}
// do search
results, err := sd.Search(searchCtx, req)
```
We could move these into `ShardDelegator`, and directly use sealed
segment num in `Search` methods, also segment num got outside could be
wrong when we specify partitions.
Signed-off-by: chasingegg <chao.gao@zilliz.com>
See also #28596
Increase ref for collection during load and unref after load completed.
Use the same logic protection from services.go `LoadSegments`
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
This check rejects load request if running out the pool workers, but
small segment would be loaded soon, another segments would been loading
again after a check interval, which leads to slow loading for collection
Block the request by go pool
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
congqixia
MrPresent-Han
Gao
aoiasd
yah01
cqy123456
SimFG
wei liu
zhagnlu
yah01
cai.zhang
yihao.dai
smellthemoon
Bingyi Sun