Related to #39840
The target could be updated async in previous code. This PR make remove
collection from target observer block until all tasks related in
dispatchers are removed preventing the metrics being updated after
collection released.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #39856
This PR add lifetime bound context for compaction trigger and use it
instead of context.Background in case of rootcoord down and some grpc
call retry forever
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #38399
- fix the nil pointer bug
- limit the resource usage for streaming e2e
- enhance the go test
- fix: rootcoord block when graceful stop
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Because when GetIndexParams is used, index params and type params are
concatenated, so when loading index, the mmap.enable parameter in type
params is also referenced.
- issue: #39801
Signed-off-by: SimFG <bang.fu@zilliz.com>
issue: #38399
- Make a timetick-commit-based write ahead buffer at write side.
- Add a switchable scanner at read side to transfer the state between
catchup and tailing read
Signed-off-by: chyezh <chyezh@outlook.com>
- Use ModTime from os.FileInfo directly instead of custom getModTime
implementation.
- Removed test case for getModTime().
Signed-off-by: Gofastasf gofastasf@gmail.com
Signed-off-by: Go gofastasf@gmail.com
Signed-off-by: Gofastasf <gofastasf@gmail.com>
issue: #39680
if compaction/gc happens, load collection may stuck due to
SegmentNotFound, we should trigger UpdateNextTarget to get a new data
view to execute loading operation.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #39681
this PR maintain workload effect in action instead of computing workload
effect from target, which may cause leak if target changes.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Previous PR #39437 only print log and add index while load operation is
still executed. This PR return early when segment decides not to load PK
index.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #38399
- Add a pchannel level checkpoint for flush processing
- Refactor the recovery of flushers of wal
- make a shared wal scanner first, then make multi datasyncservice on it
Signed-off-by: chyezh <chyezh@outlook.com>
Use string array for SealedSegmentIDs to prevent precision loss in JSON
parsers. Large integers (int64) may be incorrectly rounded when parsed
as double.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
See: #39697
In sync operations, the type conversions from message to insert data
always result in a memory copy, which is not necessary if the converting
type is identical.
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
enhance: Add schema update time verification for insert and upsert to
use cache
issue: https://github.com/milvus-io/milvus/issues/39093
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
issue: #36621#39417
1. Adjust the server-side cache size.
2. Add source information for configurations.
3. Add node ID for compaction and indexing tasks.
4. Resolve localhost access issues to fix health check failures for
etcd.
Signed-off-by: jaime <yun.zhang@zilliz.com>
issue: #38399
- broadcast message can carry multi resource key now.
- implement event-based notification for broadcast messages
- broadcast message use broadcast id as a unique identifier in message
- broadcasted message on vchannels keep the broadcasted vchannel now.
- broadcasted message and broadcast message have a common broadcast
header now.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #38399
- Make the wal scanner interface same with streaming scanner.
- Use wal if the wal is located at current node.
- Otherwise fallback the old logic.
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #38399
- The stats may be kept after wal closing if the growing segment is not
dirty.
- Change the error handling of wal open to avoid redundant manager api
call.
Signed-off-by: chyezh <chyezh@outlook.com>
Related to #39596
When updating the build param configuration, the `Formatter` could be
used to do so and completed avoid touching the `overlay` config items
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
The dsl fields are separated in sub request structs and cannot be
easiliy printed before. This PR adds a log helper to print the dsl
expressions of HybridSearchRequest.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
aoiasd
congqixia
cai.zhang
Xiaofan
SimFG
Zhen Ye
junjiejiangjjj
wei liu
Ted Xu
gofastasf
sthuang
Spade A
MimoN
XuanYang-cn
Bingyi Sun
sre-ci-robot
sparknack
Xianhui Lin
jaime
Gao
yihao.dai