milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
18,616 Commits
133 Branches
115 Tags
559 MiB
Commit Graph

7899 Commits (ed79505d31f51e63bca8f859b8a8d4c4f9a54b42)

Author SHA1 Message Date
zhagnlu 0d9d098186
enhance: Add precheck when chunk manager init (#28330) 
#28329

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-11-23 19:56:32 +08:00
wayblink da339535d5
enhance: Merge flowgraph goroutines into 1 (#28654) 
/kind enhancement
#24826

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-11-23 19:52:25 +08:00
congqixia 098c1c926d
fix: Add IndexList check for load segment request (#28601) 
See also #28022 #28034
The load segment may reaches before watch dml channel, so the index meta
may be empty as well

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-23 19:42:30 +08:00
congqixia a2fe9dad49
enhance: Make etcd kv request timeout configurable (#28661) 
See also #28660
This pr add request timeout config item for etcd kv request timeout
 Sync the default timeout value to same value for etcdKV & tikv config

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-23 19:34:23 +08:00
zhagnlu e9e9b30e8e
enhance: Disable stdout buffer (#28391) 
#28390

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-11-23 19:26:25 +08:00
XuanYang-cn b1f15fa0e8
fix: Ease the log level when sync task done (#28678) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-23 19:18:28 +08:00
XuanYang-cn 9b371067d2
feat: Add Compaction views and triggers (#27906) 
- Add Compaction l0 views
- Add Compaction scheduler
- Add Compaction triggerv2
- Add Compaction view manager

See also: #27606

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-23 17:30:25 +08:00
Bingyi Sun 4fedff6d47
feat: integrate storage v2 into the write path (#28440) 
#28378

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-11-23 17:26:24 +08:00
Bingyi Sun e5ce385ffd
enhance: remove -inl.h files (#28674) 
issue: https://github.com/milvus-io/milvus/issues/28673
Move template implementations from -inl.h to .cpp file and make explicit
instantiation

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-11-23 17:20:25 +08:00
cai.zhang 31f442915b
fix: Fix bug for parsing expression that include quotes (#28416) 
issue: #28365 
Fix bug for parsing error when a string enclosed in single quotes in an
expression contains multiple double quotes.
such as:
```
expr = "tag == '\"blue\"'"
```

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-23 17:18:32 +08:00
SimFG de13865769
enhance: Add load/release partitions to replicate msg stream (#28399) 
/kind improvement
issue: #25655

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-11-23 15:38:24 +08:00
yah01 c96d07682e
enhance: enhance the accuracy of memory usage (#28554) 
before this, Milvus use container/system's memory info to get the memory
usage, which could be inaccurate.

we allocates the memory by private anon mmap,
then `rss - shared` would be the accurate memory usage

resolve #28553

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-23 15:12:23 +08:00
XuanYang-cn 55800ade84
fix: Remove logging extra segmentIDs (#28662) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-23 14:40:23 +08:00
congqixia af18aa709b
fix: Remove not needed `BlockAll` call in `SyncSegments` (#28632) 
See also #28628
Previous compaction task blocked the segment sync task and may block the
flowgraph when sync task is generated by auto sync policy This
`BlockAll` call will block forever and cause whole fg stuck

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-22 14:52:23 +08:00
Gao 3e77365de5
fix: correct autoindex segment num (#28387) 
Fix #28386 
Current code snippet 
```
// get delegator
sd, ok := node.delegators.Get(channel)
if !ok {
err := merr.WrapErrChannelNotFound(channel)
log.Warn("Query failed, failed to get shard delegator for search", zap.Error(err))
return nil, err
}
req, err = node.optimizeSearchParams(ctx, req, sd)
if err != nil {
log.Warn("failed to optimize search params", zap.Error(err))
return nil, err
}
// do search
results, err := sd.Search(searchCtx, req)
```

We could move these into `ShardDelegator`, and directly use sealed
segment num in `Search` methods, also segment num got outside could be
wrong when we specify partitions.

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-11-22 11:12:22 +08:00
smellthemoon 29249c4bd3
enhance: create goroutine only once (#28594) 
create goroutine only once when getOrCreateMergedTimeTickerSender

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-11-22 10:26:28 +08:00
Bingyi Sun e8c5e75786
enhance: ignore proto generated files (#28537) 
ignore proto generated files
issue: https://github.com/milvus-io/milvus/issues/28566

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-11-22 10:20:22 +08:00
smellthemoon 73f2bab454
enhance:add some log when create client and get component states (#28160) 
/kind improvement

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-11-22 09:12:22 +08:00
congqixia 2fc743992a
fix: syncmgr unstable TestCompact unittest logic (#28630) 
fix #28629
orignal unit test close channel before setting the segment id, so there
is a chance that test read segment id before setting it change unit test
behavior to wait future return now

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-22 00:52:23 +08:00
yah01 d73dac52c7
enhance: validate JSON data while inserting (#28602) 
some SDKs doesn't check the JSON data validation,
add this in server.

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-21 23:30:22 +08:00
congqixia e47fe4e4d1
enhance: Reduce ListImportTasks log content (#28605) 
In proxy `ListImportTasks` may print all task information from
rootcoord, this may cause log content too large to process
This PR make this log print taskID and importState only

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-21 20:58:23 +08:00
congqixia faf38ed8dd
fix: collection not `Unref` in local worker release (#28590) 
See also #28589

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-21 20:20:24 +08:00
congqixia efe3fe7b2e
fix: collection released when localWorker load segments (#28598) 
See also #28596
Increase ref for collection during load and unref after load completed.
Use the same logic protection from services.go `LoadSegments`

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-21 20:18:22 +08:00
yah01 cc952e0486
enhance: optimize forwarding level0 deletions by respecting partition (#28456) 
- Cache the level 0 deletions after loading level0 segments
- Divide the level 0 deletions by partition
related: #27349

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-21 18:24:22 +08:00
yah01 bfccfcd0ca
enhance: refine error messages (#28424) 
- Split the simple reason and full detail
- Refine existing error messages
related: #28422

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-21 17:02:24 +08:00
aoiasd b1fbc1f4e7
fix: datacoord l0 segment bugs (#28490) 
relate: https://github.com/milvus-io/milvus/issues/27675

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-11-21 16:48:22 +08:00
congqixia 6521b519d7
enhance: Use bufSize instead of row number in sync policy (#28498) 
See also #27675, comment from #27874

This PR changes the `IsFull` logic of insert buffer
Changing the limit from rowNum to buffer size,
this shall help form better binlog file when schema has variable-length
field

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-21 15:02:25 +08:00
Bingyi Sun 0f985c9961
enhance: remove concurrency level in `checkSegmentSize` (#28533) 
`concurrency` is no longer needed in `checkSegmentSize`

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-11-21 15:00:23 +08:00
congqixia 2b3fa8f67b
fix: Add length check for `storage.NewPrimaryKeyStats` (#28576) 
See also #28575
Add zero-length check for `storage.NewPrimaryKeyStats`. This function
shall return error when non-positive rowNum passed.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-21 10:28:21 +08:00
PowderLi c238bff9fb
fix: symbol 'GetStorageMetrics' and 'enableDynamicField' (#28580) 
/kind bug
to #28579 #28504

1. replace enableDynamic with enableDynamicField
2. cgo directly link to milvus_storage

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-21 10:20:22 +08:00
Bingyi Sun d7145e2c06
enhance: Update golangci_lint version (#28535) 
Update golangci lint and fix some warnings

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-11-21 10:04:21 +08:00
yah01 02c5a649cf
enhance: store system fields in segcore (#28524) 
we need the system fields info for some usacase
fix: #28523

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-21 09:28:22 +08:00
Enwei Jiao 7445d3711c
feat: trigger compaction to handle index version (#28442) 
issue: https://github.com/milvus-io/milvus/issues/28441

---------

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-11-21 09:26:22 +08:00
wei liu fdbca10e23
fix: Fix channel manager bg checker exit when disable auto balance (#28459) 
issue: #28454

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-20 18:20:22 +08:00
aoiasd 13a5b9f64a
fix: query l0 segment bugs (#28558) 
relate: https://github.com/milvus-io/milvus/issues/27675

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-11-20 17:26:23 +08:00
wayblink 7f46f4c628
fix: failed to release collection with more than 128 partitions (#28446) 
#28343

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-11-19 18:28:21 +08:00
PowderLi a1c505dbd5
add internal storage metrics (#28278) 
/kind improvement
issue: #28277

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-19 17:22:25 +08:00
PowderLi 7029797452
Improve HTTP server include metrics and RESTful API (#28045) 
/kind improvement
issue: #27653

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-19 17:14:20 +08:00
congqixia 18dc6b61ce
enhance: fix LevelZero segment sync logic (#28482) 
See also #27675
- Fix LevelZero segment cannot be flushed
- Add level option for syncTask
- Invoke `AddSegment` when new LevelZero segment is allocated

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-17 21:46:20 +08:00
yah01 d2f53aefa5
enhance: improve load speed (#28518) 
This check rejects load request if running out the pool workers, but
small segment would be loaded soon, another segments would been loading
again after a check interval, which leads to slow loading for collection

Block the request by go pool

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-17 19:56:21 +08:00
congqixia a3cd0bc9c3
fix: Refine sync task field binlog compose logic (#28494) 
See also #27675
Since `MetaWriter` need `*datapb.FieldBinlog` struct, sync task now
generate FieldBinlog directly

Also fix merged statslog not generated if last task has no insert

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-17 14:40:26 +08:00
wei liu 7895ac96b5
enhance: Remove rpc during querycoord start (#28396) 
issue: #28332

during querycoord's recover, it try to call `DescribeCollection` and
`ShowPartitions` to root coord, to checker whether collection or
partition has been released in rootcoord. but if rootcoord isn't not
ready yet, the rpc will fail, the querycoord panic.

to fix this, we remove rpc call during querycoord's start

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-17 11:48:19 +08:00
Enwei Jiao cc713931e9
fix: Fix rocksdb compile failed (#28484) 
Fix: #27675
/kind improvement

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-11-16 16:36:19 +08:00
congqixia 81caf02554
fix: make qcv2 observer dispatcher execute exactly once (#28472) 
See also #28466

In `taskDispatcher.schedule`, same task may be resubmitted if the
previous round did not finish
In this case, TaskObserver.check may set current target by mistake,
which may cause the random search/query failure

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-16 10:24:19 +08:00
congqixia bed7467f20
enhance: Remove commented code and fix naming issue (#28450) 
This PR removes all the commented code and files from PR #28320

For naming issue:
- Renaming `MinCheckpoint` to `EarliestPosition`, see #28320 comment
- Renaming `writebuffer.Mananger` to `BufferMananger`, see #27874
comment

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-16 00:22:20 +08:00
yah01 f7d2ab6677
enhance: reduce 1x copy for variable length field while retrieving (#28345) 
- Reduce 1x copy for varchar/string/JSON/array types while retrieving
- Reduce 1x copy for int8/int16 while retrieving

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-15 18:08:20 +08:00
congqixia 0b905078e7
Use writebuffer, sync manager refactory in datanode (#28320) 
See also #27675
This PR make previously merged refactory of datanode go online
- Use write node to replace insert/delete node
- Use write buffer manager to control all buffers
- Use sync manager to control sync tasks instead of flush manager

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-15 15:24:18 +08:00
SimFG 899a5a32cd
Hide the password info when failing to authorize (#28428) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-11-15 14:40:26 +08:00
MrPresent-Han 836f300536
support skip-index based on chunk-metrics to accelerate expr filter(#27925) (#28297) 
related: #27925

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-11-15 11:20:19 +08:00
Bingyi Sun 59355cb3dc
Update arrow version to v12 (#28425) 
issue: https://github.com/milvus-io/milvus/issues/28423

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-11-15 10:36:19 +08:00
XuanYang-cn 0be22ccacd
Fix compaction unable to notify datanode (#28409) 
See also: #28214, #28368

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-14 16:38:18 +08:00
XuanYang-cn 40d5c902b6
Enable getting multiple segments in plan result (#28350) 
Compaction plan result contained one segment for one plan. For l0
compaction would write to multiple segments, this PR expand the segments
number in plan results and refactor some names for readibility.

- Name refactory: - CompactionStateResult -> CompactionPlanResult -
CompactionResult -> CompactionSegment

See also: #27606

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-14 15:56:19 +08:00
XuanYang-cn 8037f35de7
Change MetaCache interface (#28388) 
See also: #28320, #27675

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-14 15:08:19 +08:00
yah01 e9ff7ed13d
enhance: remove legacy error related code (#28385) 
/kind improvement

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-14 15:02:18 +08:00
congqixia b1eb1ea506
Refine datanode Timetick Sender (#28393) 
- Use explicit lifetime control methods: `Start` and `Stop`
- Allow control retry option
- Make sure tt sender worker exit after `Stop` return

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-13 21:46:20 +08:00
SimFG cfb6edea61
Support to trace the grpc request (#28349) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-11-13 20:16:18 +08:00
XuanYang-cn 87465f07a7
Fix unstable ConsistencyHashPolicy ut (#28375) 
Fixes: #28372

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-13 19:06:20 +08:00
yihao.dai c16fc854d1
Fix confusing quota log (#28262) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-11-13 15:38:18 +08:00
XuanYang-cn a153950b10
Change channel to Interface (#27839) 
This PR changes `*channel` into RWChannel interface

See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-13 11:16:18 +08:00
smellthemoon 0aa90de141
Reduce the goroutine in flowgraph to 2 (#28233) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-11-13 10:50:17 +08:00
smellthemoon 5365748338
Use single instance for mergedTimeTickerSender (#27730) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-11-13 10:18:17 +08:00
Xu Tong 8ec85f5f4c
Add template for VectorMemIndex (#28324) 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-11-11 13:20:22 +08:00
yah01 267c67dfee
enhance: reduce 1x copy while retrieving data from growing segment (#28323) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-10 15:44:22 +08:00
yah01 70995383bf
enhance: modify log to avoid ambiguity and improve readability (#28331) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-10 14:32:20 +08:00
zhagnlu 3920bbc55f
Force set aliyun use_virtual_host to true for all (#28158) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-11-10 10:32:20 +08:00
cai.zhang 01c5484817
No hit when the index exceeds the array length (#28301) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-10 10:04:19 +08:00
wei liu 8a370f115a
Fix external grpc server doesn't graceful stop (#28309) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-10 00:14:18 +08:00
wei liu 14c8a90517
Fix auto balance block channel reassign after datanode restart (#28275) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-09 19:00:25 +08:00
wei liu bce1054f92
Fix retry when proxy stopped (#28264) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-09 18:58:21 +08:00
yah01 30847cad3e
Handle exception while loading (#28304) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-09 17:59:12 +08:00
sre-ci-robot b7b31ce0bc
Update knowhere commit (#28285) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-11-09 14:24:19 +08:00
congqixia e576271a24
Fix buffer FieldData has no `ElementType` and array logsize always zero (#28295) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-09 14:16:20 +08:00
wei liu b9bf910039
fix unstable auto balance config ut (#28288) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-09 10:00:22 +08:00
wei liu 7f78e1dd46
fix datacoord unstable ut (#28281) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-08 18:43:31 +08:00
PowderLi 9f9726f8b7
print azure sdk log (#28240) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-08 17:50:18 +08:00
yah01 ecb3f585c3
Fix passing the wrong dropped list from current target (#28265) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-08 17:02:18 +08:00
yah01 d20ea061d6
Fix panic while forwarding empty deletions to growing segment (#28213) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-08 16:42:21 +08:00
yah01 f4341f254d
Not convert legacy error code to new merr (#28232) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-08 14:42:20 +08:00
yah01 1b90630633
Fix the target updated before version updated to cause data missing (#28250) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-08 11:36:22 +08:00
SimFG e3b7fdac61
Delay the cancellation of ctx when stopping the node (#28247) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-11-08 03:20:17 +08:00
wei liu 64f6dbabdc
Fix proxy not healthy before graceful stop (#28223) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-08 02:22:26 +08:00
cai.zhang 2b5f632fa4
Fix bug for constructing ArrayView with fixed-length type (#28185) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-07 23:38:21 +08:00
cai.zhang fd68837f84
Fix bug for setting index state when indexnode connecting failed (#28220) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-07 23:34:26 +08:00
PowderLi 7bb0fa9c70
reduce useless ObjectExists (#28156) 
replace ListBlobs() with GetProperties()
unified style std::string& / char*
config azure requestTimeoutMs

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-07 16:32:20 +08:00
congqixia af1c2044b9
Fix atomic.Int64 not found in go 1.18 (#28216) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-07 15:16:21 +08:00
groot 3f6b203018
Fix bulkinsert bug that segments are compacted after import (#28192) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-11-07 15:14:26 +08:00
wei liu 5b45a138b1
disable auto balance when old node exists (#28191) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-07 14:02:20 +08:00
wei liu da41a5b51e
fix check grpc error logic (#28182) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-07 11:54:18 +08:00
congqixia c41df18b6d
Add compaction meta in `SyncSegmentsRequest` (#28199) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-07 10:06:17 +08:00
XuanYang-cn 22c089894e
Add scheduler (#27938) 
See also: #27606

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-07 03:18:18 +08:00
yah01 ece592a42f
Deliver L0 segments delete records (#27722) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-07 01:44:18 +08:00
sre-ci-robot b1df3ead0e
Update knowhere commit (#28176) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-11-06 22:44:19 +08:00
XuanYang-cn 2bbc20c7b8
Update session manager (#27761) 
See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-06 22:10:19 +08:00
Xiaofan da19e49daf
Support purge old session for standalone (#28184) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-11-06 21:21:42 +08:00
yah01 863e26969a
Reduce 1x memory copy for retrieving data (#28106) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-06 17:48:19 +08:00
aoiasd 1d4be0d257
Adjust datacoord for L0 Delta (#28021) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-11-06 15:26:16 +08:00
chyezh 86973a8871
[Fixup] error root path for catalog when using tikv (#28163) 
Signed-off-by: chyezh <ye.zhen@zilliz.com>
 2023-11-06 12:06:19 +08:00
yah01 90e2c63d9e
Fix getting incorrect CPU num (#28146) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-11-06 06:02:16 +08:00
congqixia bf2f62c1e7
Add `WriteBuffer` to provide abstraction for delta policy (#27874) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-04 12:10:17 +08:00
cai.zhang 8011054a2a
Check length before comparing strings (#28110) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-04 10:04:29 +08:00
yihao.dai 8d080383b7
Use merr to prevent datanode panic (#28121) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-11-04 00:58:21 +08:00
congqixia 501cba90ce
Use IVFFLAT_CC instead of IVFFLAT for sealed segment tmp index (#28164) 
See also #28152

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-03 23:50:15 +08:00
wei liu 68a86471ba
fix grpc client retry on node server not match error (#28169) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-03 23:42:16 +08:00
wei liu 7485eeb689
fix sync distribution with wrong version (#28130) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-03 19:02:18 +08:00
Jiquan Long a21042dde7
Fix flushManager.isFull is too slow (#28141) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-11-03 14:42:17 +08:00
SimFG ef68680639
Fix the unstable resource manager testcase (#28033) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-11-03 11:00:17 +08:00
wayblink 00ae019ff0
Use go 1.20 csv_reader to keep milvus go=1.18 limitation (#28080) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-11-03 10:40:16 +08:00
aoiasd d2727cc0ff
Add write cache for access log (#27792) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-11-03 10:28:17 +08:00
sre-ci-robot 7f28e9d2f3
Update knowhere commit (#28087) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-11-03 10:24:16 +08:00
wei liu ecec5dfcfd
fix retry on offline node (#28079) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-03 10:14:16 +08:00
wei liu 86ec6f4832
fix load index for stopping node (#28047) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-03 07:58:18 +08:00
congqixia 1e51255c15
Implement `Injection` for SyncManager with block and meta transition (#28093) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-03 04:48:15 +08:00
cai.zhang fc2df9514f
Refine code for fixed-length types array (#28108) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-03 00:40:14 +08:00
yah01 dc89730a50
Support collection-level mmap control (#26901) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-02 23:52:16 +08:00
congqixia 5d2eba2c2f
Set qcv2 index task priority to Low (#28117) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-02 23:22:17 +08:00
wei liu 9b737b77a7
fix fromShardLeader typo (#28061) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-02 15:50:16 +08:00
Enwei Jiao f8dd589755
Refactor collection's cgo call (#28055) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-11-02 13:02:13 +08:00
yihao.dai 7bd44bd671
Fix unstable search task ut (#28101) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-11-02 12:52:15 +08:00
yihao.dai 873b29e226
Fix unstable cpp ut (#28083) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-11-02 00:58:16 +08:00
MrPresent-Han 854c0e8a16
fix precision for segcore reduce(#27325) (#28062) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-11-01 21:52:13 +08:00
Xiaofan af8ea0f3af
Revert "Remove Properties in rocksmq (#28000)" (#28071) 
This reverts commit 7c9d24cbb7.
 2023-11-01 21:44:20 +08:00
congqixia e4fdf5e68e
Refine offline segments logic in shard delegator (#28073) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-01 20:04:24 +08:00
groot abd5b199cc
Bulkinsert support pure list json (#27990) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-11-01 19:02:13 +08:00
PowderLi 0c0f012e03
add a custom http header: Accept-Type-Allow-Int64 (#27901) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-01 11:42:16 +08:00
wei liu 0c69f48ba4
refine stop order (#28016) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-01 11:40:15 +08:00
PowderLi 0252871d30
fix azure ListObjects (#27931) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-01 11:34:14 +08:00
aoiasd 2aa330146a
Delete by complex expression use partitionKey optimization (#27952) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-11-01 11:26:16 +08:00
Enwei Jiao 8ae9c947ae
Use OpenDAL to access object store (#25642) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-11-01 09:00:14 +08:00
cqy123456 4fbe3c9142
replace loaded binlog with binlog index for search performance (#27673) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-11-01 02:20:15 +08:00
KumaJie e88212ba4b
Add CSV file import function (#27149) 
Signed-off-by: kuma <675613722@qq.com>
Co-authored-by: kuma <675613722@qq.com>
 2023-10-31 22:47:23 +08:00
wei liu 0677d2623d
remove retry in getShards (#28011) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-10-31 16:16:14 +08:00
yah01 1b9b8a31a8
Fix QueryNode panic while upgrading (#28034) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-31 15:50:13 +08:00
yah01 ab6dbf7659
Limit max thread num for pool (#28018) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-31 13:00:13 +08:00
yah01 9658367a3c
Refine chunk manager errors (#27590) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-31 12:18:15 +08:00
Xiaofan 38aa53f523
Fix datacoord start timeout (#28026) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-10-31 10:04:12 +08:00
yah01 2af46d7333
Increase the ChunkManager request timeout (#28015) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-31 09:06:13 +08:00
congqixia 233bf90c55
Add SyncManager to replace flush manager (#27873) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-31 02:30:16 +08:00
Xiaofan 7c9d24cbb7
Remove Properties in rocksmq (#28000) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-10-31 00:44:14 +08:00
yah01 9b6eeb46f1
Cache segment row num, size, and insert count to reduce CGO calls (#28007) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-30 17:54:14 +08:00
yihao.dai ab6b0103a3
Get vector concurrently (#27838) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-10-30 15:44:12 +08:00
MrPresent-Han f93ad6471c
enable query reduce on querynodes for shortcut return(#27306) (#27837) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-10-30 15:08:11 +08:00
Ikko Eltociear Ashimine ed4f20b0ed
Fix typo in util.go (#27975) 
suppot -> support

Signed-off-by: Ikko Eltociear Ashimine <eltociear@gmail.com>
 2023-10-30 14:40:27 +08:00
wei liu 7e0bda8b4c
remove binlog/delta log from getRecoveryInfoV2 (#27895) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-10-30 14:28:15 +08:00
congqixia 36ddeae2cc
Refine log level when request resource fail for loading segments (#28004) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-30 14:20:14 +08:00
Jiquan Long 6913283716
Fill output fields for count request (#27967) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-10-30 10:00:12 +08:00
congqixia 98e2aad752
Refine datanode metacache and implement CoW (#27985) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-28 11:12:11 +08:00
MrPresent-Han ccaec44930
enable parameter for dataCoord broker timeout(#26972) (#27825) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-10-28 06:14:11 +08:00
sre-ci-robot 1ae6e5d8c8
Update knowhere commit (#27993) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-10-28 04:24:10 +08:00
groot e6d4cde397
Fix duplicated segment id in bulkinsert task state (#27915) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-10-27 15:14:13 +08:00
yah01 f212158d61
Fix delete records timestamp may be reordered (#27941) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-27 10:24:10 +08:00