milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
20,154 Commits
153 Branches
122 Tags
592 MiB
Commit Graph

1038 Commits (be234952335dd83d1f80d4aa36ab30e079c232e2)

Author SHA1 Message Date
congqixia b1eb1ea506
Refine datanode Timetick Sender (#28393) 
- Use explicit lifetime control methods: `Start` and `Stop`
- Allow control retry option
- Make sure tt sender worker exit after `Stop` return

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-13 21:46:20 +08:00
smellthemoon 0aa90de141
Reduce the goroutine in flowgraph to 2 (#28233) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-11-13 10:50:17 +08:00
smellthemoon 5365748338
Use single instance for mergedTimeTickerSender (#27730) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-11-13 10:18:17 +08:00
yah01 f4341f254d
Not convert legacy error code to new merr (#28232) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-08 14:42:20 +08:00
SimFG e3b7fdac61
Delay the cancellation of ctx when stopping the node (#28247) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-11-08 03:20:17 +08:00
congqixia af1c2044b9
Fix atomic.Int64 not found in go 1.18 (#28216) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-07 15:16:21 +08:00
groot 3f6b203018
Fix bulkinsert bug that segments are compacted after import (#28192) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-11-07 15:14:26 +08:00
congqixia c41df18b6d
Add compaction meta in `SyncSegmentsRequest` (#28199) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-07 10:06:17 +08:00
yah01 90e2c63d9e
Fix getting incorrect CPU num (#28146) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-11-06 06:02:16 +08:00
congqixia bf2f62c1e7
Add `WriteBuffer` to provide abstraction for delta policy (#27874) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-04 12:10:17 +08:00
yihao.dai 8d080383b7
Use merr to prevent datanode panic (#28121) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-11-04 00:58:21 +08:00
Jiquan Long a21042dde7
Fix flushManager.isFull is too slow (#28141) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-11-03 14:42:17 +08:00
congqixia 1e51255c15
Implement `Injection` for SyncManager with block and meta transition (#28093) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-03 04:48:15 +08:00
yah01 9658367a3c
Refine chunk manager errors (#27590) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-31 12:18:15 +08:00
congqixia 233bf90c55
Add SyncManager to replace flush manager (#27873) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-31 02:30:16 +08:00
congqixia 98e2aad752
Refine datanode metacache and implement CoW (#27985) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-28 11:12:11 +08:00
smellthemoon 403c6680cc
Add error description (#27959) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-10-27 10:10:13 +08:00
Filip Haltmayer 6b1a106a31
Moving etcd client into session (#27069) 
Signed-off-by: Filip Haltmayer <filip.haltmayer@zilliz.com>
 2023-10-27 07:36:12 +08:00
yah01 f79c7370f4
Fix panic while flushing dropped/compacted segment (#27927) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-25 22:10:13 +08:00
yihao.dai b9d5ef3599
Fix datanode ttNode goroutine leak (#27878) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-10-24 18:22:10 +08:00
aoiasd 9091a27832
Add meta cache to datanode for L0 Delta (#27768) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-10-23 19:42:10 +08:00
SimFG 9b0ecbdca7
Support to replicate the mq message (#27240) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-10-20 14:26:09 +08:00
smellthemoon 4b0ec156b3
Set channel work pool size in datanode (#27728) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-10-19 08:28:08 +08:00
XuanYang-cn 7358c3527b
Add iterators (#27643) 
See also: #27606

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-10-18 19:34:08 +08:00
jaime e386a62fae
Remove recollect segment stats during starting datacoord (#27410) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-10-16 10:26:09 +08:00
jaime ec1fe3549e
Add a stop hook to clean session (#27564) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-10-16 10:24:10 +08:00
congqixia 82b2edc4bd
Replace manual composed grpc call with Broker methods (#27676) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-13 09:55:34 +08:00
yah01 be980fbc38
Refine state check (#27541) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-11 21:01:35 +08:00
aoiasd ec830e1240
fix use same logidx in different field (#27492) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-10-11 18:49:33 +08:00
congqixia cbb350c552
Add broker for datanode grpc operations (#27631) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-11 17:03:34 +08:00
yah01 2df9908c6a
Map old error code to new error (#27616) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-11 12:43:33 +08:00
jaime 86ed2fd51f
[skip e2e] Fix unstable ut (#27618) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-10-11 07:21:32 +08:00
XuanYang-cn 294ff74ca5
Add new compaction types (#27608) 
See also: #27606

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-10-11 06:49:32 +08:00
XuanYang-cn 56c94cdfa7
Add channel manager in DataNode (#27308) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-10-08 21:37:33 +08:00
congqixia 5d558623fe
Add revive sub-lints and fix existing problems (#27495) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-07 20:53:38 +08:00
congqixia 80eb5434d3
Fix var-naming caused by old PRs (#27501) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-07 14:43:32 +08:00
yah01 8394b3a1ec
Block creating new error from status reason (#27426) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-07 11:29:32 +08:00
Jiquan Long 0f14d18201
Optimize the codec code of session (#27360) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-10-01 10:33:30 +08:00
XuanYang-cn 5c5f9aa05e
Enhance newDataSyncService (#27277) 
- Add flowgraph.Assemble assembles nodes in flowgraph.go
- remove fgCtx in newDataSyncService
- Add newServiceWithEtcdTickler func, reduce param numbers to 3
- Remove unnecessary params
  - config.maxQueueLength, config.maxParallelish

See also: #27207

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-27 11:07:25 +08:00
yah01 6539a5ae2c
Refine DataCoord status (#27262) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-26 17:15:27 +08:00
wayblink 7dd0be1b2c
Enable bulkinsert binlog data with partitionkey (#27241) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-09-26 10:25:25 +08:00
jaime 7f7c71ea7d
Decoupling client and server API in types interface (#27186) 
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-09-26 09:57:25 +08:00
XuanYang-cn 676024ff38
Add rpc to notify channel operation (#27172) 
- Add NotifyChannelOperation
- Add CheckChannelOperationProgress

See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-25 14:05:27 +08:00
congqixia 1d76565894
Add metrics for garbage collection (#27303) 
Also fix second metrics usage in compaction

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-22 18:47:25 +08:00
SimFG 26f06dd732
Format the code (#27275) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-21 09:45:27 +08:00
XuanYang-cn 916528f51a
Add initCtx in getDataSyncService (#27199) 
Passing initCtx to all IO funcs in newDataSyncService,
so when ctx.Canceled, newDataSyncService would return.

See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-20 16:03:23 +08:00
yah01 b4f86ea55e
Construct all success status with merr (#27226) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-20 10:57:23 +08:00
yah01 338848fbc3
Remove unused code and dup imports (#27228) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-20 10:55:23 +08:00
Xiaofan 6635398a6d
Fix Bin log concurrency by adding a pool (#27189) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-09-19 18:05:22 +08:00
XuanYang-cn 09505ea78e
Move etcd watch related code into eventmanager (#27192) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-19 15:27:38 +08:00
XuanYang-cn 10116b85ac
Release before remove when releasing flowgraphs (#27191) 
GetAndRemove removes the fg from manager immediately,
while the flowgraph is still releasing. This PR will remove
the fg from flowgraphManager AFTER flowgraphs released.

- Add Remove for ConcurrentMap
- Move collections() into flowgraph manager

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-19 11:53:22 +08:00
yihao.dai 4b2802033d
Fix datanode panic due to concurrent compaction and delete processing (#27167) 
Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-09-18 19:35:29 +08:00
yah01 a6b98740b7
Replace all status with only error string (#27125) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-18 14:05:28 +08:00
yihao.dai c162c6a4c8
Increase FlushChannel retry times (#27140) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-16 00:41:20 +08:00
yah01 168e82ee10
Fix panic while handling with the nil status (#27040) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-15 10:09:21 +08:00
yihao.dai dd2cb1d44a
Flush by flush channels (#26859) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-12 21:07:19 +08:00
yah01 00c65fa0d7
Refine QueryNode errors (#27013) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-12 16:07:18 +08:00
Xu Tong 9166011c4a
Add float16 vector (#25852) 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-09-08 10:03:16 +08:00
XuanYang-cn 7f1ae35e72
Add timeout in dispatcher, AsConsumer and Seek (#26686) 
See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-08 09:51:17 +08:00
yiwangdr 337edc321b
tikv integration (#26246) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-09-07 07:25:14 +08:00
bjzhjing 548c82eca5
Refactor storage.MergeInsertData() to optimize the merging process (#26839) 
Benchmark Milvus with https://github.com/qdrant/vector-db-benchmark and
specify the datasets as 'deep-image-96-angular'. Meanwhile, do perf
profiling during 'upload + index' stage of vector-db-benchmark and see
the following hot spots.

39.59%--github.com/milvus-io/milvus/internal/storage.MergeInsertData
        |
        |--21.43%--github.com/milvus-io/milvus/internal/storage.MergeFieldData
        |          |
        |          |--17.22%--runtime.memmove
        |                     |
        |                     |--1.53%--asm_exc_page_fault
        |                     ......
        |
        |--18.16%--runtime.memmove
                   |
                   |--1.66%--asm_exc_page_fault
                   ......

The hot code path is in storage.MergeInsertData() which updates
buffer.buffer by creating a new 'InsertData' instance and merging both
the old buffer.buffer and addedBuffer into it. When it calls golang
runtime.memmove to move buffer.buffer which is with big size (>1M), the
hot spots appear.

To avoid the above overhead, update storage.MergeInsertData() by
appending addedBuffer to buffer.buffer, instead of moving buffer.buffer
and addedBuffer to a new 'InsertData'. This change removes the hot spots
'runtime.memmove' from perf profiling output. Additionally, the 'upload
+ index' time, which is one performance metric of vector-db-benchmark,
is reduced around 60% with this change.

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
 2023-09-05 21:41:48 +08:00
congqixia fe7f7ea237
Add cpp-build and setenv for querynode mockery regen command (#26843) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-05 14:13:49 +08:00
Enwei Jiao fb0705df1b
Decouple basetable and componentparam (#26725) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-05 10:31:48 +08:00
XuanYang-cn b2e7cbdf4b
Remove TimeTravel in compactor (#26785) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-04 17:41:48 +08:00
yihao.dai 4340cbfba2
Merge syncCPLagTooBehind policy into syncPeriodically policy (#26713) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-04 14:37:47 +08:00
yihao.dai 7624c2b949
Improve rated log in insertBufferNode (#26788) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-04 09:59:10 +08:00
yah01 3349db4aa7
Refine errors to remove changes breaking design (#26521) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-04 09:57:09 +08:00
congqixia e8f1b1736e
Remove log.Error(err.error())-style log (#26783) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-01 13:09:01 +08:00
XuanYang-cn 8d54509e54
Fix CompactionLatency metrics (#26747) 
- Refine compactor logs

See also: #26743

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-31 17:35:03 +08:00
smellthemoon 6069a7d42b
Log out memory in MB (#26262) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-08-30 17:13:00 +08:00
congqixia cc584551fb
Remove unused hash value setting in DDNode (#26575) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-28 11:30:26 +08:00
SimFG b9bc6681ae
Use atomic.bool for the isWatchFailed param in the tickler (#26558) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-08-23 13:22:27 +08:00
XuanYang-cn 27691e843c
Fix channel checkpoint stuck (#26534) 
See also: #23621

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-22 18:10:22 +08:00
XuanYang-cn 08fd28b30b
Only do gracefully stop when DN Stop (#26399) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-21 11:16:20 +08:00
SimFG 3be4ac4022
Fix datanode/datacoord continuous restart (#26470) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-08-20 21:20:24 +08:00
Enwei Jiao 533f0ddf6d
Add amazonlinux image, prepare to remove Centos7 image, remove openblas image (#26368) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-18 18:32:19 +08:00
congqixia fbb5d32cb6
Make write binlog in parallel (#26325) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-17 15:44:18 +08:00
congqixia f371ec0316
Add data sync service close log for watch stuck debugging (#26332) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-16 10:57:32 +08:00
Enwei Jiao 78bc688d16
Remove QueryMsgStream in MqFactory interface (#26374) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-16 10:29:33 +08:00
congqixia f8bcf60e55
Adjust some confusing Warning log to INFO (#26356) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-15 17:49:33 +08:00
wayblink 21eeb37ce9
Save binlog timestampFrom, timestampTo meta when compact (#26203) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-08-08 21:17:21 +08:00
congqixia b9850ce5c0
Fix copylocks linter errors (#26217) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-08 20:41:15 +08:00
MrPresent-Han b84f5c560a
refine log out for datanode(#25763) (#26195) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-08 16:49:08 +08:00
MrPresent-Han fb933fe64d
fix sync memory policy lose effect when inserting too many partitions(#25763) (#26154) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-07 09:59:07 +08:00
XuanYang-cn 830f06783a
Reduce log volumes of DC&DN (#26060) 
See also: #26057

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-02 17:13:11 +08:00
XuanYang-cn 84253f255e
Fix datanode graceful stop panic (#25932) 
See also: #25925

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-07-28 10:11:08 +08:00
groot a6808e6484
Fix bulkinsert row count error (#25869) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-07-25 11:29:00 +08:00
cai.zhang 779d677eb2
Remove lock for compaction sync segments (#25711) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-07-24 17:25:00 +08:00
jaime 8e5d6193f9
Add a timeout config for bulkinsert request (#25789) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-07-24 14:23:00 +08:00
congqixia 3c503afe7c
Use typeutil.ConcurrentMap instead of sync.Map (#25846) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-24 10:23:01 +08:00
congqixia 31b40b3ccc
Use typeutil.ConcurrentMap instead of sync.Map in fg manager (#25835) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-24 10:21:07 +08:00
Enwei Jiao 66fdc71479
Refactor logs in DataCoord & DataNode (#25574) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-07-14 15:56:31 +08:00
yiwangdr b9189b9f41
Organize mocks from types.go (#25466) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-07-14 10:12:31 +08:00
wayblink 53b4b70316
Reduce log frequency in timetick_sender (#25532) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-07-13 14:38:29 +08:00
wei liu 012fd1152a
fix datacoord consume datanode tt metrics (#25524) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-13 14:26:29 +08:00
congqixia dbfade77c5
Retry on SyncSegment failure (#25540) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-13 14:12:29 +08:00
wayblink ed3e4b0bc5
Fix unstable ut in timetick_sender_test (#25539) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-07-13 10:04:29 +08:00
MrPresent-Han ea3817fbbc
refinement towards sync_cp_lag_too_behind_policy to avoid submit syncTasks too frequently(#25441) (#25442) (#25500) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-07-12 20:22:29 +08:00
SimFG f9e2d00f91
Prevent `exclusive consumer` exception in pulsar (#25376) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-07-12 17:26:30 +08:00
wayblink fc12d3997c
Rename newTimeTickManager to newTimeTickSender (#25415) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-07-12 14:56:28 +08:00
groot 96c987ed62
Bulkinsert supports partition keys (#25284) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-07-11 15:18:28 +08:00
wei liu 71d99d4ec2
fix unexpected metrics value cause by wrong time unit (#25240) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-06 08:48:27 +08:00
Xiaofan 63b6a4a639
use single instance ppol (#25159) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-28 14:54:45 +08:00
xige-16 33c2012675
Add more metrics (#25081) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-26 17:52:44 +08:00
yiwangdr c7b851f870
add interface for non-watch metakv (#25092) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-06-26 09:20:44 +08:00
congqixia 736916222b
Stop retry lazy load stats after segment released (#25054) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-25 19:20:43 +08:00
jaime 18df2ba6fd
[Cherry-Pick] Support Database (#24769) 
Support Database(#23742)
Fix db nonexists error for FlushAll (#24222)
Fix check collection limits fails (#24235)
backward compatibility with empty DB name (#24317)
Fix GetFlushAllState with DB (#24347)
Remove db from global meta cache after drop database (#24474)
Fix db name is empty for describe collection response (#24603)
Add RBAC for Database API (#24653)
Fix miss load the same name collection during recover stage (#24941)

RBAC supports Database validation (#23609)
Fix to list grant with db return empty (#23922)
Optimize PrivilegeAll permission check (#23972)
Add the default db value for the rbac request (#24307)

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-06-25 17:20:43 +08:00
Xiaofan e8911ebda7
Add retry time when lazy load BF (#25096) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-25 11:32:43 +08:00
Xiaofan 72c5e2a41a
Fix channel reassigned to other datanodes (#25015) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-21 21:26:42 +08:00
aoiasd 7f5de9aab3
add read lock for getSegment (#25036) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-06-21 14:06:42 +08:00
PowderLi 3f4356df10
fix the spelling of `field` (#25008) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-06-21 14:00:42 +08:00
yihao.dai 6168a55c04
Lower the log frequency when sync tasks are skipped (#25006) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-06-20 17:56:41 +08:00
jaime b75dcf90c0
Fix Flush hang after SyncSegments timeout (#24953) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-06-20 10:20:41 +08:00
Xiaofan af1d84e5e1
Support sync when cp is lagged (#24989) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-20 10:00:41 +08:00
Xiaofan f57fe6d70b
Add compaction log (#24976) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-19 14:18:41 +08:00
yihao.dai c73219a54d
Limit the number of concurrent sync tasks and allow only one sync task for the same segment (#24881) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-06-16 14:14:39 +08:00
congqixia d186f35895
Fix TestChannelMetaMockSkipBFLoad/transient_error not stable (#24928) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-16 09:50:44 +08:00
wayblink bfae6b49af
Remove datanode timetick mq, use rpc to report instead (#23156) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-06-14 14:16:38 +08:00
groot 5c3b744b0c
Fix potential crash bug of bulkinsert (#24763) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-06-13 10:22:38 +08:00
congqixia 41af0a98fa
Use go-api/v2 for milvus-proto (#24770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
yah01 ebd0279d3f
Check error by Error() and NoError() for better report message (#24736) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-08 15:36:36 +08:00
congqixia 31880ab427
Use conc.PoolOption instead of ants.Option (#24585) 
- Add conc.PoolOption to setup conc.Pool
- Change panic default behavior
- Make future has error when job panicks

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-01 15:52:31 +08:00
wayblink 5958b42941
Add retry on getChannelLatestMsgID (#24518) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-05-30 20:03:28 +08:00
aoiasd c84bdcea49
merge stats log when segment flushing or compacting (#23570) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-05-29 10:21:28 +08:00
Xiaofan 49f75e5e11
Make flow graph retry longer (#24355) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-05-25 09:53:30 +08:00
congqixia 03297a7ef7
Fix event manager not closed after datanode stop (#24206) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-18 11:43:23 +08:00
yah01 99cc24b283
Fix compaction doesn't support JSON data (#24151) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-05-17 19:09:22 +08:00
congqixia 73a181d226
Fix get vector it timeout and improve some string const usage (#24141) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-16 17:41:22 +08:00
congqixia 084424d636
Make data node start only once (#24031) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-11 13:51:20 +08:00
congqixia 345855c984
Make datanode load statslog lazy if SkipBFStatsLog is true (#23877) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-06 15:12:39 +08:00
MrPresent-Han b517bc9e6a
refine balance mechanism including:(#23454) (#23763) (#23791) 
1. balance granuity to replica to avoid influence unrelated replicas
2. avoid balance back and forth

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2023-05-04 12:22:40 +08:00
congqixia 796f9355ee
Fix global rateCollector init multiple times (#23703) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-26 10:16:36 +08:00
wei liu 4fb8919a97
support collection level rate limit (#22767) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-04-25 15:54:35 +08:00
zhenshan.cao 03ec804e68
Change default shard number to 1 (#23586) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-04-21 07:08:32 +08:00
congqixia 4a6c2b78a7
Fix multiple transferNewSegment and removeSegment causes panic (#23564) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-20 13:48:31 +08:00
Enwei Jiao 967a97b9bd
Support json & array types (#23408) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
 2023-04-20 11:32:31 +08:00
congqixia 4fe363c4b2
Add KNN cgo pool (#23526) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-19 23:56:31 +08:00
congqixia 4508786819
Fix test_watch_channel unit test unstable (#23507) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-19 10:00:31 +08:00
wei liu cbfe7a45ef
fix pull target (#23491) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-04-18 18:30:32 +08:00
wayblink b23723535c
Add log in AddImportSegment.getChannelLatestMsgID (#23480) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-04-18 17:50:31 +08:00
zhenshan.cao 4a32b842e8
Improve the check logic of channel remove (#23473) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-04-18 02:58:30 +08:00
aoiasd 695cc769df
Fix watch event timeout when IO Pool busy (#23293) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-04-17 11:02:30 +08:00
aoiasd f603456b72
remove segment from datanode meta when segment dropped (#23398) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-04-17 10:58:30 +08:00
congqixia ba84f52119
Fix watcher loop quit and channel shouldDrop logic (#23402) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-14 09:54:28 +08:00
cai.zhang 43a9e175a3
Exit component process when session key is deleted (#21658) (#22164) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-04-12 20:12:28 +08:00
XuanYang-cn b90fa5f459
Refine codes of datanode buffer (#23168) 
This PR refines deltabuffer of datanode:
- Add last sync time for compacted segment, see also: #23210
- Ensure all deltabuffermanager handles all delete related operations
- Change usedMemory to atomic.Int64
- Remove allocator in delete buffer

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-04-10 18:42:30 +08:00
jaime c9d0c157ec
Move some modules from internal to public package (#22572) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-04-06 19:14:32 +08:00
aoiasd 1a485044a9
fix NumConsumers metric not work (#23167) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-04-05 08:18:28 +08:00
smellthemoon ce91f61e5a
Pre-create topics (#21907) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-04-03 16:44:23 +08:00
XuanYang-cn be0827417a
Fix buffer calculate memorysize negative (#23152) 
Along with the following small patches:
- And fix logging dup segmentIDs for delete buffer status
- Add String method for PriorityQueue to debug

See also: #23105

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-03-31 18:30:23 +08:00
Jiquan Long 0f2c3ac832
Optimize log (#23121) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-03-30 10:44:22 +08:00
Enwei Jiao 50dab2d394
Add tracing for retrieve (#23033) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-03-29 18:10:02 +08:00
XuanYang-cn 9691315c21
Fix flush deltabuf with segmentID=0 (#23059) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-03-28 19:14:01 +08:00
aoiasd 5d172d0f4f
Refine errors of datanode (#22852) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-03-28 19:04:00 +08:00
zhenshan.cao e3c3c949c4
Fix datarace of flow_graph_manager_test.go in package datanode (#23043) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-03-28 10:42:00 +08:00
yihao.dai b21f03682a
Fix flowgraph manager test (#23009) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-03-27 18:00:05 +08:00
congqixia 4008ee6d60
Remove datanode ddNode forward delta logic (#23005) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-27 14:18:00 +08:00
yah01 081572d31c
Refactor QueryNode (#21625) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-03-27 00:42:00 +08:00
Xiaofan 5a914aedf3
Make Datacoord TT to be configurable (#22956) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-03-24 22:24:04 +08:00
XuanYang-cn da58ba6a49
Fix endless appending sids (#22977) 
See also: #22976

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-03-24 18:00:03 +08:00
congqixia 8f847884c2
Fix misc copylocks and remove exclude rule (#22962) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-24 11:17:58 +08:00
XuanYang-cn 93bc805933
Enhance ID allocator in DataNode (#22905) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-03-23 19:43:57 +08:00
yihao.dai f2ffb5db26
Use buffer size at memory sync policy (#22825) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-03-21 21:37:56 +08:00
SimFG 4a90490a67
Fix the `segment not found` error (#22772) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-03-17 17:27:56 +08:00
smellthemoon 8799b06dbc
Fix the data is disappeared when upsert (#22762) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-03-14 19:11:54 +08:00
congqixia 732986aa04
Remove fmt.Print from internal package (#22722) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-14 17:36:05 +08:00
aoiasd 6959332b08
Fix unittest start channel event but not close (#22662) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-03-14 17:05:54 +08:00
wayblink aa6212d5b0
[Cherry-pick] Fix channel checkpoint issues and fix GC (#22559) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: Ten Thousand Leaves <69466447+soothing-rain@users.noreply.github.com>
 2023-03-09 14:13:52 +08:00
jaime d126f06946
Decouple mq module from internal proto definition (#22536) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-03-04 23:21:50 +08:00
yah01 60cd548bf5
Rename concurrency package to conc (#22453) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-02-28 14:19:47 +08:00
yah01 167581c08a
Refine Pool to improve the ease of use (#22412) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-02-27 18:07:48 +08:00
wayblink 416866e42f
Add memory usage too large sync policy (#22244) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-02-27 17:47:51 +08:00
yah01 b758c305a7
Make Future generic (#22411) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-02-27 12:07:47 +08:00
Xiaofan 3d6bded115
Fix Row Number Mismatch (#22307) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-02-27 10:41:46 +08:00
Enwei Jiao 697dedac7e
Use cockroachdb/errors to replace other error pkg (#22390) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-02-26 11:31:49 +08:00
zhenshan.cao e768437681
Correct usage of Timer and Ticker (#22228) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-02-23 18:59:45 +08:00
aoiasd 27b456dd46
Fix data race of data node test (#22221) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-02-22 10:28:26 +08:00
wei liu 0ce70b4f10
fix compact task data race (#22299) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-02-21 16:10:26 +08:00
Enwei Jiao 0a9a9058b9
Fix UT timeout (#22261) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-02-20 21:14:25 +08:00
aoiasd 9167773eff
Remove needless kv at dataSyncService (#22206) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-02-16 16:36:39 +08:00
congqixia 9346f1752a
Convert msg pchannel to vchannel before check IsCloseMsg (#22182) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-02-15 17:26:34 +08:00
aoiasd 148a024e05
Add tickle for datacoord watch event (#21193) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-02-15 16:20:34 +08:00
aoiasd 1894766235
Update datacoord compaction plan after datanode update plan to ensure consistency (#22143) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-02-15 16:00:33 +08:00
smellthemoon e20d79a8a1
Fix data race (#22171) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-02-15 15:40:33 +08:00
bigsheeper d2667064bb
Add msgDispatcher to support sharing msgs for different vChannel (#21917) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-02-13 16:38:33 +08:00
cai.zhang 511265c68c
Modify lastSyncTime in advance to prevent multiple flush binlogs (#22048) (#22088) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-02-12 18:56:33 +08:00
XuanYang-cn 2b982b5d8f
Fix DataNode ut never meet condition (#22093) 
See also: #22079

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-02-10 11:04:32 +08:00
yah01 9b491858a5
Fix datanode unit test not stable (#21911) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-02-01 18:19:50 +08:00
Xiaofan 949d5d078f
Fix memory calculation in dataCodec (#21800) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-01-28 11:09:52 +08:00
congqixia 5986106037
Make paramtable init only once (#21782) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-01-19 14:53:44 +08:00
Enwei Jiao 90d9e165d4
Fix some configs not shown (#21653) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-01-13 15:31:41 +08:00
wayblink 6a722396bd
Integration test framework (#21283) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-01-12 19:49:40 +08:00
Enwei Jiao fb42466c65
Use opentelemetry (#21509) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-01-12 16:09:39 +08:00
groot 9f8095996b
Report bulkinsert progress (#21612) (#21638) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2023-01-11 17:37:44 +08:00
groot df0ffd08ce
Update segment id for import task (#21621) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2023-01-11 11:55:39 +08:00
Xiaofan 4b4944ecee
Support sync all segments while close (#21421) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-01-06 14:49:36 +08:00
cai.zhang efefba4da4
Fix checkpoint will not update after delete and compact (#21495) (#21499) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-05 11:31:35 +08:00
yah01 7b39873ae0
limit the frequency of GetMetrics() log (#21514) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-01-04 17:39:35 +08:00
Jiquan Long ff2a68e65a
Fix collection not exist when tried to do recovery (#21471) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-01-04 16:37:35 +08:00
congqixia 3a58ac9fba
Change node stop to TearDownTest (#21422) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-28 20:11:30 +08:00
Xiaofan 3667bb09b8
Avoid Sync when the segment open (#21397) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-12-28 10:51:34 +08:00
congqixia 3c9840d6d3
Fix some datanode instance not stopped in unit tests (#21385) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-26 16:59:30 +08:00
Enwei Jiao 166e9f0da5
Refactor GrpcConfig (#21142) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-16 15:59:23 +08:00
bigsheeper f595500383
Move sync delete policy from deleteNode to insertBufferNode (#21152) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-12-13 16:15:21 +08:00
XuanYang-cn a7b24cbc53
Move APIs of types.DataNode to services.go (#21042) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-12-12 10:57:22 +08:00
Xiaofan e977e014a9
Fix flush didn't respect binaryvector and other schemas (#21120) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-12-12 10:33:26 +08:00
aoiasd de0ab9e2cf
Refactor showConfigurations to allow return global config rather than only return config of this component (#21063) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-12-09 14:31:21 +08:00
Enwei Jiao 89b810a4db
Refactor all params into ParamItem (#20987) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-07 18:01:19 +08:00
groot 33a102c6fc
Refine bulkinsert (#20986) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-12-07 14:53:18 +08:00
XuanYang-cn f844aa0cab
Remove not inuse stale segments in flushmsg (#20981) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-12-06 17:51:18 +08:00
jaime 039e9ce4bb
Refine msgstream interface (#20832) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-12-05 20:55:17 +08:00
congqixia 2791f5526c
Fix BufferData limit calculation for BinaryVector (#20963) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-05 11:15:17 +08:00
congqixia 4238b5dbab
Make datanode unittest use embed etcd server (#20948) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-02 16:05:16 +08:00
congqixia 40abb13413
Generate unique id with ts for sync task (#20928) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-01 20:35:18 +08:00
congqixia f745d7f489
Fix compaction target segment rowNum is always 0 (#20937) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-01 20:33:17 +08:00
zhenshan.cao 9724ae5e27
Refine log level (#20904) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-12-01 20:07:15 +08:00
Ten Thousand Leaves b238f4ee58
Remove bad & duplicate metric (#20905) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-12-01 16:39:16 +08:00
congqixia 82fe655032
Setup lastSyncTs when add segment to avoid sync next timetick (#20920) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-01 12:29:16 +08:00
Enwei Jiao 2ecdb4ba4a
Etcd config source support TLS (#20874) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-30 18:23:15 +08:00
cai.zhang 83244afcf3
Reduce some logs level which is frequent (#20894) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-11-30 14:07:15 +08:00
SimFG 44d45452fa
Make the sync segment request idempotent in the data node (#20707) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-11-24 10:21:13 +08:00
groot bf2107ecf5
Fix a regression of local storage (#20653) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-11-21 10:19:10 +08:00
groot c8022ea63c
Change import task timeout value (#20717) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-11-21 10:11:11 +08:00
jaime 0a5270e8c0
Fix alter collection hang (#20694) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-11-21 10:09:10 +08:00
Bingyi Sun 2390095232
Fix load uses compacted segments' binlogs (#20655) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-11-17 20:37:10 +08:00
Enwei Jiao c05b9ad539
Add event dispatcher for config (#20393) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-17 18:59:09 +08:00
Enwei Jiao 19524a5344
Fix nodeID mismatch at standalone mode (#20648) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-17 17:15:08 +08:00
XuanYang-cn bd20536c6a
Reduce unnecessary logs of get compaction results (#20642) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-11-17 10:09:12 +08:00
bigsheeper 2d307c1909
Set delBuf's start-end position of compacted segment (#20614) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-11-15 19:41:16 +08:00
MrPresent-Han c848896bc8
Fix panic error due to rollDeleteOp ahead by load-and-delete (#20563) 
issue:#20501
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2022-11-15 14:45:07 +08:00
bigsheeper db33ffa518
Pass endPosition with vchannel in datanode flowgraph (#20589) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-11-15 14:37:07 +08:00
smellthemoon a7ba416b5a
Remove invalid timestamp (#20557) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-11-14 16:21:07 +08:00
bigsheeper fc15789da9
Ensure compatibility of channel seek position and move syncPeriod to config (#20504) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-11-12 21:09:04 +08:00
bigsheeper cd19d99ad7
Add channel level checkpoint (#20350) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-11-10 22:13:04 +08:00
XuanYang-cn 6bff0b6688
Remove too frequent logs (#20454) 
Half of the log messages come from this log if DataNode in stale.

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-11-10 14:51:05 +08:00
Enwei Jiao 7e56e7a976
Fix datarace at Setlogger (#20376) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-08 14:33:03 +08:00
wayblink a7bed1c927
Pass backup flag through Import request options (#20334) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-11-08 11:33:03 +08:00
MrPresent-Han 21b54709a2
enable delbuf auto flush function to avoid OOM when del msg accumulating (#20213) 
issue: #19713

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
Co-authored-by: MrPresent-Han <zilliz@zillizdeMacBook-Pro-2.local>
 2022-11-07 18:49:02 +08:00
jaime 709a0a94e9
Add some msgstream metrics (#20296) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-11-07 10:15:02 +08:00
Enwei Jiao 956c5e1b9d
Make Params singleton (#20088) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-04 14:25:38 +08:00
congqixia f557af1143
Verify target id for FlushSegments request in datanode (#20308) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-11-03 22:47:35 +08:00
bigsheeper d528aa3d90
Print seek postion time and seek elapse (#20271) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-11-03 15:03:35 +08:00
MrPresent-Han e0baf1536e
[skip e2e] Improve a commment for initial pr (#20287) 
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2022-11-03 14:05:36 +08:00
Ten Thousand Leaves 5708352a80
Improve DataCoord and DataNode logs (#20265) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-11-03 12:15:35 +08:00
Xiaofan 33349677fc
Fix unsafe lock (#20219) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-11-01 19:37:35 +08:00
Xiaofan 99b958e360
Fix BF Concurrency issue (#20211) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-10-31 19:27:35 +08:00
Xiaofan 2bfecf5b4e
Refine bloomfilter and memory usage (#20168) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-10-31 17:41:34 +08:00
jaime 48f5c60070
Refine msgstream close (#20194) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-10-31 15:13:33 +08:00
congqixia a8e2a69aa2
Fix mergeSyncTask logic (#20185) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-10-29 18:47:33 +08:00
XuanYang-cn d5bc4e2585
Make DN able to process syncing request in batch (#20126) 
See also: #19834

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-10-27 21:25:33 +08:00