milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
21,567 Commits
153 Branches
122 Tags
591 MiB
Commit Graph

1140 Commits (8fadcde403067812d3b0a7822d5178d980306e65)

Author SHA1 Message Date
congqixia dc6a6a50fa
enhance: reduce SyncTask AllocID call and refine code (#29701) 
See also #27675

`Allocator.Alloc` and `Allocator.AllocOne` might be invoked multiple
times if there were multiple blobs set in one sync task.

This PR add pre-fetch logic for all blobs and cache logIDs in sync task
so that at most only one call of the allocator is needed.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-05 10:04:46 +08:00
XuanYang-cn a3aff37f73
fix: Correct flush buffer size metrics (#29571) 
See also: #29204

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-01-04 17:22:46 +08:00
congqixia 79c06c5e73
fix: serializer shall bypass L0 segment merge stats step (#29636) 
See also #27675
Fix logic problem introduced by #29413, which is serializer tries to
merge statslog list while level segments do not have statslog. This
shall result returning error. `writeBufferBase` ignores this error but
it shall only ignore `ErrSegmentNotFound`.

This PR add logic checking segment level before execution of merging
statslog list. And add error type check for getSyncTask failure.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-04 16:52:45 +08:00
congqixia 55af8f611f
fix: always sync level zero segments as flushed (#29569) 
See also #27675

For now, Level zero segments shall always be synced as `Flushed` ones.
This PR fixes when level zero segments selected by policies other than
flush ts policy will be synced as growing state.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-29 10:34:47 +08:00
MrPresent-Han ed644983e2
enhance: add param for bloomfilter(#29388) (#29490) 
related: #29388

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-12-28 18:10:46 +08:00
yah01 a8a0aa9357
fix: missing to support compact for Array type (#29505) 
the array type can't be compacted, the system could continue with the
inserted segments, but these segments can be never compacted

fix #29503

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-28 15:42:48 +08:00
XuanYang-cn 632d8b3743
enhance: Change DN channelmanger into interface (#29307) 
See also: #28854

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-12-27 16:00:48 +08:00
XuanYang-cn fe04598900
enhance: Add compaction type label to metrics (#29485) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-12-27 15:56:48 +08:00
congqixia f6cff25712
enhance: fix serialization record span & flushed buffer size metrics (#29482) 
See also #27675 #29413

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-27 10:20:48 +08:00
congqixia 277849a915
enhance: separate serializer logic from sync task (#29413) 
See also #27675

Since serialization segment buffer does not related to sync manager can
shall be done before submit into sync manager. So that the pk statistic
file could be more accurate and reduce complex logic inside sync
manager.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-26 10:40:47 +08:00
congqixia a937e4c232
fix: segment may never get flushed if sealed before watch (#29436) 
See also #29092

`FlushSegments` transfer only `Growing` segment to flushing, if the
segment is in `Sealed` state before Datanode watch channel, the state
will never got satisfied for a segment be selected to be flushed.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-23 21:32:43 +08:00
SimFG dd9c61831d
enhance: Support to get the param value in the runtime (#29297) 
/kind improvement
issue: #29299

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-12-22 18:36:44 +08:00
XuanYang-cn 7a6aa8552a
fix: add back existing datanode metrics (#29360) 
See also: #29204

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-12-22 14:20:43 +08:00
Xiaofan 77b291c5dc
fix: Add jitter in GetSyncStaleBufferPolicy (#28626) 
related to #28427
Add a jitter in syncStatleBuffer policy so all segments won't flush at
the same time

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2023-12-20 15:00:52 +08:00
congqixia 1ee016709d
fix: Unstable `TestDataSyncService/TestStartStop` unit test (#29291) 
fix #29290

Change EXPECT `NotifyCheckpointUpdated` call to `Maybe` expectation

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-19 23:16:42 +08:00
XuanYang-cn 5164377e68
fix: Skip updating checkpoint after dropcollection (#29220) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-12-15 16:04:45 +08:00
congqixia 4731c1b0d5
enhance: make SyncManager pool size refreshable (#29224) 
See also #29223

This PR make `conc.Pool` resizable by adding `Resize` method for it. 
Also make newly added datanode `MaxParallelSyncMgrTasks` config
refreshable

---------

Signed-off-by: Congqi.Xia <congqi.xia@zilliz.com>
 2023-12-15 09:58:43 +08:00
congqixia 25a4525297
enhance: Change sync manager parallel config item (#29216) 
Since the sync manager is global in datanode now, the old
`maxParallelSyncTaskNum` does not fit into current implementation
anymore.

This PR add a new param item for sync mgr parallel control and enlarge
default value

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-14 20:46:41 +08:00
Bingyi Sun ad866d2889
feat: integrate storagev2 into index build process (#28995) 
issue: https://github.com/milvus-io/milvus/issues/28994

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-12-13 17:24:38 +08:00
wayblink 51f870da7e
feat: Introduce channelCheckpointUpdater to reduce goroutine use in ttNode (#28570) 
/kind enhancement

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-12-12 13:48:42 +08:00
congqixia cb43647b9e
enhance: Log channel checkpoint source info in writebuffer (#28993) 
See also #27675
Print channel checkpoint source with rated log will help debugging
system behavior

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-07 11:50:36 +08:00
congqixia cb31016640
enhance: Write buffer time range when syncing logs (#28970) 
Related to #27675
The timestamp from, to field is not field for new implementation of
writebuffer & sync manager
This pr fills these field for better log information

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-05 17:36:36 +08:00
Bingyi Sun 36f69ea031
feat: integrate storagev2 in building index of segcore (#28768) 
issue: https://github.com/milvus-io/milvus/issues/28655

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-12-05 16:48:54 +08:00
yihao.dai d26b563a8b
feat: Define import API and metadata (#28731) 
Define the new rpc and metadata for ImportV2.

see also: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-12-04 19:56:35 +08:00
congqixia 393b1f943c
fix: Reject compaction task with growing segments (#28925) 
See also #28924
The compaction task generated before datanode finish `SaveBinlogPath`
grpc call contains segments which are still in Growing state DataNode
shall verify each non-levelzero segments before submit compaction task
to executor

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-04 19:06:40 +08:00
XuanYang-cn e62edb991a
enhance: Add FlowgraphManager interface (#28852) 
- Change flowgraphManager to fgManagerImpl
- Change close to stop
- change execute to controlMemWaterLevel
- Change method name of fgManager for readability
- Add mockery for fgmanager

Issue: #28853

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-30 18:42:32 +08:00
XuanYang-cn 5d0a9f9344
fix: Forget to set EntriesNum for deltalogs (#28858) 
See also: #28520

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-30 16:52:31 +08:00
XuanYang-cn aae7e62729
feat: Add levelzero compaction in DN (#28470) 
See also: #27606

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-30 14:30:28 +08:00
congqixia 2cd8daaf0b
fix: compacted segment still buffers delta data (#28816) 
Related to #28628
Compacted segment syncing counter is not set correctly in sync task and
the bf write buffer shall not use compacted segment as candidate when
buffering delta data

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-30 10:20:28 +08:00
XuanYang-cn 321c5c32e3
fix: Separate schedule and check results loop (#28692) 
This PR:

- Separates compaction scheduler and check results loop So that slow in
check-loop doesn't influence execution.

- Cleans compaction tasks when drop a vchannel so dropped-channel's
compaction tasks won't be checked over and over again.

  - Skips meta change when meta's already changed, avoid panic
  - Remove not inuse injectDone(bool) parameter

See also: #28628, #28209

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-29 10:50:29 +08:00
XuanYang-cn 606ec77b66
enhance: Unify levelzero segment config in DN (#28720) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-28 18:04:26 +08:00
congqixia eaabe0293b
fix: Update segment compactTo when compactTo segment is compacted (#28755) 
Related to #28736 #28748
See also #27675
Previous PR: #28646

This PR fixes `SegmentNotFound` issue when compaction happens multiple
times and the buffer of first generation segment is sync due to stale
policy

Now the `CompactSegments` API of metacache shall update the compactTo
field of segmentInfo if the compactTo segment is also compacted to keep
the bloodline clean

Also, add the `CompactedSegment` SyncPolicy to sync the compacted
segment asap to keep metacache clean

Now the `SyncPolicy` is an interface instead of a function type so that
when it selects some segments to sync, we colud log the reason and
target segment

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-27 19:48:26 +08:00
jaime b1e0a27f31
enhance: Add logs for each step during service initialization (#28624) 
/kind improvement

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-11-27 16:30:26 +08:00
congqixia 8a9ab69369
fix: Skip statslog generation flushing empty L0 segment (#28733) 
See also #27675

When L0 segment contains only delta data, merged statslog shall be
skiped when performing sync task

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-25 15:10:25 +08:00
congqixia 39be35804c
enhance: Add back clean compacted segment info logic (#28646) 
See also #27675
Compacted segment info shall be removed after all buffer belongs to it
is sync-ed.
This PR add the cleanup function after triggerSyncTask logic:
- The buffer is stable and protected by mutex
- Cleanup fetches compacted & non-sync segment
- Remove segment info only there is no buffered maintained in manager

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-24 15:38:25 +08:00
smellthemoon 79c0edb1d8
enhance:Remove msgbase unnecessary assignments (#28511) 
remove some unnecessary assignments, for the reason that
commonpbutil.NewMsgBase has default value.

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-11-24 15:02:39 +08:00
congqixia a2fe9dad49
enhance: Make etcd kv request timeout configurable (#28661) 
See also #28660
This pr add request timeout config item for etcd kv request timeout
 Sync the default timeout value to same value for etcdKV & tikv config

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-23 19:34:23 +08:00
XuanYang-cn b1f15fa0e8
fix: Ease the log level when sync task done (#28678) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-23 19:18:28 +08:00
Bingyi Sun 4fedff6d47
feat: integrate storage v2 into the write path (#28440) 
#28378

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-11-23 17:26:24 +08:00
XuanYang-cn 55800ade84
fix: Remove logging extra segmentIDs (#28662) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-23 14:40:23 +08:00
congqixia af18aa709b
fix: Remove not needed `BlockAll` call in `SyncSegments` (#28632) 
See also #28628
Previous compaction task blocked the segment sync task and may block the
flowgraph when sync task is generated by auto sync policy This
`BlockAll` call will block forever and cause whole fg stuck

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-22 14:52:23 +08:00
smellthemoon 29249c4bd3
enhance: create goroutine only once (#28594) 
create goroutine only once when getOrCreateMergedTimeTickerSender

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-11-22 10:26:28 +08:00
smellthemoon 73f2bab454
enhance:add some log when create client and get component states (#28160) 
/kind improvement

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-11-22 09:12:22 +08:00
congqixia 2fc743992a
fix: syncmgr unstable TestCompact unittest logic (#28630) 
fix #28629
orignal unit test close channel before setting the segment id, so there
is a chance that test read segment id before setting it change unit test
behavior to wait future return now

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-22 00:52:23 +08:00
congqixia 6521b519d7
enhance: Use bufSize instead of row number in sync policy (#28498) 
See also #27675, comment from #27874

This PR changes the `IsFull` logic of insert buffer
Changing the limit from rowNum to buffer size,
this shall help form better binlog file when schema has variable-length
field

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-21 15:02:25 +08:00
congqixia 2b3fa8f67b
fix: Add length check for `storage.NewPrimaryKeyStats` (#28576) 
See also #28575
Add zero-length check for `storage.NewPrimaryKeyStats`. This function
shall return error when non-positive rowNum passed.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-21 10:28:21 +08:00
congqixia 18dc6b61ce
enhance: fix LevelZero segment sync logic (#28482) 
See also #27675
- Fix LevelZero segment cannot be flushed
- Add level option for syncTask
- Invoke `AddSegment` when new LevelZero segment is allocated

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-17 21:46:20 +08:00
congqixia a3cd0bc9c3
fix: Refine sync task field binlog compose logic (#28494) 
See also #27675
Since `MetaWriter` need `*datapb.FieldBinlog` struct, sync task now
generate FieldBinlog directly

Also fix merged statslog not generated if last task has no insert

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-17 14:40:26 +08:00
congqixia bed7467f20
enhance: Remove commented code and fix naming issue (#28450) 
This PR removes all the commented code and files from PR #28320

For naming issue:
- Renaming `MinCheckpoint` to `EarliestPosition`, see #28320 comment
- Renaming `writebuffer.Mananger` to `BufferMananger`, see #27874
comment

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-16 00:22:20 +08:00
congqixia 0b905078e7
Use writebuffer, sync manager refactory in datanode (#28320) 
See also #27675
This PR make previously merged refactory of datanode go online
- Use write node to replace insert/delete node
- Use write buffer manager to control all buffers
- Use sync manager to control sync tasks instead of flush manager

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-15 15:24:18 +08:00
XuanYang-cn 40d5c902b6
Enable getting multiple segments in plan result (#28350) 
Compaction plan result contained one segment for one plan. For l0
compaction would write to multiple segments, this PR expand the segments
number in plan results and refactor some names for readibility.

- Name refactory: - CompactionStateResult -> CompactionPlanResult -
CompactionResult -> CompactionSegment

See also: #27606

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-14 15:56:19 +08:00
XuanYang-cn 8037f35de7
Change MetaCache interface (#28388) 
See also: #28320, #27675

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-14 15:08:19 +08:00
congqixia b1eb1ea506
Refine datanode Timetick Sender (#28393) 
- Use explicit lifetime control methods: `Start` and `Stop`
- Allow control retry option
- Make sure tt sender worker exit after `Stop` return

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-13 21:46:20 +08:00
smellthemoon 0aa90de141
Reduce the goroutine in flowgraph to 2 (#28233) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-11-13 10:50:17 +08:00
smellthemoon 5365748338
Use single instance for mergedTimeTickerSender (#27730) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-11-13 10:18:17 +08:00
yah01 f4341f254d
Not convert legacy error code to new merr (#28232) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-08 14:42:20 +08:00
SimFG e3b7fdac61
Delay the cancellation of ctx when stopping the node (#28247) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-11-08 03:20:17 +08:00
congqixia af1c2044b9
Fix atomic.Int64 not found in go 1.18 (#28216) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-07 15:16:21 +08:00
groot 3f6b203018
Fix bulkinsert bug that segments are compacted after import (#28192) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-11-07 15:14:26 +08:00
congqixia c41df18b6d
Add compaction meta in `SyncSegmentsRequest` (#28199) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-07 10:06:17 +08:00
yah01 90e2c63d9e
Fix getting incorrect CPU num (#28146) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-11-06 06:02:16 +08:00
congqixia bf2f62c1e7
Add `WriteBuffer` to provide abstraction for delta policy (#27874) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-04 12:10:17 +08:00
yihao.dai 8d080383b7
Use merr to prevent datanode panic (#28121) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-11-04 00:58:21 +08:00
Jiquan Long a21042dde7
Fix flushManager.isFull is too slow (#28141) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-11-03 14:42:17 +08:00
congqixia 1e51255c15
Implement `Injection` for SyncManager with block and meta transition (#28093) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-03 04:48:15 +08:00
yah01 9658367a3c
Refine chunk manager errors (#27590) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-31 12:18:15 +08:00
congqixia 233bf90c55
Add SyncManager to replace flush manager (#27873) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-31 02:30:16 +08:00
congqixia 98e2aad752
Refine datanode metacache and implement CoW (#27985) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-28 11:12:11 +08:00
smellthemoon 403c6680cc
Add error description (#27959) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-10-27 10:10:13 +08:00
Filip Haltmayer 6b1a106a31
Moving etcd client into session (#27069) 
Signed-off-by: Filip Haltmayer <filip.haltmayer@zilliz.com>
 2023-10-27 07:36:12 +08:00
yah01 f79c7370f4
Fix panic while flushing dropped/compacted segment (#27927) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-25 22:10:13 +08:00
yihao.dai b9d5ef3599
Fix datanode ttNode goroutine leak (#27878) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-10-24 18:22:10 +08:00
aoiasd 9091a27832
Add meta cache to datanode for L0 Delta (#27768) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-10-23 19:42:10 +08:00
SimFG 9b0ecbdca7
Support to replicate the mq message (#27240) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-10-20 14:26:09 +08:00
smellthemoon 4b0ec156b3
Set channel work pool size in datanode (#27728) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-10-19 08:28:08 +08:00
XuanYang-cn 7358c3527b
Add iterators (#27643) 
See also: #27606

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-10-18 19:34:08 +08:00
jaime e386a62fae
Remove recollect segment stats during starting datacoord (#27410) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-10-16 10:26:09 +08:00
jaime ec1fe3549e
Add a stop hook to clean session (#27564) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-10-16 10:24:10 +08:00
congqixia 82b2edc4bd
Replace manual composed grpc call with Broker methods (#27676) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-13 09:55:34 +08:00
yah01 be980fbc38
Refine state check (#27541) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-11 21:01:35 +08:00
aoiasd ec830e1240
fix use same logidx in different field (#27492) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-10-11 18:49:33 +08:00
congqixia cbb350c552
Add broker for datanode grpc operations (#27631) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-11 17:03:34 +08:00
yah01 2df9908c6a
Map old error code to new error (#27616) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-11 12:43:33 +08:00
jaime 86ed2fd51f
[skip e2e] Fix unstable ut (#27618) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-10-11 07:21:32 +08:00
XuanYang-cn 294ff74ca5
Add new compaction types (#27608) 
See also: #27606

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-10-11 06:49:32 +08:00
XuanYang-cn 56c94cdfa7
Add channel manager in DataNode (#27308) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-10-08 21:37:33 +08:00
congqixia 5d558623fe
Add revive sub-lints and fix existing problems (#27495) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-07 20:53:38 +08:00
congqixia 80eb5434d3
Fix var-naming caused by old PRs (#27501) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-07 14:43:32 +08:00
yah01 8394b3a1ec
Block creating new error from status reason (#27426) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-07 11:29:32 +08:00
Jiquan Long 0f14d18201
Optimize the codec code of session (#27360) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-10-01 10:33:30 +08:00
XuanYang-cn 5c5f9aa05e
Enhance newDataSyncService (#27277) 
- Add flowgraph.Assemble assembles nodes in flowgraph.go
- remove fgCtx in newDataSyncService
- Add newServiceWithEtcdTickler func, reduce param numbers to 3
- Remove unnecessary params
  - config.maxQueueLength, config.maxParallelish

See also: #27207

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-27 11:07:25 +08:00
yah01 6539a5ae2c
Refine DataCoord status (#27262) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-26 17:15:27 +08:00
wayblink 7dd0be1b2c
Enable bulkinsert binlog data with partitionkey (#27241) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-09-26 10:25:25 +08:00
jaime 7f7c71ea7d
Decoupling client and server API in types interface (#27186) 
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-09-26 09:57:25 +08:00
XuanYang-cn 676024ff38
Add rpc to notify channel operation (#27172) 
- Add NotifyChannelOperation
- Add CheckChannelOperationProgress

See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-25 14:05:27 +08:00
congqixia 1d76565894
Add metrics for garbage collection (#27303) 
Also fix second metrics usage in compaction

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-22 18:47:25 +08:00
SimFG 26f06dd732
Format the code (#27275) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-21 09:45:27 +08:00
XuanYang-cn 916528f51a
Add initCtx in getDataSyncService (#27199) 
Passing initCtx to all IO funcs in newDataSyncService,
so when ctx.Canceled, newDataSyncService would return.

See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-20 16:03:23 +08:00
yah01 b4f86ea55e
Construct all success status with merr (#27226) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-20 10:57:23 +08:00
yah01 338848fbc3
Remove unused code and dup imports (#27228) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-20 10:55:23 +08:00
Xiaofan 6635398a6d
Fix Bin log concurrency by adding a pool (#27189) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-09-19 18:05:22 +08:00
XuanYang-cn 09505ea78e
Move etcd watch related code into eventmanager (#27192) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-19 15:27:38 +08:00
XuanYang-cn 10116b85ac
Release before remove when releasing flowgraphs (#27191) 
GetAndRemove removes the fg from manager immediately,
while the flowgraph is still releasing. This PR will remove
the fg from flowgraphManager AFTER flowgraphs released.

- Add Remove for ConcurrentMap
- Move collections() into flowgraph manager

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-19 11:53:22 +08:00
yihao.dai 4b2802033d
Fix datanode panic due to concurrent compaction and delete processing (#27167) 
Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-09-18 19:35:29 +08:00
yah01 a6b98740b7
Replace all status with only error string (#27125) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-18 14:05:28 +08:00
yihao.dai c162c6a4c8
Increase FlushChannel retry times (#27140) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-16 00:41:20 +08:00
yah01 168e82ee10
Fix panic while handling with the nil status (#27040) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-15 10:09:21 +08:00
yihao.dai dd2cb1d44a
Flush by flush channels (#26859) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-12 21:07:19 +08:00
yah01 00c65fa0d7
Refine QueryNode errors (#27013) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-12 16:07:18 +08:00
Xu Tong 9166011c4a
Add float16 vector (#25852) 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-09-08 10:03:16 +08:00
XuanYang-cn 7f1ae35e72
Add timeout in dispatcher, AsConsumer and Seek (#26686) 
See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-08 09:51:17 +08:00
yiwangdr 337edc321b
tikv integration (#26246) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-09-07 07:25:14 +08:00
bjzhjing 548c82eca5
Refactor storage.MergeInsertData() to optimize the merging process (#26839) 
Benchmark Milvus with https://github.com/qdrant/vector-db-benchmark and
specify the datasets as 'deep-image-96-angular'. Meanwhile, do perf
profiling during 'upload + index' stage of vector-db-benchmark and see
the following hot spots.

39.59%--github.com/milvus-io/milvus/internal/storage.MergeInsertData
        |
        |--21.43%--github.com/milvus-io/milvus/internal/storage.MergeFieldData
        |          |
        |          |--17.22%--runtime.memmove
        |                     |
        |                     |--1.53%--asm_exc_page_fault
        |                     ......
        |
        |--18.16%--runtime.memmove
                   |
                   |--1.66%--asm_exc_page_fault
                   ......

The hot code path is in storage.MergeInsertData() which updates
buffer.buffer by creating a new 'InsertData' instance and merging both
the old buffer.buffer and addedBuffer into it. When it calls golang
runtime.memmove to move buffer.buffer which is with big size (>1M), the
hot spots appear.

To avoid the above overhead, update storage.MergeInsertData() by
appending addedBuffer to buffer.buffer, instead of moving buffer.buffer
and addedBuffer to a new 'InsertData'. This change removes the hot spots
'runtime.memmove' from perf profiling output. Additionally, the 'upload
+ index' time, which is one performance metric of vector-db-benchmark,
is reduced around 60% with this change.

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
 2023-09-05 21:41:48 +08:00
congqixia fe7f7ea237
Add cpp-build and setenv for querynode mockery regen command (#26843) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-05 14:13:49 +08:00
Enwei Jiao fb0705df1b
Decouple basetable and componentparam (#26725) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-05 10:31:48 +08:00
XuanYang-cn b2e7cbdf4b
Remove TimeTravel in compactor (#26785) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-04 17:41:48 +08:00
yihao.dai 4340cbfba2
Merge syncCPLagTooBehind policy into syncPeriodically policy (#26713) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-04 14:37:47 +08:00
yihao.dai 7624c2b949
Improve rated log in insertBufferNode (#26788) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-04 09:59:10 +08:00
yah01 3349db4aa7
Refine errors to remove changes breaking design (#26521) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-04 09:57:09 +08:00
congqixia e8f1b1736e
Remove log.Error(err.error())-style log (#26783) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-01 13:09:01 +08:00
XuanYang-cn 8d54509e54
Fix CompactionLatency metrics (#26747) 
- Refine compactor logs

See also: #26743

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-31 17:35:03 +08:00
smellthemoon 6069a7d42b
Log out memory in MB (#26262) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-08-30 17:13:00 +08:00
congqixia cc584551fb
Remove unused hash value setting in DDNode (#26575) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-28 11:30:26 +08:00
SimFG b9bc6681ae
Use atomic.bool for the isWatchFailed param in the tickler (#26558) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-08-23 13:22:27 +08:00
XuanYang-cn 27691e843c
Fix channel checkpoint stuck (#26534) 
See also: #23621

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-22 18:10:22 +08:00
XuanYang-cn 08fd28b30b
Only do gracefully stop when DN Stop (#26399) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-21 11:16:20 +08:00
SimFG 3be4ac4022
Fix datanode/datacoord continuous restart (#26470) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-08-20 21:20:24 +08:00
Enwei Jiao 533f0ddf6d
Add amazonlinux image, prepare to remove Centos7 image, remove openblas image (#26368) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-18 18:32:19 +08:00
congqixia fbb5d32cb6
Make write binlog in parallel (#26325) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-17 15:44:18 +08:00
congqixia f371ec0316
Add data sync service close log for watch stuck debugging (#26332) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-16 10:57:32 +08:00
Enwei Jiao 78bc688d16
Remove QueryMsgStream in MqFactory interface (#26374) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-16 10:29:33 +08:00
congqixia f8bcf60e55
Adjust some confusing Warning log to INFO (#26356) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-15 17:49:33 +08:00
wayblink 21eeb37ce9
Save binlog timestampFrom, timestampTo meta when compact (#26203) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-08-08 21:17:21 +08:00
congqixia b9850ce5c0
Fix copylocks linter errors (#26217) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-08 20:41:15 +08:00
MrPresent-Han b84f5c560a
refine log out for datanode(#25763) (#26195) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-08 16:49:08 +08:00
MrPresent-Han fb933fe64d
fix sync memory policy lose effect when inserting too many partitions(#25763) (#26154) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-07 09:59:07 +08:00
XuanYang-cn 830f06783a
Reduce log volumes of DC&DN (#26060) 
See also: #26057

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-02 17:13:11 +08:00
XuanYang-cn 84253f255e
Fix datanode graceful stop panic (#25932) 
See also: #25925

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-07-28 10:11:08 +08:00
groot a6808e6484
Fix bulkinsert row count error (#25869) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-07-25 11:29:00 +08:00
cai.zhang 779d677eb2
Remove lock for compaction sync segments (#25711) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-07-24 17:25:00 +08:00
jaime 8e5d6193f9
Add a timeout config for bulkinsert request (#25789) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-07-24 14:23:00 +08:00
congqixia 3c503afe7c
Use typeutil.ConcurrentMap instead of sync.Map (#25846) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-24 10:23:01 +08:00
congqixia 31b40b3ccc
Use typeutil.ConcurrentMap instead of sync.Map in fg manager (#25835) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-24 10:21:07 +08:00
Enwei Jiao 66fdc71479
Refactor logs in DataCoord & DataNode (#25574) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-07-14 15:56:31 +08:00
yiwangdr b9189b9f41
Organize mocks from types.go (#25466) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-07-14 10:12:31 +08:00
wayblink 53b4b70316
Reduce log frequency in timetick_sender (#25532) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-07-13 14:38:29 +08:00
wei liu 012fd1152a
fix datacoord consume datanode tt metrics (#25524) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-13 14:26:29 +08:00
congqixia dbfade77c5
Retry on SyncSegment failure (#25540) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-13 14:12:29 +08:00
wayblink ed3e4b0bc5
Fix unstable ut in timetick_sender_test (#25539) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-07-13 10:04:29 +08:00
MrPresent-Han ea3817fbbc
refinement towards sync_cp_lag_too_behind_policy to avoid submit syncTasks too frequently(#25441) (#25442) (#25500) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-07-12 20:22:29 +08:00
SimFG f9e2d00f91
Prevent `exclusive consumer` exception in pulsar (#25376) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-07-12 17:26:30 +08:00
wayblink fc12d3997c
Rename newTimeTickManager to newTimeTickSender (#25415) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-07-12 14:56:28 +08:00
groot 96c987ed62
Bulkinsert supports partition keys (#25284) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-07-11 15:18:28 +08:00
wei liu 71d99d4ec2
fix unexpected metrics value cause by wrong time unit (#25240) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-06 08:48:27 +08:00
Xiaofan 63b6a4a639
use single instance ppol (#25159) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-28 14:54:45 +08:00
xige-16 33c2012675
Add more metrics (#25081) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-26 17:52:44 +08:00
yiwangdr c7b851f870
add interface for non-watch metakv (#25092) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-06-26 09:20:44 +08:00
congqixia 736916222b
Stop retry lazy load stats after segment released (#25054) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-25 19:20:43 +08:00
jaime 18df2ba6fd
[Cherry-Pick] Support Database (#24769) 
Support Database(#23742)
Fix db nonexists error for FlushAll (#24222)
Fix check collection limits fails (#24235)
backward compatibility with empty DB name (#24317)
Fix GetFlushAllState with DB (#24347)
Remove db from global meta cache after drop database (#24474)
Fix db name is empty for describe collection response (#24603)
Add RBAC for Database API (#24653)
Fix miss load the same name collection during recover stage (#24941)

RBAC supports Database validation (#23609)
Fix to list grant with db return empty (#23922)
Optimize PrivilegeAll permission check (#23972)
Add the default db value for the rbac request (#24307)

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-06-25 17:20:43 +08:00
Xiaofan e8911ebda7
Add retry time when lazy load BF (#25096) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-25 11:32:43 +08:00
Xiaofan 72c5e2a41a
Fix channel reassigned to other datanodes (#25015) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-21 21:26:42 +08:00
aoiasd 7f5de9aab3
add read lock for getSegment (#25036) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-06-21 14:06:42 +08:00
PowderLi 3f4356df10
fix the spelling of `field` (#25008) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-06-21 14:00:42 +08:00
yihao.dai 6168a55c04
Lower the log frequency when sync tasks are skipped (#25006) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-06-20 17:56:41 +08:00
jaime b75dcf90c0
Fix Flush hang after SyncSegments timeout (#24953) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-06-20 10:20:41 +08:00
Xiaofan af1d84e5e1
Support sync when cp is lagged (#24989) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-20 10:00:41 +08:00
Xiaofan f57fe6d70b
Add compaction log (#24976) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-19 14:18:41 +08:00
yihao.dai c73219a54d
Limit the number of concurrent sync tasks and allow only one sync task for the same segment (#24881) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-06-16 14:14:39 +08:00
congqixia d186f35895
Fix TestChannelMetaMockSkipBFLoad/transient_error not stable (#24928) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-16 09:50:44 +08:00
wayblink bfae6b49af
Remove datanode timetick mq, use rpc to report instead (#23156) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-06-14 14:16:38 +08:00
groot 5c3b744b0c
Fix potential crash bug of bulkinsert (#24763) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-06-13 10:22:38 +08:00
congqixia 41af0a98fa
Use go-api/v2 for milvus-proto (#24770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
yah01 ebd0279d3f
Check error by Error() and NoError() for better report message (#24736) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-08 15:36:36 +08:00
congqixia 31880ab427
Use conc.PoolOption instead of ants.Option (#24585) 
- Add conc.PoolOption to setup conc.Pool
- Change panic default behavior
- Make future has error when job panicks

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-01 15:52:31 +08:00
wayblink 5958b42941
Add retry on getChannelLatestMsgID (#24518) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-05-30 20:03:28 +08:00
aoiasd c84bdcea49
merge stats log when segment flushing or compacting (#23570) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-05-29 10:21:28 +08:00
Xiaofan 49f75e5e11
Make flow graph retry longer (#24355) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-05-25 09:53:30 +08:00
congqixia 03297a7ef7
Fix event manager not closed after datanode stop (#24206) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-18 11:43:23 +08:00
yah01 99cc24b283
Fix compaction doesn't support JSON data (#24151) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-05-17 19:09:22 +08:00
congqixia 73a181d226
Fix get vector it timeout and improve some string const usage (#24141) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-16 17:41:22 +08:00
congqixia 084424d636
Make data node start only once (#24031) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-11 13:51:20 +08:00
congqixia 345855c984
Make datanode load statslog lazy if SkipBFStatsLog is true (#23877) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-06 15:12:39 +08:00
MrPresent-Han b517bc9e6a
refine balance mechanism including:(#23454) (#23763) (#23791) 
1. balance granuity to replica to avoid influence unrelated replicas
2. avoid balance back and forth

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2023-05-04 12:22:40 +08:00
congqixia 796f9355ee
Fix global rateCollector init multiple times (#23703) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-26 10:16:36 +08:00
wei liu 4fb8919a97
support collection level rate limit (#22767) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-04-25 15:54:35 +08:00
zhenshan.cao 03ec804e68
Change default shard number to 1 (#23586) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-04-21 07:08:32 +08:00
congqixia 4a6c2b78a7
Fix multiple transferNewSegment and removeSegment causes panic (#23564) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-20 13:48:31 +08:00
Enwei Jiao 967a97b9bd
Support json & array types (#23408) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
 2023-04-20 11:32:31 +08:00
congqixia 4fe363c4b2
Add KNN cgo pool (#23526) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-19 23:56:31 +08:00
congqixia 4508786819
Fix test_watch_channel unit test unstable (#23507) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-19 10:00:31 +08:00
wei liu cbfe7a45ef
fix pull target (#23491) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-04-18 18:30:32 +08:00
wayblink b23723535c
Add log in AddImportSegment.getChannelLatestMsgID (#23480) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-04-18 17:50:31 +08:00
zhenshan.cao 4a32b842e8
Improve the check logic of channel remove (#23473) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-04-18 02:58:30 +08:00
aoiasd 695cc769df
Fix watch event timeout when IO Pool busy (#23293) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-04-17 11:02:30 +08:00
aoiasd f603456b72
remove segment from datanode meta when segment dropped (#23398) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-04-17 10:58:30 +08:00
congqixia ba84f52119
Fix watcher loop quit and channel shouldDrop logic (#23402) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-14 09:54:28 +08:00
cai.zhang 43a9e175a3
Exit component process when session key is deleted (#21658) (#22164) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-04-12 20:12:28 +08:00
XuanYang-cn b90fa5f459
Refine codes of datanode buffer (#23168) 
This PR refines deltabuffer of datanode:
- Add last sync time for compacted segment, see also: #23210
- Ensure all deltabuffermanager handles all delete related operations
- Change usedMemory to atomic.Int64
- Remove allocator in delete buffer

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-04-10 18:42:30 +08:00
jaime c9d0c157ec
Move some modules from internal to public package (#22572) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-04-06 19:14:32 +08:00
aoiasd 1a485044a9
fix NumConsumers metric not work (#23167) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-04-05 08:18:28 +08:00
smellthemoon ce91f61e5a
Pre-create topics (#21907) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-04-03 16:44:23 +08:00
XuanYang-cn be0827417a
Fix buffer calculate memorysize negative (#23152) 
Along with the following small patches:
- And fix logging dup segmentIDs for delete buffer status
- Add String method for PriorityQueue to debug

See also: #23105

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-03-31 18:30:23 +08:00
Jiquan Long 0f2c3ac832
Optimize log (#23121) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-03-30 10:44:22 +08:00
Enwei Jiao 50dab2d394
Add tracing for retrieve (#23033) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-03-29 18:10:02 +08:00
XuanYang-cn 9691315c21
Fix flush deltabuf with segmentID=0 (#23059) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-03-28 19:14:01 +08:00
aoiasd 5d172d0f4f
Refine errors of datanode (#22852) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-03-28 19:04:00 +08:00
zhenshan.cao e3c3c949c4
Fix datarace of flow_graph_manager_test.go in package datanode (#23043) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-03-28 10:42:00 +08:00
yihao.dai b21f03682a
Fix flowgraph manager test (#23009) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-03-27 18:00:05 +08:00
congqixia 4008ee6d60
Remove datanode ddNode forward delta logic (#23005) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-27 14:18:00 +08:00
yah01 081572d31c
Refactor QueryNode (#21625) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-03-27 00:42:00 +08:00
Xiaofan 5a914aedf3
Make Datacoord TT to be configurable (#22956) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-03-24 22:24:04 +08:00
XuanYang-cn da58ba6a49
Fix endless appending sids (#22977) 
See also: #22976

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-03-24 18:00:03 +08:00
congqixia 8f847884c2
Fix misc copylocks and remove exclude rule (#22962) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-24 11:17:58 +08:00
XuanYang-cn 93bc805933
Enhance ID allocator in DataNode (#22905) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-03-23 19:43:57 +08:00
yihao.dai f2ffb5db26
Use buffer size at memory sync policy (#22825) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-03-21 21:37:56 +08:00
SimFG 4a90490a67
Fix the `segment not found` error (#22772) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-03-17 17:27:56 +08:00
smellthemoon 8799b06dbc
Fix the data is disappeared when upsert (#22762) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-03-14 19:11:54 +08:00
congqixia 732986aa04
Remove fmt.Print from internal package (#22722) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-14 17:36:05 +08:00
aoiasd 6959332b08
Fix unittest start channel event but not close (#22662) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-03-14 17:05:54 +08:00
wayblink aa6212d5b0
[Cherry-pick] Fix channel checkpoint issues and fix GC (#22559) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: Ten Thousand Leaves <69466447+soothing-rain@users.noreply.github.com>
 2023-03-09 14:13:52 +08:00
jaime d126f06946
Decouple mq module from internal proto definition (#22536) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-03-04 23:21:50 +08:00
yah01 60cd548bf5
Rename concurrency package to conc (#22453) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-02-28 14:19:47 +08:00
yah01 167581c08a
Refine Pool to improve the ease of use (#22412) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-02-27 18:07:48 +08:00
wayblink 416866e42f
Add memory usage too large sync policy (#22244) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-02-27 17:47:51 +08:00
yah01 b758c305a7
Make Future generic (#22411) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-02-27 12:07:47 +08:00
Xiaofan 3d6bded115
Fix Row Number Mismatch (#22307) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-02-27 10:41:46 +08:00
Enwei Jiao 697dedac7e
Use cockroachdb/errors to replace other error pkg (#22390) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-02-26 11:31:49 +08:00
zhenshan.cao e768437681
Correct usage of Timer and Ticker (#22228) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-02-23 18:59:45 +08:00
aoiasd 27b456dd46
Fix data race of data node test (#22221) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-02-22 10:28:26 +08:00
wei liu 0ce70b4f10
fix compact task data race (#22299) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-02-21 16:10:26 +08:00
Enwei Jiao 0a9a9058b9
Fix UT timeout (#22261) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-02-20 21:14:25 +08:00
aoiasd 9167773eff
Remove needless kv at dataSyncService (#22206) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-02-16 16:36:39 +08:00
congqixia 9346f1752a
Convert msg pchannel to vchannel before check IsCloseMsg (#22182) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-02-15 17:26:34 +08:00
aoiasd 148a024e05
Add tickle for datacoord watch event (#21193) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-02-15 16:20:34 +08:00
aoiasd 1894766235
Update datacoord compaction plan after datanode update plan to ensure consistency (#22143) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-02-15 16:00:33 +08:00
smellthemoon e20d79a8a1
Fix data race (#22171) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-02-15 15:40:33 +08:00
bigsheeper d2667064bb
Add msgDispatcher to support sharing msgs for different vChannel (#21917) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-02-13 16:38:33 +08:00
cai.zhang 511265c68c
Modify lastSyncTime in advance to prevent multiple flush binlogs (#22048) (#22088) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-02-12 18:56:33 +08:00
XuanYang-cn 2b982b5d8f
Fix DataNode ut never meet condition (#22093) 
See also: #22079

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-02-10 11:04:32 +08:00
yah01 9b491858a5
Fix datanode unit test not stable (#21911) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-02-01 18:19:50 +08:00
Xiaofan 949d5d078f
Fix memory calculation in dataCodec (#21800) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-01-28 11:09:52 +08:00
congqixia 5986106037
Make paramtable init only once (#21782) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-01-19 14:53:44 +08:00
Enwei Jiao 90d9e165d4
Fix some configs not shown (#21653) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-01-13 15:31:41 +08:00
wayblink 6a722396bd
Integration test framework (#21283) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-01-12 19:49:40 +08:00
Enwei Jiao fb42466c65
Use opentelemetry (#21509) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-01-12 16:09:39 +08:00
groot 9f8095996b
Report bulkinsert progress (#21612) (#21638) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2023-01-11 17:37:44 +08:00
groot df0ffd08ce
Update segment id for import task (#21621) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2023-01-11 11:55:39 +08:00
Xiaofan 4b4944ecee
Support sync all segments while close (#21421) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-01-06 14:49:36 +08:00
cai.zhang efefba4da4
Fix checkpoint will not update after delete and compact (#21495) (#21499) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-05 11:31:35 +08:00
yah01 7b39873ae0
limit the frequency of GetMetrics() log (#21514) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-01-04 17:39:35 +08:00
Jiquan Long ff2a68e65a
Fix collection not exist when tried to do recovery (#21471) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-01-04 16:37:35 +08:00
congqixia 3a58ac9fba
Change node stop to TearDownTest (#21422) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-28 20:11:30 +08:00
Xiaofan 3667bb09b8
Avoid Sync when the segment open (#21397) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-12-28 10:51:34 +08:00
congqixia 3c9840d6d3
Fix some datanode instance not stopped in unit tests (#21385) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-26 16:59:30 +08:00
Enwei Jiao 166e9f0da5
Refactor GrpcConfig (#21142) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-16 15:59:23 +08:00
bigsheeper f595500383
Move sync delete policy from deleteNode to insertBufferNode (#21152) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-12-13 16:15:21 +08:00
XuanYang-cn a7b24cbc53
Move APIs of types.DataNode to services.go (#21042) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-12-12 10:57:22 +08:00
Xiaofan e977e014a9
Fix flush didn't respect binaryvector and other schemas (#21120) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-12-12 10:33:26 +08:00
aoiasd de0ab9e2cf
Refactor showConfigurations to allow return global config rather than only return config of this component (#21063) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-12-09 14:31:21 +08:00
Enwei Jiao 89b810a4db
Refactor all params into ParamItem (#20987) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-07 18:01:19 +08:00
groot 33a102c6fc
Refine bulkinsert (#20986) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-12-07 14:53:18 +08:00
XuanYang-cn f844aa0cab
Remove not inuse stale segments in flushmsg (#20981) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-12-06 17:51:18 +08:00
jaime 039e9ce4bb
Refine msgstream interface (#20832) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-12-05 20:55:17 +08:00
congqixia 2791f5526c
Fix BufferData limit calculation for BinaryVector (#20963) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-05 11:15:17 +08:00
congqixia 4238b5dbab
Make datanode unittest use embed etcd server (#20948) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-02 16:05:16 +08:00
congqixia 40abb13413
Generate unique id with ts for sync task (#20928) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-01 20:35:18 +08:00
congqixia f745d7f489
Fix compaction target segment rowNum is always 0 (#20937) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-01 20:33:17 +08:00
zhenshan.cao 9724ae5e27
Refine log level (#20904) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-12-01 20:07:15 +08:00
Ten Thousand Leaves b238f4ee58
Remove bad & duplicate metric (#20905) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-12-01 16:39:16 +08:00
congqixia 82fe655032
Setup lastSyncTs when add segment to avoid sync next timetick (#20920) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-01 12:29:16 +08:00
Enwei Jiao 2ecdb4ba4a
Etcd config source support TLS (#20874) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-30 18:23:15 +08:00
cai.zhang 83244afcf3
Reduce some logs level which is frequent (#20894) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-11-30 14:07:15 +08:00
SimFG 44d45452fa
Make the sync segment request idempotent in the data node (#20707) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-11-24 10:21:13 +08:00
groot bf2107ecf5
Fix a regression of local storage (#20653) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-11-21 10:19:10 +08:00
groot c8022ea63c
Change import task timeout value (#20717) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-11-21 10:11:11 +08:00
jaime 0a5270e8c0
Fix alter collection hang (#20694) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-11-21 10:09:10 +08:00
Bingyi Sun 2390095232
Fix load uses compacted segments' binlogs (#20655) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-11-17 20:37:10 +08:00
Enwei Jiao c05b9ad539
Add event dispatcher for config (#20393) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-17 18:59:09 +08:00
Enwei Jiao 19524a5344
Fix nodeID mismatch at standalone mode (#20648) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-17 17:15:08 +08:00
XuanYang-cn bd20536c6a
Reduce unnecessary logs of get compaction results (#20642) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-11-17 10:09:12 +08:00
bigsheeper 2d307c1909
Set delBuf's start-end position of compacted segment (#20614) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-11-15 19:41:16 +08:00
MrPresent-Han c848896bc8
Fix panic error due to rollDeleteOp ahead by load-and-delete (#20563) 
issue:#20501
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2022-11-15 14:45:07 +08:00
bigsheeper db33ffa518
Pass endPosition with vchannel in datanode flowgraph (#20589) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-11-15 14:37:07 +08:00
smellthemoon a7ba416b5a
Remove invalid timestamp (#20557) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-11-14 16:21:07 +08:00
bigsheeper fc15789da9
Ensure compatibility of channel seek position and move syncPeriod to config (#20504) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-11-12 21:09:04 +08:00
bigsheeper cd19d99ad7
Add channel level checkpoint (#20350) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-11-10 22:13:04 +08:00
XuanYang-cn 6bff0b6688
Remove too frequent logs (#20454) 
Half of the log messages come from this log if DataNode in stale.

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-11-10 14:51:05 +08:00
Enwei Jiao 7e56e7a976
Fix datarace at Setlogger (#20376) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-08 14:33:03 +08:00
wayblink a7bed1c927
Pass backup flag through Import request options (#20334) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-11-08 11:33:03 +08:00
MrPresent-Han 21b54709a2
enable delbuf auto flush function to avoid OOM when del msg accumulating (#20213) 
issue: #19713

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
Co-authored-by: MrPresent-Han <zilliz@zillizdeMacBook-Pro-2.local>
 2022-11-07 18:49:02 +08:00
jaime 709a0a94e9
Add some msgstream metrics (#20296) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-11-07 10:15:02 +08:00
Enwei Jiao 956c5e1b9d
Make Params singleton (#20088) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-04 14:25:38 +08:00
congqixia f557af1143
Verify target id for FlushSegments request in datanode (#20308) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-11-03 22:47:35 +08:00
bigsheeper d528aa3d90
Print seek postion time and seek elapse (#20271) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-11-03 15:03:35 +08:00
MrPresent-Han e0baf1536e
[skip e2e] Improve a commment for initial pr (#20287) 
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2022-11-03 14:05:36 +08:00
Ten Thousand Leaves 5708352a80
Improve DataCoord and DataNode logs (#20265) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-11-03 12:15:35 +08:00
Xiaofan 33349677fc
Fix unsafe lock (#20219) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-11-01 19:37:35 +08:00
Xiaofan 99b958e360
Fix BF Concurrency issue (#20211) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-10-31 19:27:35 +08:00
Xiaofan 2bfecf5b4e
Refine bloomfilter and memory usage (#20168) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-10-31 17:41:34 +08:00
jaime 48f5c60070
Refine msgstream close (#20194) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-10-31 15:13:33 +08:00