milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
21,936 Commits
148 Branches
122 Tags
578 MiB
Commit Graph

1158 Commits (master)

Author SHA1 Message Date
cai.zhang 04c306e63f
fix: Fix clustering compaction task leak () 
issue:  

bug reason:
- The clustering compaction tasks on the datanode were never cleaned up.
- The clustering compaction task contains a mapping from clustering key
to buffer, this caused a large memory leak.

fix:
- clean the tasks on datanode by datacoord when clustering compaction
finished.
- reset the mapping that from clustering key to buffer on datanode when
clustering finished.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-10-17 20:43:30 +08:00
XuanYang-cn b172ea1093
fix: Remove enableLevelZeroSegment config () 
See also: 

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-17 11:59:24 +08:00
aoiasd 5ec4163d0f
feat: support bm25 logs mixcompaction () 
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-10-14 16:57:22 +08:00
Buqian Zheng 82c5cf2fa2
feat: add bulk insert support for Functions () 
issue: https://github.com/milvus-io/milvus/issues/35853 and
https://github.com/milvus-io/milvus/issues/35856

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-10-12 17:19:20 +08:00
CharlesFeng 7c8b71e26c
fix: BinlogDeserializeReader leak in mix_compactor.go () 
https://github.com/milvus-io/milvus/issues/36269

Signed-off-by: fengjun2016 <jornfeng@gmail.com>
 2024-10-11 15:41:20 +08:00
XuanYang-cn 290ceb4e84
enhance: Add more info in logs () 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-10 17:51:25 +08:00
yihao.dai 0fc2a4aa53
enhance: Optimize import scheduling and add time cost metric () 
1. Optimize import scheduling strategic:
a. Revise slot weights, calculating them based on the number of files
and segments for both import and pre-import tasks.
b. Ensure that the DN executes tasks in ascending order of task ID.
2. Add time cost metric and log.

issue: https://github.com/milvus-io/milvus/issues/36600,
https://github.com/milvus-io/milvus/issues/36518

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-09 14:41:20 +08:00
wayblink 00a5025949
enhance: support clustering compaction on null value () 
issue: 

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-09-30 14:33:17 +08:00
yihao.dai 80f25d497f
enhance: Add metrics to monitor import throughput and imported rows () 
issue: https://github.com/milvus-io/milvus/issues/36518

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-28 17:31:15 +08:00
cai.zhang 2adca8b754
fix: Fix data race for cluerting compaction () 
issue: 

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-28 17:19:21 +08:00
yihao.dai 9e8cafcbe2
enhance: Skip loading bf in datanode () 
Skip loading bf in datanode:
1. When watching vchannels, skip loading bloom filters for segments.
2. Bypass bloom filter checks for delete messages, directly writing to
L0 segments.
3. Remove flushed segments proactively after flush.

issue: https://github.com/milvus-io/milvus/issues/34585

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-26 10:11:15 +08:00
aoiasd 139787371e
feat: support embedding bm25 sparse vector and flush bm25 stats log () 
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-09-19 10:57:12 +08:00
congqixia 3352030a84
enhance: Graceful stop flowgraph manager when stopping datanode () 
Flowgraph manager is not stopped durong datanode stopping procedure
which may lead to unexpect flowgraph behavior during/after datanode stop
progress.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-14 15:55:08 +08:00
cai.zhang 8395c8a8db
enhance: Update stats task to optional () 
issue: 

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-12 20:37:08 +08:00
smellthemoon 3f75bf1f20
fix: clustering compact not support null () 


Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-09-11 14:49:06 +08:00
XuanYang-cn 2687747278
fix: Set an empty segment if compaction deleted all inserts () 
See also: 

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-09-09 14:23:05 +08:00
Chun Han e480b103bd
feat: supporing hybrid search group_by () 
related: 

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-09-08 17:09:04 +08:00
SimFG 5247631289
fix: fill the metric type field in the LoadMetaInfo object () 
- issue: 

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-09-05 20:50:23 -07:00
cai.zhang 90bdb171ab
fix: Fix data race for clustering compaction writer () 
issue: 

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-05 04:07:10 +08:00
CharlesFeng 6eb8b3f745
fix: err degenerated to a new variable () 
https://github.com/milvus-io/milvus/issues/35890

Signed-off-by: fengjun2016 <jornfeng@gmail.com>
 2024-09-04 14:57:04 +08:00
yihao.dai 6fd33285e1
fix: Fix compile error () 
/kind improvement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-02 14:50:35 +08:00
cai.zhang 2c9bb4dfa3
feat: Support stats task to sort segment by PK () 
issue:  

This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-02 14:19:03 +08:00
yihao.dai 1413ffe9b1
enhance: Rename preAllocatedSegments () 
Rename `preAllocatedSegments` to `preAllocatedSegmentIDs` to avoid
confusion.

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-01 17:09:01 +08:00
XuanYang-cn 323400c190
enhance: Enable to write multiple segments in mix compactor () 
Prevent segments to be written larger than maxSize * expansionRate

See also: 

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-08-30 11:29:01 +08:00
Zhen Ye 99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode () 
issue: 

- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service

---------

Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-29 10:03:08 +08:00
yihao.dai 9868fe4e6c
fix: Fix panic due to empty candidate import segments () 
issue: https://github.com/milvus-io/milvus/issues/35662

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-27 17:08:59 +08:00
congqixia 59387f0b3a
fix: Use correct release func when datanode unwatch a channel () 
See also 

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-23 14:09:03 +08:00
congqixia ab532ae199
enhance: Add back BF lazy load logic for datanode watch channel () 
Add back lazy loading statslog when watch dml channel on datanode.

Related to  

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-22 19:42:57 +08:00
Ted Xu 41646c8439
feat: integrate new deltalog format () 
See 

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-08-20 19:06:56 +08:00
smellthemoon 80a7c78f28
enhance: import supports null in parquet and json formats () 


---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-08-20 16:50:55 +08:00
XuanYang-cn 967f38672a
enhance: Add integration tests for l0 () 
See also: 

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-08-19 10:56:54 +08:00
cai.zhang 1bbf7a3c0e
enhance: Optimize the use of locks and avoid double flush clustering buffer writer () 
issue: 

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-08-16 02:24:58 +08:00
wei liu c0200eec39
enhance: limit getSegmentInfo batch size to avoid excced grpc message limit () 
issue: 

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-15 19:17:00 +08:00
CharlesFeng b4b1ed6910
fix: before exiting, make sure the goroutine has exited () 
https://github.com/milvus-io/milvus/issues/35409

Signed-off-by: fengjun2016 <jornfeng@gmail.com>
 2024-08-15 18:16:53 +08:00
cai.zhang 196b343a94
fix: Fix data race for clustering compaction () 
issue: 

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-08-13 17:10:20 +08:00
cai.zhang aaab827a16
fix: Fix the issue of missing stats log after clustering compaction () 
issue: 

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-08-08 14:24:17 +08:00
yihao.dai a4439cc911
enhance: Implement flusher in streamingNode () 
- Implement flusher to:
  - Manage the pipelines (creation, deletion, etc.)
  - Manage the segment write buffer
  - Manage sync operation (including receive flushMsg and execute flush)
- Add a new `GetChannelRecoveryInfo` RPC in DataCoord.
- Reorganize packages: `flushcommon` and `datanode`.

issue: https://github.com/milvus-io/milvus/issues/33285

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-02 18:30:23 +08:00
congqixia 095d77269b
enhance: Set WatchProgress for channel operation progress response () 
The watch progress is always zero in CheckChannelOperationProgress
response, which is meaningless and confusing. This PR set progress value
in rpc response to fix this problem.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-02 12:06:14 +08:00
wayblink 95462668ca
enhance: unify time in clustering compaction task to unix () 


Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-08-02 10:30:19 +08:00
zhenshan.cao aa247f192d
enhance: remove unused code for StorageV2 () 
issue: https://github.com/milvus-io/milvus/issues/34168

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-08-01 12:08:13 +08:00
cai.zhang 9412002d7d
fix: Fix data race for clustering buffer writer () 
issue: 

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-08-01 11:20:13 +08:00
congqixia de8a266d8a
enhance: Enable linux code checker () 
See also 

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-30 15:53:51 +08:00
wei liu c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 () 
issue: 

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-29 11:31:51 +08:00
cai.zhang 2372452fac
enhance: Optimized the GC logic to ensure that memory is released in time () 
issue: 

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-28 23:53:47 +08:00
cai.zhang ea13b97b6a
fix: Fix bug for block clustering compaction () 
issue: 

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-26 11:33:58 +08:00
congqixia e8038a76e1
fix: Remove timeout in datanode watch ctx () 
See also 

Use tickle timeout logic instead of hardcode context timeout

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-25 22:17:45 +08:00
cai.zhang 260a6e2ba9
enhance: Send flush signal when the water level reaches the high watermark () 
issue: 

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-23 10:29:42 +08:00
cai.zhang 4c45bc412f
enhance: Add integration test for clustering compaction () 
issue: 

---------

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-23 10:13:43 +08:00
cai.zhang 643b9d521c
fix: Fix the issue of concurrent packing of the same segment () 
issue: 

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-23 10:11:49 +08:00
yihao.dai 8aab6cbfac
enhance: Organize the common modules of streamingNode and dataNode () 
1. Move the common modules of streamingNode and dataNode to flushcommon
2. Add new GetVChannels interface for rootcoord

issue: https://github.com/milvus-io/milvus/issues/33285

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-22 11:33:51 +08:00
wayblink d294fdd060
enhance: add ut for clustering_compactor () 
issue: 

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-21 19:29:42 +08:00
SimFG 16c002b51b
enhance: unified channel name generation and conversion () 
/kind improvement
- issue: 

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-20 23:11:39 +08:00
wayblink c79d1af390
enhance: Add compaction task slot usage logic () 


Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-18 10:27:41 +08:00
cai.zhang 0c0ca4cf0e
fix: Fix bug where binlogs already flushed with new segment during pack () 
issue: 

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-18 09:15:48 +08:00
yihao.dai ca758c36cc
enhance: Pre-allocate ids for compaction () 
This PR removes the dependency of compaction on the ID allocator by
pre-allocating the logID and segmentID.

issue: https://github.com/milvus-io/milvus/issues/33957

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-17 13:23:42 +08:00
cai.zhang 17c96e1bde
fix: Fix the bug that caused small segment flush frequently () 
issue: 

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-17 12:15:40 +08:00
cai.zhang fbc10ac695
fix:Reset flushed row num after pack segment for clustering compaction () 
issue: 

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-16 15:37:38 +08:00
SimFG b58a5617ef
enhance: add the seal segment when dispatch delete msgs () 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-10 19:14:51 +08:00
SimFG 5016038781
enhance: release the record in delete codec and add some log for compaction () 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-09 15:40:17 +08:00
yihao.dai e15ac2b472
fix: Fix incorrect segment num rows () 
Repeated calls to UpdateStatistics, this PR correct it.

issue: https://github.com/milvus-io/milvus/issues/34440

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-08 20:00:14 +08:00
yihao.dai 4e5f1d5f75
enhance: Pre-allocate ids for import () 
The import is dependent on syncTask, which in turn relies on the
allocator. This PR pre-allocate the necessary IDs for import syncTask.

issue: https://github.com/milvus-io/milvus/issues/33957

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-07 21:26:14 +08:00
yihao.dai 43fd8d19c2
enhance: Check segment existence when FlushSegments and add some key logs () 
Check if the segment exists during FlushSegments and add some key logs
in write path.

issue: https://github.com/milvus-io/milvus/issues/34255

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-06 08:50:11 +08:00
jaime 21fc5f5d46
enhance: Remove datanode reporting TT based on MQ implementation () 
issue: 

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-05 15:48:09 +08:00
yihao.dai 0b404bff22
fix: Correct the update logic of timerecorder () 
Correct the update logic of timerecorder in the flowgraph to avoid false
failure: "some node(s) haven't received input".

issue: https://github.com/milvus-io/milvus/issues/34337

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-04 16:34:17 +08:00
XuanYang-cn 8a2be8a457
fix: DataNode might OOM by estimating based on MemorySize () 
See also: 

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-04 10:16:17 +08:00
cai.zhang 2e434e4453
fix: Only load or release Flushed segment in datanode meta () 
issue:  ,  , 

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-03 17:52:09 +08:00
cai.zhang 8165961223
enhance: Optimeize clustering compaction () 
issue: 

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-03 15:14:08 +08:00
cai.zhang 4cf1a358ba
fix: Sync the sealed and flushed segments to datanode () 
issue: 

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-01 22:42:08 +08:00
jaime d6afb31b94
enhance: make subfunctions of datanode component modular () 
issue: 

also remove deprecated channel manager based on the etcd implementation

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-01 14:46:07 +08:00
wayblink e5d691d854
Use new stream segment reader in clustering compaction () 


Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-30 20:26:07 +08:00
wayblink fbe3231b1f
fix: fix error ignore in compactor () 


Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-26 10:24:03 +08:00
jaime 9630974fbb
enhance: move rocksmq from internal to pkg module () 
issue: 

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-25 21:18:15 +08:00
cai.zhang c65f41dc60
fix: Only sync flushed segments to datanode () 
issue: 

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-25 20:46:07 +08:00
congqixia 962a5446f8
enhance: Add ctx in `SyncTask.Run` to be cancellable () 
Related to 

This PR add context param in SyncTask.Run execution functions to make it
cancellable from the caller.

This make it possible to cancel task when datanode/data sync service is
beeing shut down.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-25 14:22:04 +08:00
congqixia 506a915272
fix: Deep copy ImportTask.segmentsInfo to prevent data race () 
See also 

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-25 10:06:02 +08:00
congqixia b5c9a7364b
fix: Prevent remove new growing L1 segment when SyncSegments () 
Related to 

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-24 10:18:01 +08:00
yihao.dai 6c1d815894
enhance: Remove the unused compaction logic from shard () 
1. Remove the `compactTo` field in `SegmentInfo`.
2. Remove the target segment not match and its retry logic in
`SyncManager`.

issue: https://github.com/milvus-io/milvus/issues/32809

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-23 21:12:01 +08:00
wayblink 380d3f4469
fix: Fix memory buffer error & some renaming () 


---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-21 17:30:01 +08:00
XuanYang-cn 04edb07d82
enhance: Add deltaRowCount in l0 compaction () 
See also: 

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-20 17:46:01 +08:00
cqy123456 dc4437ff82
enhance: use segment id and type to register in MmapChunkManager and opt malloc in variableChunk () 
issue: https://github.com/milvus-io/milvus/issues/32984

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-20 17:42:02 +08:00
wei liu 31ef0a1fe8
enhance: Add trace for bf cost in l0 compactor () 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-20 10:10:05 +08:00
smellthemoon 2a1356985d
enhance: support null in go payload () 


---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-06-19 17:08:00 +08:00
wayblink 5cb0760187
fix: Small fixs of major compaction () 


Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-18 16:53:58 +08:00
cqy123456 32f685ff12
enhance: growing segment support mmap () 
issue: https://github.com/milvus-io/milvus/issues/32984

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-18 14:42:00 +08:00
cai.zhang 95148866ed
fix: Don't remove growing L0 segment in datanode metacache () 
issue:  
1. gorwing L0 segments is invisible to datacoord.
2. flushed L0 segments need to clean by datacoord.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-17 10:09:57 +08:00
yihao.dai 1a9ab52f66
enhance: Ensure the idempotency of compaction task () 
/kind enhancement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-16 22:09:57 +08:00
yihao.dai 8537f3daeb
enhance: Rename Compaction to CompactionV2 () 
Due to the removal of injection and syncSegments from the compaction, we
need to ensure that no compaction is successfully executed during the
rolling upgrade. This PR renames Compaction to CompactionV2, with the
following effects:
- New datacoord + old datanode: Utilizes the CompactionV2 interface,
resulting in the datanode error "CompactionV2 not implemented," causing
compaction to fail;
- Old datacoord + new datanode: Utilizes the CompactionV1 interface,
resulting in the datanode error "CompactionV1 not implemented," causing
compaction to fail.

issue: https://github.com/milvus-io/milvus/issues/32809

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-16 22:07:57 +08:00
yihao.dai 86a36b105a
enhance: Tidy compaction executor () 
Move compaction executor to compaction pacakge.

issue: https://github.com/milvus-io/milvus/issues/32451

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-14 14:34:01 +08:00
wei liu 4987067375
enhance: Execute bloom filter apply in parallel to speed up segment predict () 
issue: 

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-14 11:37:56 +08:00
wei liu ab93d9c23d
enhance: Use BatchPkExist to reduce bloom filter func call cost () 
issue:#33610

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-13 17:57:56 +08:00
ArenaSu 2dfa752527
doc: [skip-e2e] add comments for event manager () 
Add comments for event manager(internal/datanode/event_manager.go).

Signed-off-by: ArenaSu <704427617@qq.com>
 2024-06-13 17:56:06 +08:00
congqixia 512ea6be5f
enhance: Avoid merging insert data when buffering insert msgs () 
See also 

This PR:
- Use zero copy when buffering insert messages
- Make `storage.InsertCodec` support serialize multiple insert data
chunk into same batch binlog files

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-13 11:15:56 +08:00
congqixia 9ab3058da2
fix: Prevent restart timetick sender creating ut datanode () 
See also 

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-12 22:29:58 +08:00
yihao.dai 9a3e4080f1
enhance: Add comment for channel cp updater () 
/kind enhancement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-12 20:01:55 +08:00
coldWater 6b9901c59f
enhance: add a semaphore for CompactionExecutor () 


---------

Signed-off-by: coldWater <254244460@qq.com>
 2024-06-11 17:25:55 +08:00
yihao.dai eb5d4de390
fix: Check if the import job exists () 
issue: https://github.com/milvus-io/milvus/issues/33671

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-10 21:51:55 +08:00
wayblink a1232fafda
feat: Major compaction () 


Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-10 21:34:08 +08:00
yihao.dai 3540eee977
enhance: Support L0 import () 
issue: https://github.com/milvus-io/milvus/issues/33157

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-07 14:17:20 +08:00
XuanYang-cn 4dd0c54ca0
fix: Fix l0 compactor may cause DN from OOM () 
See also: 

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-06 14:33:52 +08:00
XuanYang-cn 68c9e7db8c
fix: Sync dropped segment for dropped partition () 
See also: 

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-06 10:25:52 +08:00