Commit Graph

45 Commits (eb046863485fdf3e130fc60484485c901b81276b)

Author SHA1 Message Date
congqixia cb7f2fa6fd
enhance: Use v2 package name for pkg module (#39990)
Related to #39095

https://go.dev/doc/modules/version-numbers

Update pkg version according to golang dep version convention

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
Ted Xu 2978b0890e
enhance: iterative download data during compaction to reduce memory cost (#39724)
See #37234

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2025-02-13 10:36:47 +08:00
XuanYang-cn b8fca4f5c1
fix: Clustering compaction ignoring deltalogs (#39132)
See also: #39131

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-01-10 14:07:05 +08:00
Zhen Ye bb8d1ab3bf
enhance: make new go package to manage proto (#39114)
issue: #39095

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:49:01 +08:00
XuanYang-cn c731357538
enhance: Add missing delete metrics (#38634)
Add 2 counter metrics:
- Total delete entries from deltalog:
milvus_datanode_compaction_delete_count
- Total missing deletes: milvus_datanode_compaction_missing_delete_count

See also: #34665

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-12-25 10:24:50 +08:00
tinswzy 27229f7907
enhance: refine exists log print with ctx (#38080)
issue: #35917 
Refines exists log print with ctx

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-14 22:36:44 +08:00
cai.zhang 6ffc57c8dc
fix: Fix sorting buffer in clustering compaction (#38417)
issue: #28410

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-12-13 10:12:49 +08:00
cai.zhang 41b19c6b1d
enhance: Determine the number of buffers based on the resource limits of the DataNode (#38209)
issue: #28410

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-12-08 18:02:40 +08:00
yihao.dai 81879425e1
enhance: Optimize the performance of stats task (#37374)
1. Increase the writer's `batchSize` to avoid multiple serialization
operations.
2. Perform asynchronous upload of binlog files to prevent blocking the
data processing flow.
3. Reduce multiple calls to `writer.Flush()`.

issue: https://github.com/milvus-io/milvus/issues/37373

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-08 10:08:27 +08:00
Ted Xu b792b199d7
enhance: load deltalogs on demand when doing compactions (#37310)
See #37234

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-11-01 16:40:21 +08:00
cai.zhang 04c306e63f
fix: Fix clustering compaction task leak (#36800)
issue: #36686 

bug reason:
- The clustering compaction tasks on the datanode were never cleaned up.
- The clustering compaction task contains a mapping from clustering key
to buffer, this caused a large memory leak.

fix:
- clean the tasks on datanode by datacoord when clustering compaction
finished.
- reset the mapping that from clustering key to buffer on datanode when
clustering finished.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-10-17 20:43:30 +08:00
aoiasd 5ec4163d0f
feat: support bm25 logs mixcompaction (#36072)
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-14 16:57:22 +08:00
wayblink 00a5025949
enhance: support clustering compaction on null value (#36372)
issue: #36055

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-09-30 14:33:17 +08:00
cai.zhang 2adca8b754
fix: Fix data race for cluerting compaction (#36440)
issue: #36438

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-28 17:19:21 +08:00
smellthemoon 3f75bf1f20
fix: clustering compact not support null (#36152)
#36055

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-11 14:49:06 +08:00
cai.zhang 90bdb171ab
fix: Fix data race for clustering compaction writer (#35957)
issue: #35950

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-05 04:07:10 +08:00
cai.zhang 2c9bb4dfa3
feat: Support stats task to sort segment by PK (#35054)
issue: #33744 

This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-02 14:19:03 +08:00
yihao.dai 1413ffe9b1
enhance: Rename preAllocatedSegments (#35871)
Rename `preAllocatedSegments` to `preAllocatedSegmentIDs` to avoid
confusion.

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-01 17:09:01 +08:00
XuanYang-cn 323400c190
enhance: Enable to write multiple segments in mix compactor (#35705)
Prevent segments to be written larger than maxSize * expansionRate

See also: #35584

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-30 11:29:01 +08:00
XuanYang-cn 967f38672a
enhance: Add integration tests for l0 (#35429)
See also: #34796

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-19 10:56:54 +08:00
cai.zhang 1bbf7a3c0e
enhance: Optimize the use of locks and avoid double flush clustering buffer writer (#35486)
issue: #35436

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-16 02:24:58 +08:00
cai.zhang 196b343a94
fix: Fix data race for clustering compaction (#35435)
issue: #35436

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-13 17:10:20 +08:00
cai.zhang aaab827a16
fix: Fix the issue of missing stats log after clustering compaction (#35266)
issue: #35265

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-08 14:24:17 +08:00
yihao.dai a4439cc911
enhance: Implement flusher in streamingNode (#34942)
- Implement flusher to:
  - Manage the pipelines (creation, deletion, etc.)
  - Manage the segment write buffer
  - Manage sync operation (including receive flushMsg and execute flush)
- Add a new `GetChannelRecoveryInfo` RPC in DataCoord.
- Reorganize packages: `flushcommon` and `datanode`.

issue: https://github.com/milvus-io/milvus/issues/33285

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-02 18:30:23 +08:00
cai.zhang 9412002d7d
fix: Fix data race for clustering buffer writer (#35145)
issue: #34495

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-01 11:20:13 +08:00
congqixia de8a266d8a
enhance: Enable linux code checker (#35084)
See also #34483

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 15:53:51 +08:00
wei liu c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 (#34394)
issue: #34252

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
cai.zhang 2372452fac
enhance: Optimized the GC logic to ensure that memory is released in time (#34949)
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-28 23:53:47 +08:00
cai.zhang ea13b97b6a
fix: Fix bug for block clustering compaction (#35019)
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-26 11:33:58 +08:00
cai.zhang 260a6e2ba9
enhance: Send flush signal when the water level reaches the high watermark (#34907)
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:29:42 +08:00
cai.zhang 4c45bc412f
enhance: Add integration test for clustering compaction (#34881)
issue: #34792

---------

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:13:43 +08:00
cai.zhang 643b9d521c
fix: Fix the issue of concurrent packing of the same segment (#34840)
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:11:49 +08:00
wayblink d294fdd060
enhance: add ut for clustering_compactor (#34852)
issue: #34792

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-21 19:29:42 +08:00
wayblink c79d1af390
enhance: Add compaction task slot usage logic (#34581)
#34544

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-18 10:27:41 +08:00
cai.zhang 0c0ca4cf0e
fix: Fix bug where binlogs already flushed with new segment during pack (#34762)
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-18 09:15:48 +08:00
yihao.dai ca758c36cc
enhance: Pre-allocate ids for compaction (#34187)
This PR removes the dependency of compaction on the ID allocator by
pre-allocating the logID and segmentID.

issue: https://github.com/milvus-io/milvus/issues/33957

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-17 13:23:42 +08:00
cai.zhang 17c96e1bde
fix: Fix the bug that caused small segment flush frequently (#34725)
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-17 12:15:40 +08:00
cai.zhang fbc10ac695
fix:Reset flushed row num after pack segment for clustering compaction (#34702)
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-16 15:37:38 +08:00
jaime 21fc5f5d46
enhance: Remove datanode reporting TT based on MQ implementation (#34421)
issue: #34420

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-05 15:48:09 +08:00
cai.zhang 8165961223
enhance: Optimeize clustering compaction (#34313)
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-03 15:14:08 +08:00
wayblink e5d691d854
Use new stream segment reader in clustering compaction (#34232)
#32939

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-30 20:26:07 +08:00
wayblink fbe3231b1f
fix: fix error ignore in compactor (#34169)
#34170

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-26 10:24:03 +08:00
wayblink 380d3f4469
fix: Fix memory buffer error & some renaming (#33850)
#30633

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-21 17:30:01 +08:00
wayblink 5cb0760187
fix: Small fixs of major compaction (#33929)
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-18 16:53:58 +08:00
wayblink a1232fafda
feat: Major compaction (#33620)
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00