a8a0aa9357
fix: missing to support compact for Array type ( #29505 )
...
the array type can't be compacted, the system could continue with the
inserted segments, but these segments can be never compacted
fix #29503
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-28 15:42:48 +08:00
fe04598900
enhance: Add compaction type label to metrics ( #29485 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-27 15:56:48 +08:00
aae7e62729
feat: Add levelzero compaction in DN ( #28470 )
...
See also: #27606
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-30 14:30:28 +08:00
321c5c32e3
fix: Separate schedule and check results loop ( #28692 )
...
This PR:
- Separates compaction scheduler and check results loop So that slow in
check-loop doesn't influence execution.
- Cleans compaction tasks when drop a vchannel so dropped-channel's
compaction tasks won't be checked over and over again.
- Skips meta change when meta's already changed, avoid panic
- Remove not inuse injectDone(bool) parameter
See also: #28628 , #28209
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-29 10:50:29 +08:00
2b3fa8f67b
fix: Add length check for `storage.NewPrimaryKeyStats` ( #28576 )
...
See also #28575
Add zero-length check for `storage.NewPrimaryKeyStats`. This function
shall return error when non-positive rowNum passed.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-21 10:28:21 +08:00
bed7467f20
enhance: Remove commented code and fix naming issue ( #28450 )
...
This PR removes all the commented code and files from PR #28320
For naming issue:
- Renaming `MinCheckpoint` to `EarliestPosition`, see #28320 comment
- Renaming `writebuffer.Mananger` to `BufferMananger`, see #27874
comment
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-16 00:22:20 +08:00
0b905078e7
Use writebuffer, sync manager refactory in datanode ( #28320 )
...
See also #27675
This PR make previously merged refactory of datanode go online
- Use write node to replace insert/delete node
- Use write buffer manager to control all buffers
- Use sync manager to control sync tasks instead of flush manager
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-15 15:24:18 +08:00
40d5c902b6
Enable getting multiple segments in plan result ( #28350 )
...
Compaction plan result contained one segment for one plan. For l0
compaction would write to multiple segments, this PR expand the segments
number in plan results and refactor some names for readibility.
- Name refactory: - CompactionStateResult -> CompactionPlanResult -
CompactionResult -> CompactionSegment
See also: #27606
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-14 15:56:19 +08:00
294ff74ca5
Add new compaction types ( #27608 )
...
See also: #27606
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-10-11 06:49:32 +08:00
1d76565894
Add metrics for garbage collection ( #27303 )
...
Also fix second metrics usage in compaction
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-22 18:47:25 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
6635398a6d
Fix Bin log concurrency by adding a pool ( #27189 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-09-19 18:05:22 +08:00
9166011c4a
Add float16 vector ( #25852 )
...
Signed-off-by: Writer-X <1256866856@qq.com>
2023-09-08 10:03:16 +08:00
b2e7cbdf4b
Remove TimeTravel in compactor ( #26785 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-04 17:41:48 +08:00
8d54509e54
Fix CompactionLatency metrics ( #26747 )
...
- Refine compactor logs
See also: #26743
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-08-31 17:35:03 +08:00
21eeb37ce9
Save binlog timestampFrom, timestampTo meta when compact ( #26203 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-08-08 21:17:21 +08:00
33c2012675
Add more metrics ( #25081 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-26 17:52:44 +08:00
3f4356df10
fix the spelling of `field` ( #25008 )
...
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-06-21 14:00:42 +08:00
b75dcf90c0
Fix Flush hang after SyncSegments timeout ( #24953 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-06-20 10:20:41 +08:00
f57fe6d70b
Add compaction log ( #24976 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-06-19 14:18:41 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
c84bdcea49
merge stats log when segment flushing or compacting ( #23570 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-05-29 10:21:28 +08:00
99cc24b283
Fix compaction doesn't support JSON data ( #24151 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-05-17 19:09:22 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
93bc805933
Enhance ID allocator in DataNode ( #22905 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-03-23 19:43:57 +08:00
4a90490a67
Fix the `segment not found` error ( #22772 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-03-17 17:27:56 +08:00
8799b06dbc
Fix the data is disappeared when upsert ( #22762 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-03-14 19:11:54 +08:00
697dedac7e
Use cockroachdb/errors to replace other error pkg ( #22390 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
0ce70b4f10
fix compact task data race ( #22299 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-02-21 16:10:26 +08:00
949d5d078f
Fix memory calculation in dataCodec ( #21800 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-28 11:09:52 +08:00
e977e014a9
Fix flush didn't respect binaryvector and other schemas ( #21120 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-12-12 10:33:26 +08:00
89b810a4db
Refactor all params into ParamItem ( #20987 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
9724ae5e27
Refine log level ( #20904 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-12-01 20:07:15 +08:00
2390095232
Fix load uses compacted segments' binlogs ( #20655 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-11-17 20:37:10 +08:00
21b54709a2
enable delbuf auto flush function to avoid OOM when del msg accumulating ( #20213 )
...
issue: #19713
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
Co-authored-by: MrPresent-Han <zilliz@zillizdeMacBook-Pro-2.local>
2022-11-07 18:49:02 +08:00
956c5e1b9d
Make Params singleton ( #20088 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-04 14:25:38 +08:00
2bfecf5b4e
Refine bloomfilter and memory usage ( #20168 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-31 17:41:34 +08:00
192151bc11
Refine Data Log ( #19996 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-25 19:31:30 +08:00
e081eb287d
Refine flowgraph nodes for readability ( #19864 )
...
- delete compaction tasks from executor while dropping
See also: #19780
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-10-20 16:39:29 +08:00
0095869d5b
Keep segment statslog if no insert applied ( #19910 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-10-20 14:33:27 +08:00
2785f98de6
Refactor segment replica in DN ( #19815 )
...
- Replace isNew and isFlushed with sType
- Add compacted into proto
- Rename replica to channel
- Rename segmentReplica to channelMeta
- Replace 4 maps with 1 map
- Add segment.go and segment_test.go
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-10-18 15:33:26 +08:00
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
31db01b6ae
Support TTL for collection level ( #19383 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-10 20:31:22 +08:00
a0ce533727
Add a SyncSegments to sync meta between DN and DC ( #19298 )
...
See also: #19072
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-09-27 16:02:53 +08:00
59f8e96522
Change compactor policy from loading all binlogs to loading serially ( #13716 ) ( #19280 )
...
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
2022-09-25 20:12:52 +08:00
d7f38a803d
Separate some proto files ( #19218 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
91663a6603
Fix bloom filter size not match ( #19140 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-13 14:12:31 +08:00
362c663b22
Remove deprecated inner compaction type ( #19024 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-09-07 19:16:34 +08:00
ff5201cbe5
Improve log of datanode compactor ( #18981 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-02 14:31:06 +08:00
29c72cad15
Use segment statslog instead of separate statslog ( #18775 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-02 10:34:59 +08:00
yah01
XuanYang-cn
congqixia
SimFG
Xiaofan
Xu Tong
wayblink
xige-16
PowderLi
jaime
aoiasd
yah01
smellthemoon
Enwei Jiao
wei liu
zhenshan.cao
Bingyi Sun
MrPresent-Han
zhagnlu