949d5d078f
Fix memory calculation in dataCodec ( #21800 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-28 11:09:52 +08:00
e977e014a9
Fix flush didn't respect binaryvector and other schemas ( #21120 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-12-12 10:33:26 +08:00
89b810a4db
Refactor all params into ParamItem ( #20987 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
9724ae5e27
Refine log level ( #20904 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-12-01 20:07:15 +08:00
2390095232
Fix load uses compacted segments' binlogs ( #20655 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-11-17 20:37:10 +08:00
21b54709a2
enable delbuf auto flush function to avoid OOM when del msg accumulating ( #20213 )
...
issue: #19713
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
Co-authored-by: MrPresent-Han <zilliz@zillizdeMacBook-Pro-2.local>
2022-11-07 18:49:02 +08:00
956c5e1b9d
Make Params singleton ( #20088 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-04 14:25:38 +08:00
2bfecf5b4e
Refine bloomfilter and memory usage ( #20168 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-31 17:41:34 +08:00
192151bc11
Refine Data Log ( #19996 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-25 19:31:30 +08:00
e081eb287d
Refine flowgraph nodes for readability ( #19864 )
...
- delete compaction tasks from executor while dropping
See also: #19780
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-10-20 16:39:29 +08:00
0095869d5b
Keep segment statslog if no insert applied ( #19910 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-10-20 14:33:27 +08:00
2785f98de6
Refactor segment replica in DN ( #19815 )
...
- Replace isNew and isFlushed with sType
- Add compacted into proto
- Rename replica to channel
- Rename segmentReplica to channelMeta
- Replace 4 maps with 1 map
- Add segment.go and segment_test.go
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-10-18 15:33:26 +08:00
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
31db01b6ae
Support TTL for collection level ( #19383 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-10 20:31:22 +08:00
a0ce533727
Add a SyncSegments to sync meta between DN and DC ( #19298 )
...
See also: #19072
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-09-27 16:02:53 +08:00
59f8e96522
Change compactor policy from loading all binlogs to loading serially ( #13716 ) ( #19280 )
...
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
2022-09-25 20:12:52 +08:00
d7f38a803d
Separate some proto files ( #19218 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
91663a6603
Fix bloom filter size not match ( #19140 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-13 14:12:31 +08:00
362c663b22
Remove deprecated inner compaction type ( #19024 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-09-07 19:16:34 +08:00
ff5201cbe5
Improve log of datanode compactor ( #18981 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-02 14:31:06 +08:00
29c72cad15
Use segment statslog instead of separate statslog ( #18775 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-02 10:34:59 +08:00
d9c7519107
refactor compaction concurency logic ( #18660 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-08-23 15:50:52 +08:00
c9174d55ba
Refine merge operation during compacting phase ( #18399 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-07-28 14:52:31 +08:00
74fd28df9c
Refine some datanode code ( #18351 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-07-21 15:52:27 +08:00
9bd35dd84e
Refine logs to reduce print ( #17759 ) ( #18028 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-07-04 11:10:20 +08:00
ed51ffd50f
Fix Compact Entity ttl param ( #16665 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-28 11:51:47 +08:00
89b4a34892
Fix Data race in NodeID ( #16603 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-24 22:03:44 +08:00
99984b88e1
Support delete varChar value ( #16229 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-04-02 17:43:29 +08:00
205c92e54b
Support insert string data ( #15993 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-03-25 14:27:25 +08:00
f3eeecf146
Reduce the number of metrics ( #16030 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-03-15 21:51:21 +08:00
74f66dce3b
Fix passing empty binlog content in compaction ( #15909 ) ( #15927 )
...
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-03-08 16:34:01 +08:00
13177a90af
Add prometheus metrics for DataNode ( #15650 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
2022-02-28 19:11:55 +08:00
0cfb6a85be
Make compaction not generate empty segment ( #15707 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2022-02-28 10:17:54 +08:00
d011073269
Fix DataNode panic when compact empty segment ( #15580 )
...
remove the TODO in compactor
See also: #15573
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-02-18 18:47:51 +08:00
5751759c85
Implement mix compaction logic ( #15542 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-02-18 14:51:49 +08:00
43b6c295a4
Put logs of compaction elapse in defer ( #15485 )
...
Before this PR, time elapse are logged after `compact()` finished
normally. No elapse will be logged with any errors.
This PR put elapse logs in defer once time-consuming steps are finished.
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-02-08 19:15:45 +08:00
46dea0f006
Add elapse logs in compaction ( #15414 )
...
See also: #15413
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-02-08 14:11:45 +08:00
f6873d3dc1
Implement automatic item expiration on compaction ( #14418 )
...
* Implement automatic item expiration on compaction
Signed-off-by: JunSung Park <writtic@gmail.com>
* Change entityExpiration value comparison logic
Signed-off-by: JunSung Park <writtic@gmail.com>
* Add logging for the number of expired entity
Signed-off-by: JunSung Park <writtic@gmail.com>
* Set default value of CompactionEntityExpiration to off
Signed-off-by: JunSung Park <writtic@gmail.com>
* Fix errors for code checker
Signed-off-by: JunSung Park <writtic@gmail.com>
* Set default entityExpiration to max value
Signed-off-by: JunSung Park <writtic@gmail.com>
2022-02-07 22:45:46 +08:00
8ab9d769de
Fix logs of compaction_executor ( #15365 )
...
- Remove not useful log in `stopExecutingtaskByVChannelName`
- Add planID info in `mergeFlushedSegments`
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-01-25 13:29:40 +08:00
87394e29b6
Fix compaction ut datarace ( #15267 )
...
Fixes : #15241
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-01-18 17:49:39 +08:00
7031a511ce
[skip e2e]Sort imports ( #14806 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-01-05 14:00:38 +08:00
3cfa88d6f2
Fix delete node not aware of merged segments error ( #14190 )
...
This PR made merge gracefully remove merged segment by moving
them into `compactedSegments`
And everytime when delete_node operates, if compactedSegments
have contents, delete_node will replace the latest segment in delBuf
merged from the merged segments, and truely remove the compactedSegments
Fixes : #14085
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-12-27 11:02:18 +08:00
891c202b73
Use GlobalParamTable for all components ( #13885 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-23 18:39:11 +08:00
a7dfd53f53
Improve import statement ( #13736 )
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-12-20 14:38:25 +08:00
1dcd06cc78
Add log file info for all kind of segment binlogs ( #13606 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-19 20:00:42 +08:00
154f11e3a1
[skip e2e]Fix redundant conversion ( #13648 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2021-12-17 22:28:50 +08:00
e191b23249
[skip e2e]Fix error string ( #13647 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2021-12-17 19:22:42 +08:00
a9a332dbcf
Fix empty segment compact and load ( #12710 )
...
1. Fix compaction save empty segment bug
2. Fix load empty segment bug
3. Add UT for dmlchannels to 100%
Resolves : #12450
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-12-09 11:03:08 +08:00
8f9e62fa18
Apply DropVirtualChannel and FlushManager drop mode ( #12563 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-02 16:39:33 +08:00
59d19384dc
Fix flush manager injection logic with multiple segments ( #12260 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-24 18:41:16 +08:00
Xiaofan
Enwei Jiao
zhenshan.cao
Bingyi Sun
MrPresent-Han
XuanYang-cn
congqixia
SimFG
jaime
zhagnlu
xige-16
Letian Jiang
bigsheeper
yah01
JunSung Park
godchen
Cai Yudong
Xiangyu Wang