Commit Graph

20 Commits (ffaead6ad972ba9592ed06330b197cbd3134f2c8)

Author SHA1 Message Date
Xiaofan ed51ffd50f
Fix Compact Entity ttl param (#16665)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-28 11:51:47 +08:00
xige-16 99984b88e1
Support delete varChar value (#16229)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-04-02 17:43:29 +08:00
xige-16 205c92e54b
Support insert string data (#15993)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-03-25 14:27:25 +08:00
godchen 78557ca6d7
Replace minio kv with minio chunk manager (#15936)
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-03-17 18:03:23 +08:00
Letian Jiang 74f66dce3b
Fix passing empty binlog content in compaction (#15909) (#15927)
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-03-08 16:34:01 +08:00
yah01 0cfb6a85be
Make compaction not generate empty segment (#15707)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2022-02-28 10:17:54 +08:00
XuanYang-cn 6f85b49a4f
Add more compaction test cases (#15715)
Test merge compaction with 2 segments with the same PK and only 1
valid deletion.

This test can varify the compaction behavior of DataNode for the
dup PK in different segment cases.

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-02-24 11:27:53 +08:00
XuanYang-cn d011073269
Fix DataNode panic when compact empty segment (#15580)
remove the TODO in compactor

See also: #15573

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-02-18 18:47:51 +08:00
JunSung Park f6873d3dc1
Implement automatic item expiration on compaction (#14418)
* Implement automatic item expiration on compaction

Signed-off-by: JunSung Park <writtic@gmail.com>

* Change entityExpiration value comparison logic

Signed-off-by: JunSung Park <writtic@gmail.com>

* Add logging for the number of expired entity

Signed-off-by: JunSung Park <writtic@gmail.com>

* Set default value of CompactionEntityExpiration to off

Signed-off-by: JunSung Park <writtic@gmail.com>

* Fix errors for code checker

Signed-off-by: JunSung Park <writtic@gmail.com>

* Set default entityExpiration to max value

Signed-off-by: JunSung Park <writtic@gmail.com>
2022-02-07 22:45:46 +08:00
XuanYang-cn 3cfa88d6f2
Fix delete node not aware of merged segments error (#14190)
This PR made merge gracefully remove merged segment by moving
them into `compactedSegments`

And everytime when delete_node operates, if compactedSegments
have contents, delete_node will replace the latest segment in delBuf
merged from the merged segments, and truely remove the compactedSegments

Fixes: #14085

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-12-27 11:02:18 +08:00
Xiangyu Wang cabc40cb09
Improve import statement (#13737)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-12-20 14:39:12 +08:00
congqixia 1dcd06cc78
Add log file info for all kind of segment binlogs (#13606)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-19 20:00:42 +08:00
congqixia e813867652
Implement Drop mode for flush manager (#12469)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-01 10:11:39 +08:00
congqixia 59d19384dc
Fix flush manager injection logic with multiple segments (#12260)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-24 18:41:16 +08:00
XuanYang-cn 6bf0c4326d
Fix compaction generates wrong insert binlogs (#12204)
This PR
- changes Get() to GetRow() in BinlogIterator
- removes Get() and Length() of FieldData
- adds ut in BinlogIterator for floatvector and binaryvector
- fix a log bug

Fixes: #12146

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-23 17:09:15 +08:00
congqixia 5e550a8793
Add close for flushManager (#11874)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-17 10:39:15 +08:00
XuanYang-cn 9bd3664672
Fix compaction bug (#11816)
This PR fixes:
- timeout in compaction
- BinlogIterator using rawID as PKField bug

See also: #11757

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-17 10:07:13 +08:00
XuanYang-cn c4bb73dc74
Complete drop collection procedure in DN (#11552)
drop, cancel compaction, flush all buffers and release fg

See also: #11426

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-11 20:56:49 +08:00
godchen 863f1bb34e
Fix multi delete data not effect (#11422)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-11-09 15:01:17 +08:00
XuanYang-cn 3cd0584a04
Add compaction in datanode (#11353)
See also: #9530

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-08 19:49:07 +08:00