Commit Graph

659 Commits (2265e81d974719516c8012bef3ed3119b003d8e2)

Author SHA1 Message Date
congqixia 7adabe09f1
Fix Alloc timstamp failure blocks compaction queue forever ()
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-02-07 19:02:31 +08:00
XuanYang-cn a8db19fbdb
Fix ut multi-remove the timer ()
See also: 

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-02-07 14:15:54 +08:00
cai.zhang e25f987a5c
Fix bug that calculated an extra DataNode of register policy ()
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-02-06 10:33:54 +08:00
cai.zhang ae305a54bd
Set scheduler duration of index task can be configurable ()
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-02-05 17:21:53 +08:00
Ten Thousand Leaves f9495fef00
Minor fix on segment metric ()
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2023-02-03 19:33:52 +08:00
jaime 748a935a11
Modify segment seal policy with number of statslog files ()
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-02-03 17:07:52 +08:00
congqixia 787ce7c121
Add Rated group for checkpoint gc skip case ()
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-02-03 14:23:56 +08:00
Xiaofan 2213ed7bed
Fix bulkload segment can not be flushed ()
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-02-02 13:11:51 +08:00
congqixia 11a3d0a50b
Use embed ETCD for datacoord unit test ()
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-01-31 14:47:49 +08:00
Xiaofan f1daef22a1
Fix failed bulkload segment marked as sealed ()
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-31 12:41:53 +08:00
cai.zhang bcae97b125
Register after start to prevent there are tow coordinator at the same time () ()
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-30 11:11:50 +08:00
congqixia 5986106037
Make paramtable init only once ()
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-01-19 14:53:44 +08:00
jaime 1dcd9eeb79
Support rename collection ()
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-01-19 14:13:43 +08:00
yah01 78fd1987eb
Fix data race caused by no clone while updating index ()
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-01-17 18:11:42 +08:00
Enwei Jiao 90d9e165d4
Fix some configs not shown ()
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-01-13 15:31:41 +08:00
congqixia 204db183f3
Prevent panicking when index segment info not exists ()
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-01-12 21:39:40 +08:00
wayblink 6a722396bd
Integration test framework ()
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-01-12 19:49:40 +08:00
Enwei Jiao fb42466c65
Use opentelemetry ()
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-01-12 16:09:39 +08:00
Jiquan Long 9fd9bed2b9
Remove collection meta after GC finished ()
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-01-12 09:55:42 +08:00
Ten Thousand Leaves 8c2c04517e
Fix an issue where flushed segments can turn into sealed ()
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2023-01-11 18:57:43 +08:00
cai.zhang f02ccfb422
Set GetIndexBuildProgress is compatible with SDK version 2.1 ()
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-11 16:59:42 +08:00
cai.zhang e127cf7b99
Reset indexpb for upgrade ()
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-11 14:35:40 +08:00
cai.zhang 5d660f16d6
Forbid IndexCoord start with version<2.3 ()
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-10 14:33:38 +08:00
Ten Thousand Leaves 23ceb9c638
Remove index checking phase for bulk insert ()
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2023-01-09 21:15:38 +08:00
congqixia 42ab1b0a48
Add segment gc restraint of dml position before channel cp ()
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-01-06 21:33:36 +08:00
jaime 73d8509600
Improve ListSegments performance for datacoord catalog ()
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-01-06 14:33:36 +08:00
cai.zhang aa203acfb3
Low IndexCoord weight ()
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-06 14:21:37 +08:00
cai.zhang e5f408dceb
Merge IndexCoord and DataCoord ()
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-04 19:37:36 +08:00
yah01 7b39873ae0
limit the frequency of GetMetrics() log ()
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-01-04 17:39:35 +08:00
Jiquan Long ff2a68e65a
Fix collection not exist when tried to do recovery ()
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-01-04 16:37:35 +08:00
bigsheeper cf40bb9b5c
Improve datacoord log ()
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-12-30 15:37:31 +08:00
Xiaofan 5d044c4f29
Add check for stats log and bin log number ()
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-12-26 16:35:30 +08:00
jaime d401608899
Add a segment seal policy by number of binlog files ()
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-12-20 14:09:25 +08:00
Ten Thousand Leaves ef3fcdf65b
Small candidate compaction should only happen with >1 segments ()
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-12-16 10:01:23 +08:00
jaime 0954014c6d
[skip e2e] Fix broken default parameters for seal segment policy ()
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-12-15 18:31:23 +08:00
bigsheeper fc10c74005
Use channel-cp as seekPosition when FromDmlCPLoadDelete ()
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-12-13 16:17:22 +08:00
Ten Thousand Leaves a81c9cc11c
Recalculate segment max size during compaction ()
issue: 

/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-12-13 15:39:21 +08:00
XuanYang-cn 71cab36e37
Fix channelmanager ut ()
See also: 

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-12-12 19:31:22 +08:00
congqixia 53782ab83e
Fill collection schema after DataCoord restarts ()
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-12-12 19:05:21 +08:00
Xiaofan e977e014a9
Fix flush didn't respect binaryvector and other schemas ()
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-12-12 10:33:26 +08:00
Ten Thousand Leaves d67db2ed9b
Make compaction target able to exceed segment max size by a bit ()
issue: 
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-12-09 16:03:20 +08:00
aoiasd de0ab9e2cf
Refactor showConfigurations to allow return global config rather than only return config of this component ()
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2022-12-09 14:31:21 +08:00
Enwei Jiao 89b810a4db
Refactor all params into ParamItem ()
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
congqixia 5e49b095f5
Fix compaction handler may double release and block forever ()
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-12-06 20:09:18 +08:00
Ten Thousand Leaves 2f5b03fef7
Split small segment proportion and small segment compactable proportion ()
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-12-06 14:37:18 +08:00
jaime 039e9ce4bb
Refine msgstream interface ()
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-12-05 20:55:17 +08:00
XuanYang-cn 243d8cff82
Provide different channelNames for channelmanager test ()
Fixes: 

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-12-05 15:39:22 +08:00
cai.zhang 9a42cff9b4
GetFlushedSegments should include flushing segment ()
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-12-01 16:35:20 +08:00
Ten Thousand Leaves c373b44c2c
Verify and adjust segment # of rows before saving and passing SegmentInfo ()
issue: 

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-12-01 16:31:16 +08:00
Enwei Jiao 2ecdb4ba4a
Etcd config source support TLS ()
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-30 18:23:15 +08:00