Commit Graph

740 Commits (a577cac46b86e338382ac7191188342ea6db7440)

Author SHA1 Message Date
smellthemoon ce91f61e5a
Pre-create topics (#21907)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-04-03 16:44:23 +08:00
XuanYang-cn be0827417a
Fix buffer calculate memorysize negative (#23152)
Along with the following small patches:
- And fix logging dup segmentIDs for delete buffer status
- Add String method for PriorityQueue to debug

See also: #23105

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-03-31 18:30:23 +08:00
Jiquan Long 0f2c3ac832
Optimize log (#23121)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-03-30 10:44:22 +08:00
Enwei Jiao 50dab2d394
Add tracing for retrieve (#23033)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-03-29 18:10:02 +08:00
XuanYang-cn 9691315c21
Fix flush deltabuf with segmentID=0 (#23059)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-03-28 19:14:01 +08:00
aoiasd 5d172d0f4f
Refine errors of datanode (#22852)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-03-28 19:04:00 +08:00
zhenshan.cao e3c3c949c4
Fix datarace of flow_graph_manager_test.go in package datanode (#23043)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-03-28 10:42:00 +08:00
yihao.dai b21f03682a
Fix flowgraph manager test (#23009)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-03-27 18:00:05 +08:00
congqixia 4008ee6d60
Remove datanode ddNode forward delta logic (#23005)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-27 14:18:00 +08:00
yah01 081572d31c
Refactor QueryNode (#21625)
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
2023-03-27 00:42:00 +08:00
Xiaofan 5a914aedf3
Make Datacoord TT to be configurable (#22956)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-03-24 22:24:04 +08:00
XuanYang-cn da58ba6a49
Fix endless appending sids (#22977)
See also: #22976

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-03-24 18:00:03 +08:00
congqixia 8f847884c2
Fix misc copylocks and remove exclude rule (#22962)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-24 11:17:58 +08:00
XuanYang-cn 93bc805933
Enhance ID allocator in DataNode (#22905)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-03-23 19:43:57 +08:00
yihao.dai f2ffb5db26
Use buffer size at memory sync policy (#22825)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-03-21 21:37:56 +08:00
SimFG 4a90490a67
Fix the `segment not found` error (#22772)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-03-17 17:27:56 +08:00
smellthemoon 8799b06dbc
Fix the data is disappeared when upsert (#22762)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-03-14 19:11:54 +08:00
congqixia 732986aa04
Remove fmt.Print from internal package (#22722)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-14 17:36:05 +08:00
aoiasd 6959332b08
Fix unittest start channel event but not close (#22662)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-03-14 17:05:54 +08:00
wayblink aa6212d5b0
[Cherry-pick] Fix channel checkpoint issues and fix GC (#22559)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: Ten Thousand Leaves <69466447+soothing-rain@users.noreply.github.com>
2023-03-09 14:13:52 +08:00
jaime d126f06946
Decouple mq module from internal proto definition (#22536)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-03-04 23:21:50 +08:00
yah01 60cd548bf5
Rename concurrency package to conc (#22453)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-02-28 14:19:47 +08:00
yah01 167581c08a
Refine Pool to improve the ease of use (#22412)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-02-27 18:07:48 +08:00
wayblink 416866e42f
Add memory usage too large sync policy (#22244)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-02-27 17:47:51 +08:00
yah01 b758c305a7
Make Future generic (#22411)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-02-27 12:07:47 +08:00
Xiaofan 3d6bded115
Fix Row Number Mismatch (#22307)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-02-27 10:41:46 +08:00
Enwei Jiao 697dedac7e
Use cockroachdb/errors to replace other error pkg (#22390)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
zhenshan.cao e768437681
Correct usage of Timer and Ticker (#22228)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-02-23 18:59:45 +08:00
aoiasd 27b456dd46
Fix data race of data node test (#22221)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-02-22 10:28:26 +08:00
wei liu 0ce70b4f10
fix compact task data race (#22299)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-02-21 16:10:26 +08:00
Enwei Jiao 0a9a9058b9
Fix UT timeout (#22261)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-20 21:14:25 +08:00
aoiasd 9167773eff
Remove needless kv at dataSyncService (#22206)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-02-16 16:36:39 +08:00
congqixia 9346f1752a
Convert msg pchannel to vchannel before check IsCloseMsg (#22182)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-02-15 17:26:34 +08:00
aoiasd 148a024e05
Add tickle for datacoord watch event (#21193)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-02-15 16:20:34 +08:00
aoiasd 1894766235
Update datacoord compaction plan after datanode update plan to ensure consistency (#22143)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-02-15 16:00:33 +08:00
smellthemoon e20d79a8a1
Fix data race (#22171)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-02-15 15:40:33 +08:00
bigsheeper d2667064bb
Add msgDispatcher to support sharing msgs for different vChannel (#21917)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-02-13 16:38:33 +08:00
cai.zhang 511265c68c
Modify lastSyncTime in advance to prevent multiple flush binlogs (#22048) (#22088)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-02-12 18:56:33 +08:00
XuanYang-cn 2b982b5d8f
Fix DataNode ut never meet condition (#22093)
See also: #22079

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-02-10 11:04:32 +08:00
yah01 9b491858a5
Fix datanode unit test not stable (#21911)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-02-01 18:19:50 +08:00
Xiaofan 949d5d078f
Fix memory calculation in dataCodec (#21800)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-28 11:09:52 +08:00
congqixia 5986106037
Make paramtable init only once (#21782)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-01-19 14:53:44 +08:00
Enwei Jiao 90d9e165d4
Fix some configs not shown (#21653)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-01-13 15:31:41 +08:00
wayblink 6a722396bd
Integration test framework (#21283)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-01-12 19:49:40 +08:00
Enwei Jiao fb42466c65
Use opentelemetry (#21509)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-01-12 16:09:39 +08:00
groot 9f8095996b
Report bulkinsert progress (#21612) (#21638)
Signed-off-by: groot <yihua.mo@zilliz.com>
2023-01-11 17:37:44 +08:00
groot df0ffd08ce
Update segment id for import task (#21621)
Signed-off-by: groot <yihua.mo@zilliz.com>
2023-01-11 11:55:39 +08:00
Xiaofan 4b4944ecee
Support sync all segments while close (#21421)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-06 14:49:36 +08:00
cai.zhang efefba4da4
Fix checkpoint will not update after delete and compact (#21495) (#21499)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-05 11:31:35 +08:00
yah01 7b39873ae0
limit the frequency of GetMetrics() log (#21514)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-01-04 17:39:35 +08:00
Jiquan Long ff2a68e65a
Fix collection not exist when tried to do recovery (#21471)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-01-04 16:37:35 +08:00
congqixia 3a58ac9fba
Change node stop to TearDownTest (#21422)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-12-28 20:11:30 +08:00
Xiaofan 3667bb09b8
Avoid Sync when the segment open (#21397)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-12-28 10:51:34 +08:00
congqixia 3c9840d6d3
Fix some datanode instance not stopped in unit tests (#21385)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-12-26 16:59:30 +08:00
Enwei Jiao 166e9f0da5
Refactor GrpcConfig (#21142)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-16 15:59:23 +08:00
bigsheeper f595500383
Move sync delete policy from deleteNode to insertBufferNode (#21152)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-12-13 16:15:21 +08:00
XuanYang-cn a7b24cbc53
Move APIs of types.DataNode to services.go (#21042)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-12-12 10:57:22 +08:00
Xiaofan e977e014a9
Fix flush didn't respect binaryvector and other schemas (#21120)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-12-12 10:33:26 +08:00
aoiasd de0ab9e2cf
Refactor showConfigurations to allow return global config rather than only return config of this component (#21063)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2022-12-09 14:31:21 +08:00
Enwei Jiao 89b810a4db
Refactor all params into ParamItem (#20987)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
groot 33a102c6fc
Refine bulkinsert (#20986)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-12-07 14:53:18 +08:00
XuanYang-cn f844aa0cab
Remove not inuse stale segments in flushmsg (#20981)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-12-06 17:51:18 +08:00
jaime 039e9ce4bb
Refine msgstream interface (#20832)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-12-05 20:55:17 +08:00
congqixia 2791f5526c
Fix BufferData limit calculation for BinaryVector (#20963)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-12-05 11:15:17 +08:00
congqixia 4238b5dbab
Make datanode unittest use embed etcd server (#20948)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-12-02 16:05:16 +08:00
congqixia 40abb13413
Generate unique id with ts for sync task (#20928)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-12-01 20:35:18 +08:00
congqixia f745d7f489
Fix compaction target segment rowNum is always 0 (#20937)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-12-01 20:33:17 +08:00
zhenshan.cao 9724ae5e27
Refine log level (#20904)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-12-01 20:07:15 +08:00
Ten Thousand Leaves b238f4ee58
Remove bad & duplicate metric (#20905)
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-12-01 16:39:16 +08:00
congqixia 82fe655032
Setup lastSyncTs when add segment to avoid sync next timetick (#20920)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-12-01 12:29:16 +08:00
Enwei Jiao 2ecdb4ba4a
Etcd config source support TLS (#20874)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-30 18:23:15 +08:00
cai.zhang 83244afcf3
Reduce some logs level which is frequent (#20894)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-11-30 14:07:15 +08:00
SimFG 44d45452fa
Make the sync segment request idempotent in the data node (#20707)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-11-24 10:21:13 +08:00
groot bf2107ecf5
Fix a regression of local storage (#20653)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-11-21 10:19:10 +08:00
groot c8022ea63c
Change import task timeout value (#20717)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-11-21 10:11:11 +08:00
jaime 0a5270e8c0
Fix alter collection hang (#20694)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-11-21 10:09:10 +08:00
Bingyi Sun 2390095232
Fix load uses compacted segments' binlogs (#20655)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-11-17 20:37:10 +08:00
Enwei Jiao c05b9ad539
Add event dispatcher for config (#20393)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-17 18:59:09 +08:00
Enwei Jiao 19524a5344
Fix nodeID mismatch at standalone mode (#20648)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-17 17:15:08 +08:00
XuanYang-cn bd20536c6a
Reduce unnecessary logs of get compaction results (#20642)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-11-17 10:09:12 +08:00
bigsheeper 2d307c1909
Set delBuf's start-end position of compacted segment (#20614)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-11-15 19:41:16 +08:00
MrPresent-Han c848896bc8
Fix panic error due to rollDeleteOp ahead by load-and-delete (#20563)
issue:#20501
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
2022-11-15 14:45:07 +08:00
bigsheeper db33ffa518
Pass endPosition with vchannel in datanode flowgraph (#20589)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-11-15 14:37:07 +08:00
smellthemoon a7ba416b5a
Remove invalid timestamp (#20557)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-11-14 16:21:07 +08:00
bigsheeper fc15789da9
Ensure compatibility of channel seek position and move syncPeriod to config (#20504)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-11-12 21:09:04 +08:00
bigsheeper cd19d99ad7
Add channel level checkpoint (#20350)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-11-10 22:13:04 +08:00
XuanYang-cn 6bff0b6688
Remove too frequent logs (#20454)
Half of the log messages come from this log if DataNode in stale.

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-11-10 14:51:05 +08:00
Enwei Jiao 7e56e7a976
Fix datarace at Setlogger (#20376)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-08 14:33:03 +08:00
wayblink a7bed1c927
Pass backup flag through Import request options (#20334)
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-11-08 11:33:03 +08:00
MrPresent-Han 21b54709a2
enable delbuf auto flush function to avoid OOM when del msg accumulating (#20213)
issue: #19713

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
Co-authored-by: MrPresent-Han <zilliz@zillizdeMacBook-Pro-2.local>
2022-11-07 18:49:02 +08:00
jaime 709a0a94e9
Add some msgstream metrics (#20296)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-11-07 10:15:02 +08:00
Enwei Jiao 956c5e1b9d
Make Params singleton (#20088)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-04 14:25:38 +08:00
congqixia f557af1143
Verify target id for FlushSegments request in datanode (#20308)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-11-03 22:47:35 +08:00
bigsheeper d528aa3d90
Print seek postion time and seek elapse (#20271)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-11-03 15:03:35 +08:00
MrPresent-Han e0baf1536e
[skip e2e] Improve a commment for initial pr (#20287)
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
2022-11-03 14:05:36 +08:00
Ten Thousand Leaves 5708352a80
Improve DataCoord and DataNode logs (#20265)
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-11-03 12:15:35 +08:00
Xiaofan 33349677fc
Fix unsafe lock (#20219)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-11-01 19:37:35 +08:00
Xiaofan 99b958e360
Fix BF Concurrency issue (#20211)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-31 19:27:35 +08:00
Xiaofan 2bfecf5b4e
Refine bloomfilter and memory usage (#20168)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-31 17:41:34 +08:00
jaime 48f5c60070
Refine msgstream close (#20194)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-31 15:13:33 +08:00