b90fa5f459
Refine codes of datanode buffer ( #23168 )
...
This PR refines deltabuffer of datanode:
- Add last sync time for compacted segment, see also: #23210
- Ensure all deltabuffermanager handles all delete related operations
- Change usedMemory to atomic.Int64
- Remove allocator in delete buffer
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-04-10 18:42:30 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
be0827417a
Fix buffer calculate memorysize negative ( #23152 )
...
Along with the following small patches:
- And fix logging dup segmentIDs for delete buffer status
- Add String method for PriorityQueue to debug
See also: #23105
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-03-31 18:30:23 +08:00
0f2c3ac832
Optimize log ( #23121 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-03-30 10:44:22 +08:00
93bc805933
Enhance ID allocator in DataNode ( #22905 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-03-23 19:43:57 +08:00
d126f06946
Decouple mq module from internal proto definition ( #22536 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-03-04 23:21:50 +08:00
fb42466c65
Use opentelemetry ( #21509 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-01-12 16:09:39 +08:00
4b4944ecee
Support sync all segments while close ( #21421 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-06 14:49:36 +08:00
efefba4da4
Fix checkpoint will not update after delete and compact ( #21495 ) ( #21499 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-05 11:31:35 +08:00
f595500383
Move sync delete policy from deleteNode to insertBufferNode ( #21152 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-12-13 16:15:21 +08:00
83244afcf3
Reduce some logs level which is frequent ( #20894 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-11-30 14:07:15 +08:00
c848896bc8
Fix panic error due to rollDeleteOp ahead by load-and-delete ( #20563 )
...
issue:#20501
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
2022-11-15 14:45:07 +08:00
cd19d99ad7
Add channel level checkpoint ( #20350 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-11-10 22:13:04 +08:00
21b54709a2
enable delbuf auto flush function to avoid OOM when del msg accumulating ( #20213 )
...
issue: #19713
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
Co-authored-by: MrPresent-Han <zilliz@zillizdeMacBook-Pro-2.local>
2022-11-07 18:49:02 +08:00
956c5e1b9d
Make Params singleton ( #20088 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-04 14:25:38 +08:00
2bfecf5b4e
Refine bloomfilter and memory usage ( #20168 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-31 17:41:34 +08:00
d5bc4e2585
Make DN able to process syncing request in batch ( #20126 )
...
See also: #19834
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-10-27 21:25:33 +08:00
192151bc11
Refine Data Log ( #19996 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-25 19:31:30 +08:00
e081eb287d
Refine flowgraph nodes for readability ( #19864 )
...
- delete compaction tasks from executor while dropping
See also: #19780
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-10-20 16:39:29 +08:00
0095869d5b
Keep segment statslog if no insert applied ( #19910 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-10-20 14:33:27 +08:00
2785f98de6
Refactor segment replica in DN ( #19815 )
...
- Replace isNew and isFlushed with sType
- Add compacted into proto
- Rename replica to channel
- Rename segmentReplica to channelMeta
- Replace 4 maps with 1 map
- Add segment.go and segment_test.go
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-10-18 15:33:26 +08:00
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
db3f4be49d
Simplify flow graph node to pipeline mode ( #19667 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-10 22:15:22 +08:00
432b5ddf2e
Add SyncSegments into proto ( #19312 )
...
See also: #19072
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-09-23 10:22:52 +08:00
d7f38a803d
Separate some proto files ( #19218 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
91663a6603
Fix bloom filter size not match ( #19140 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-13 14:12:31 +08:00
16dcd9718b
Fix flush panic after compaction ( #18677 )
...
See also: #18565
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-08-17 17:08:49 +08:00
4c86cc63ba
Fix DN flowgraph delete buffer logs ( #18657 )
...
1. Remove logs about not existing segments.
2. Group logs by timestamp.
3. Log changed segments only.
4. Pair the segments reference lock and unlock log by taskID.
Resolves : #18655
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-08-15 16:28:48 +08:00
0baeb609dc
Fix bugs in monitoring metrics ( #17894 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-06-30 17:26:19 +08:00
6fdf88f452
Fix compaction selection policy ( #17690 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-06-22 19:00:14 +08:00
25883840b2
Remove delete log in datanode because it may influence datanode performance ( #17415 )
...
Resolves #16984
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-06-08 19:10:07 +08:00
a4ea2fb18a
Handle errors in DataNode and QueryNode flow graph ( #17096 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-05-24 21:11:59 +08:00
89b4a34892
Fix Data race in NodeID ( #16603 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-24 22:03:44 +08:00
768bd4035f
Drop log level in delete node showDelBuf as they confuse our users. ( #16410 )
...
issue: 16349
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-07 17:59:31 +08:00
99984b88e1
Support delete varChar value ( #16229 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-04-02 17:43:29 +08:00
7a3a721380
Reconstruct mqstream module ( #15784 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-03-03 21:57:56 +08:00
88527fc0a9
Fix metrics label conflict ( #15839 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-03-03 16:05:57 +08:00
13177a90af
Add prometheus metrics for DataNode ( #15650 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
2022-02-28 19:11:55 +08:00
ecdc3f7bf7
[skip e2e] Improve comment for DataNode delete node ( #14825 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-01-05 14:29:35 +08:00
ebb9b24b47
Improve name of flowgraph node ( #14538 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-12-30 10:33:46 +08:00
3cfa88d6f2
Fix delete node not aware of merged segments error ( #14190 )
...
This PR made merge gracefully remove merged segment by moving
them into `compactedSegments`
And everytime when delete_node operates, if compactedSegments
have contents, delete_node will replace the latest segment in delBuf
merged from the merged segments, and truely remove the compactedSegments
Fixes : #14085
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-12-27 11:02:18 +08:00
3d24007efa
Add more logs for load sealed segment ( #14079 )
...
Add logs before&after flowgraph_ddnode forwarding delete msg
See also: #13879
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-12-24 10:50:07 +08:00
1dcd06cc78
Add log file info for all kind of segment binlogs ( #13606 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-19 20:00:42 +08:00
8f9e62fa18
Apply DropVirtualChannel and FlushManager drop mode ( #12563 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-02 16:39:33 +08:00
8a8ebed23d
Fix DN GC bug ( #12258 )
...
Fixes : #12249
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-25 09:43:15 +08:00
c4bb73dc74
Complete drop collection procedure in DN ( #11552 )
...
drop, cancel compaction, flush all buffers and release fg
See also: #11426
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-11 20:56:49 +08:00
863f1bb34e
Fix multi delete data not effect ( #11422 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-11-09 15:01:17 +08:00
0eb71f7ee5
Logging traceID in process of processing delete request ( #11374 )
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-11-09 11:01:02 +08:00
a0a3a889e7
Add common endian for global use ( #11092 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-11-02 18:16:32 +08:00
225b2c66a0
Fix datanode ut using register bug ( #10885 )
...
Resolves : #10881
See also: #8058
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-01 11:01:49 +08:00
XuanYang-cn
jaime
Jiquan Long
Enwei Jiao
Xiaofan
cai.zhang
bigsheeper
MrPresent-Han
congqixia
SimFG
zhenshan.cao
wayblink
Ten Thousand Leaves
xige-16
godchen
Xiangyu Wang