milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
17,475 Commits
130 Branches
115 Tags
556 MiB
Commit Graph

740 Commits (a577cac46b86e338382ac7191188342ea6db7440)

Author SHA1 Message Date
congqixia 68a6587374
Set insert&stats binlog timestamp range (#19005) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-04 09:05:09 +08:00
congqixia ff5201cbe5
Improve log of datanode compactor (#18981) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-02 14:31:06 +08:00
congqixia 29c72cad15
Use segment statslog instead of separate statslog (#18775) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-02 10:34:59 +08:00
congqixia c91bb0b015
Unify rootPath in configs and ChunkManager (#18808) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-08-25 19:32:53 +08:00
wei liu 9338ad45c5
fix insert meets vChannels reuse pChannel (#18695) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2022-08-23 19:12:55 +08:00
Enwei Jiao d9c7519107
refactor compaction concurency logic (#18660) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-08-23 15:50:52 +08:00
XuanYang-cn 16dcd9718b
Fix flush panic after compaction (#18677) 
See also: #18565

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-08-17 17:08:49 +08:00
XuanYang-cn 4c86cc63ba
Fix DN flowgraph delete buffer logs (#18657) 
1. Remove logs about not existing segments.
2. Group logs by timestamp.
3. Log changed segments only.
4. Pair the segments reference lock and unlock log by taskID.

Resolves: #18655

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-08-15 16:28:48 +08:00
aoiasd 8a70f2badb
Add ShowConfigurations for all remaining session (#18594) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-08-12 13:20:39 +08:00
congqixia 28aadc988d
Use vchannel name in datanode subName (#18519) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-08-03 17:10:36 +08:00
jaime c9174d55ba
Refine merge operation during compacting phase (#18399) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-07-28 14:52:31 +08:00
SimFG 74fd28df9c
Refine some datanode code (#18351) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-07-21 15:52:27 +08:00
Ten Thousand Leaves 0ebe407221
Make DataNode not panic if stale segment is not found during flush (#18322) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-07-20 15:06:30 +08:00
congqixia be6ca6358d
Use serialized size before compression in log meta (#18336) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-19 21:30:30 +08:00
congqixia f0846fb79b
Handles DropPartitionMsg in datanode flowgraph (#18292) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-15 17:12:27 +08:00
Ten Thousand Leaves ecc250c20e
Improve logs to better cover segment flush lifecycle (#18157) 
/kind improvements

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-07-08 14:50:21 +08:00
XuanYang-cn 53734bef58
Add seek msgID in DataNode (#17881) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-07-08 10:18:28 +08:00
wayblink 43ae6791e6
Make VChannelInfo read compatible with 2.0.2 (#17995) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-07-07 21:58:22 +08:00
XuanYang-cn d9e8231728
Filter sealed segments (#18065) 
- Remove FilterThreshold in DataNode
- Alter filter logic in flowgraph DDNode

See also: #17943

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-07-07 18:26:20 +08:00
congqixia 8f1ba6a0bb
Concurrent load segment stats log when initialize dataSyncService (#18125) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-07 14:32:21 +08:00
Ten Thousand Leaves d4a1e94f32
Fix flush failure caused by empty stale segments (#18097) 
/bug

issue: #18096
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-07-06 18:10:21 +08:00
Xiaofan a9b1d71a8f
change the etcd compareAndSwap interface (#18068) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-07-06 13:54:21 +08:00
zhenshan.cao 9bd35dd84e
Refine logs to reduce print (#17759) (#18028) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-07-04 11:10:20 +08:00
bigsheeper cdbd75d4dd
Log delta channel timetick in DataNode (#17947) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-30 17:44:18 +08:00
zhenshan.cao 0baeb609dc
Fix bugs in monitoring metrics (#17894) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-06-30 17:26:19 +08:00
Ten Thousand Leaves f0b036a35a
Move bulk load segment lock happen early (#17612) 
issue: #17600
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-06-27 13:56:17 +08:00
SimFG 50d5d04552
Refine some codes about the datanode (#17720) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-06-23 09:58:14 +08:00
Xiaofan 6fdf88f452
Fix compaction selection policy (#17690) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-22 19:00:14 +08:00
XuanYang-cn 12158432bb
Clear segment cache when closing flowgraph (#17671) 
See also: #17537

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-22 10:16:14 +08:00
congqixia 12b3a29db8
Fix segments info in drop virtual channel request is not set (#17641) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-20 13:40:12 +08:00
wayblink 074ec3060a
Support return dropped segments info in GetSegmentInfo rpc (#17617) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-06-17 18:24:12 +08:00
Xiaofan 1f6fbf91b2
Fix pulsar unsubsribe issue (#17562) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-16 17:28:11 +08:00
wayblink eb5b0b7fc8
Move SegmentInfo out of VchannelInfo, leave Id only to decrease kv size. Get complete SegmentInfo through RPC (#17441) 
Resolves: #17233 #16047

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-06-16 12:00:10 +08:00
Ten Thousand Leaves 65ab847815
Add segment reference lock for bulk load (#17457) 
So we can support DataQueryable and DataIndexed states

issue: #16607
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-06-15 12:20:10 +08:00
groot ab9b186f15
Change bulkload bufsize for row-based parser (#17473) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-06-14 16:18:09 +08:00
wayblink 25883840b2
Remove delete log in datanode because it may influence datanode performance (#17415) 
Resolves #16984

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-06-08 19:10:07 +08:00
XuanYang-cn 4f66bcbb22
Fix DataNode processes event out of order (#17440) 
The probability is low so very unlikly to reproduce

See also: #15966

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-08 17:24:07 +08:00
groot f31be941bb
Fix bulkload row count bug for multi datanodes (#17384) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-06-07 16:56:07 +08:00
xige-16 b5c11a216d
Alter varChar type params's name to max_length (#17409) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-07 15:58:06 +08:00
Xiaofan af994b5e1c
Fix etcd watch error (#17364) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-06 16:26:06 +08:00
Ten Thousand Leaves f5e63177d2
Fix an issue with bulk load with multiple DataNodes (#17324) 
issue: #17300
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-06-02 18:54:04 +08:00
bigsheeper e0cbacba59
Check ignorable error to prevent unnecessary panic (#17317) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-02 10:12:03 +08:00
XuanYang-cn d920e5c915
Fix DataNode panic while allocating IDs (#17294) 
See also: #17270

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-05-31 18:02:03 +08:00
groot affcc2acad
Fix bulkload entity number bug (#17278) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-05-31 15:40:04 +08:00
congqixia 55f1e03ca7
Allow DataNode clean vchannel with meta error without panicking (#17247) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-27 16:20:00 +08:00
Jiquan Long c12ee2f5cb
Fix flow_graph_insert_buffer_node.go code style (#17216) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-05-25 17:03:59 +08:00
Ten Thousand Leaves 5e1e7a6896
Fix an issue that causes flush hanging (#17169) 
issue: #16628
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-05-25 14:34:00 +08:00
xige-16 a8829554eb
Merge multiple sets of expr's optypes in diff dirs (#17192) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-24 21:56:00 +08:00
bigsheeper a4ea2fb18a
Handle errors in DataNode and QueryNode flow graph (#17096) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-24 21:11:59 +08:00
groot 168935f4cf
Fix bulkload bug for string primary key (#16958) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-05-13 16:07:54 +08:00
Ten Thousand Leaves 31ddff2056
Some minor fixes and improvements (#16814) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-05-06 21:35:51 +08:00
Ten Thousand Leaves a8e1c8fa9d
Make duplicate flush recalls success and other logic update (#16755) 
Latest logic:
1) Duplicate flush calls on same segments will not result in errors (same as the original design)
2) `FlushSegments` now still flushes stale segments even if non-stale segments failed to get flushed

issue: #16749

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-05-06 17:49:51 +08:00
cai.zhang 2be46a01c0
Reduce the number of retries and add error log (#16754) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-05-05 09:31:51 +08:00
Xiaofan ed51ffd50f
Fix Compact Entity ttl param (#16665) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-04-28 11:51:47 +08:00
zhenshan.cao 0a953948af
Simplify monitoring metrics (#16687) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-04-27 23:03:47 +08:00
Xiaofan b657b10eb5
Remove useless log (#16624) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-04-27 10:01:47 +08:00
groot bc5e9ec929
Add list import tasks api (#16605) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-04-25 17:37:46 +08:00
Ten Thousand Leaves e66ac6a77c
Resolve an issue with bulk load where segments in flow graph replica are not updated (#16609) 
issue: #15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-25 11:07:47 +08:00
Xiaofan 89b4a34892
Fix Data race in NodeID (#16603) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-04-24 22:03:44 +08:00
groot 0cec1b9448
Fix bulkload bug (#16591) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-04-22 18:43:43 +08:00
groot a6a3b69d91
Report bulk load state (#16555) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-04-21 21:37:42 +08:00
XuanYang-cn 93777e0c9d
Handle flowgraph releasing properly (#16169) 
See also: #15846

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-04-20 16:45:41 +08:00
Ten Thousand Leaves 289e468a7a
Make bulk load fully work (#16512) 
issue: #15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-20 14:03:40 +08:00
zhenshan.cao eae0747f14
Fix bug: data race caused by read and write Params.DataNodeCfg.NodeID (#16478) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-04-13 20:09:34 +08:00
godchen 4781db8a2a
Add datanode import (#16414) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-04-12 22:19:34 +08:00
Ten Thousand Leaves 442c73a544
Support passing channel names to DataNode (#16424) 
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-12 13:25:34 +08:00
godchen bb7a0766fe
Add dependency factory (#16204) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-04-07 22:05:32 +08:00
Ten Thousand Leaves 768bd4035f
Drop log level in delete node showDelBuf as they confuse our users. (#16410) 
issue: 16349

/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-07 17:59:31 +08:00
xige-16 99984b88e1
Support delete varChar value (#16229) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-04-02 17:43:29 +08:00
congqixia 6111dcde52
Fix ticker leakage in datanode flowgraph (#16346) 
Using  in case ticker need to be closed may cause leakage
Use time.NewTicker with deferred Stop instead

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-04-01 19:35:29 +08:00
Ten Thousand Leaves 45be3deb3f
Implement RC, DC, DN calling path for import. (#16321) 
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-01 11:33:28 +08:00
Ten Thousand Leaves 0d54697d42
Implement ReportImport related logic. (#16312) 
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-03-31 13:51:28 +08:00
XuanYang-cn 23b3a9ed8a
Log state in datanode handleWatchInfo (#16231) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-03-29 00:15:28 +08:00
XuanYang-cn a77dd1034c
Adopt the extended state in DataCoord (#16200) 
ChannelWatchInfo with ToWatch, ToRelease will trigger a timer.
ChannelManager now reacts to different ChannelWatch states.

- WatchSuccess > log this info

- WatchFailure/WatchTimeout > ToRelease

- ReleaseSuccess > Delete, reassign if not from DropCollection

- ReleaseFailure/ReleaseTimeout > Cleanup subscription and Delete,
  reassgin if not from DropCollection.

Some Notes:
1. Reassignment will add this channel to buffer if there's only one node.

See also: #15846

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-03-28 22:33:27 +08:00
xige-16 205c92e54b
Support insert string data (#15993) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-03-25 14:27:25 +08:00
xiyichan ca129d4308
Support configurable msgstream (#16131) 
Signed-off-by: xiyichan <2863768433@qq.com>
 2022-03-24 10:15:25 +08:00
groot 2078b24dca
Prepare for import (#16083) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-03-21 15:47:23 +08:00
godchen 78557ca6d7
Replace minio kv with minio chunk manager (#15936) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-03-17 18:03:23 +08:00
Ji Bin 3cd28420f1
Support compile under windows (#15786) 
This patch makes compile milvus under windows(MSYS), including:
- some cpp adaptation for compile under msys/gcc-10.3
- install toolchain scripts for setup from MinGW/MSYS `scripts/install_deps_msys.sh`
- adaptation for POSIX API use in golang
  * using gofrs/flock instead of syscall.Flock
  * using x/exp/mmap instead of syscall.Mmap
- introducing github actions for build milvus.exe under windows/MSYS
- rocksdb's patch for MSYS
- adaptation for compile knowhere under windows
- a windows package script for pack zip file, `scripts/package_windows.sh`

issue #7706

Signed-off-by: Ji Bin <matrixji@live.com>
 2022-03-17 17:17:22 +08:00
zhenshan.cao f3eeecf146
Reduce the number of metrics (#16030) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-03-15 21:51:21 +08:00
jaime 97b1ed7bca
Remove reader API from Mqstream and replace it with consumer API (#15971) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-15 14:45:22 +08:00
congqixia fab0263616
Use RmqFactory in DataNode unit test (#16039) 
Use Rocksmq instead of Pulsar in DataNode unittest
- Avoid pulsar seek problem caused timeout issue
- Remove third party component dependency in unit test

See also: #15986

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-03-14 19:22:01 +08:00
jaime 29975a7a26
Remove Consume() API from mqstream interface (#15886) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-11 20:09:59 +08:00
groot 89356ebcb9
Modify import rpc interfaces (#16006) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-03-11 17:13:59 +08:00
groot 5fdef607d5
Add rpc interfaces for import (#15930) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-03-09 18:33:59 +08:00
congqixia 6513be5c44
Extend watch/release channel logic in DataNode (#15925) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-03-09 15:39:58 +08:00
Letian Jiang 74f66dce3b
Fix passing empty binlog content in compaction (#15909) (#15927) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-03-08 16:34:01 +08:00
Jiquan Long f71651e294
Support column-based insert data in message stream (#15802) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-04 15:09:56 +08:00
Cai Yudong 3e862ecdfb
Merge msgChannelConfig and knowhereConfig into commonConfig (#15843) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-03-04 11:17:56 +08:00
jaime 7a3a721380
Reconstruct mqstream module (#15784) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-03 21:57:56 +08:00
Letian Jiang 2be217bcf4
Specify appropriate timestamp in initPKBloomFilter (#15823) (#15826) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-03-03 17:33:56 +08:00
cai.zhang 88527fc0a9
Fix metrics label conflict (#15839) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-03-03 16:05:57 +08:00
XuanYang-cn 63eec7ffc8
Refactor logs in binlog_io to reduce output (#15827) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-03-02 16:49:55 +08:00
Xiaofan bd7d294c6d
Refine log level in data (#15821) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-03-02 15:35:55 +08:00
bigsheeper 13177a90af
Add prometheus metrics for DataNode (#15650) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
 2022-02-28 19:11:55 +08:00
yah01 0cfb6a85be
Make compaction not generate empty segment (#15707) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2022-02-28 10:17:54 +08:00
XuanYang-cn e8edaa02fa
Refine code of DataNode binlogIO (#15759) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-02-28 10:09:54 +08:00
XuanYang-cn 6f85b49a4f
Add more compaction test cases (#15715) 
Test merge compaction with 2 segments with the same PK and only 1
valid deletion.

This test can varify the compaction behavior of DataNode for the
dup PK in different segment cases.

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-02-24 11:27:53 +08:00
XuanYang-cn d011073269
Fix DataNode panic when compact empty segment (#15580) 
remove the TODO in compactor

See also: #15573

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-02-18 18:47:51 +08:00
XuanYang-cn 23dfbf1f34
Fix eventmanager retry interval (#15631) 
See also: #15608

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-02-18 17:31:49 +08:00
congqixia 5751759c85
Implement mix compaction logic (#15542) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-02-18 14:51:49 +08:00
Xiaofan be42627ab2
Move Logutil to Util dir (#15447) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-02-09 18:55:46 +08:00
Cai Yudong 8255b713ac
Move ProxySubName/QueryNodeSubName/DataNodeSubName to msgChannelConfig (#15501) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-09 14:41:45 +08:00
congqixia 2263425cde
Fix EntriesNum is not passed in flush task (#15482) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-02-09 10:05:46 +08:00
Cai Yudong 1f337891e1
Rename base_param to service_param, rename global_param to component_param (#15463) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-08 20:57:47 +08:00
XuanYang-cn 43b6c295a4
Put logs of compaction elapse in defer (#15485) 
Before this PR, time elapse are logged after `compact()` finished
normally. No elapse will be logged with any errors.

This PR put elapse logs in defer once time-consuming steps are finished.

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-02-08 19:15:45 +08:00
XuanYang-cn 46dea0f006
Add elapse logs in compaction (#15414) 
See also: #15413

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-02-08 14:11:45 +08:00
JunSung Park f6873d3dc1
Implement automatic item expiration on compaction (#14418) 
* Implement automatic item expiration on compaction

Signed-off-by: JunSung Park <writtic@gmail.com>

* Change entityExpiration value comparison logic

Signed-off-by: JunSung Park <writtic@gmail.com>

* Add logging for the number of expired entity

Signed-off-by: JunSung Park <writtic@gmail.com>

* Set default value of CompactionEntityExpiration to off

Signed-off-by: JunSung Park <writtic@gmail.com>

* Fix errors for code checker

Signed-off-by: JunSung Park <writtic@gmail.com>

* Set default entityExpiration to max value

Signed-off-by: JunSung Park <writtic@gmail.com>
 2022-02-07 22:45:46 +08:00
XuanYang-cn 5eef9886b3
Add more specific log when exiting BackGroundGC goroutine (#15273) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-02-07 16:55:44 +08:00
Cai Yudong f1ea9613ae
Add EtcdConfig and move PulsarConfig/RocksdbConfig/MinioConfig into BaseParamTable (#15434) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-07 10:09:45 +08:00
Cai Yudong 4bccc85534
Use MsgChannelConfig in GlobalParams for all components (#15163) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-02 00:35:43 +08:00
congqixia 273890a91e
Fix datanode flush manager injection unit test (#15358) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-01-27 10:23:40 +08:00
XuanYang-cn 8ab9d769de
Fix logs of compaction_executor (#15365) 
- Remove not useful log in `stopExecutingtaskByVChannelName`
- Add planID info in `mergeFlushedSegments`

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-25 13:29:40 +08:00
congqixia 1e61112b5a
Add retry logic in pulsar consumer unsubscribe (#15284) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-01-20 01:23:37 +08:00
XuanYang-cn 87394e29b6
Fix compaction ut datarace (#15267) 
Fixes: #15241

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-18 17:49:39 +08:00
bigsheeper 7f775096c5
Unsubscribe when close pulsar consumer (#15221) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-17 18:53:36 +08:00
yah01 fd8d452cfd
remove unused fieldIDs and fieldTypes (#15229) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2022-01-16 17:37:35 +08:00
congqixia 2528b6850d
Refactor flush manager injection to reduce goroutine number (#15180) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-01-13 18:49:34 +08:00
XuanYang-cn 008e08a996
Add flowgraph manager and event manager (#15102) 
1. Add flowgraph manager to manager start and release of dataSyncService
2. Add event manager to manager etcd watchInfo event
  - Make put event able to retry
  - Make delete event able to terminate the retry loop

See also: #14604, #14300

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-13 14:21:34 +08:00
XuanYang-cn 6e847ff194
Make dn handleChannelInfo log more consist (#14936) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-11 09:37:35 +08:00
Cai Yudong b6a48817b0
Add MinioConfig in GlobalParams for all components (#15099) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-10 17:29:34 +08:00
XuanYang-cn 45f1f97ac4
Fix deltalog savebinlogpath save log (#15092) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-10 15:45:34 +08:00
Cai Yudong 7efb02a4c5
Use PulsarConfig in GlobalParams for all components (#15046) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-09 23:45:33 +08:00
Cai Yudong f8f61fb116
Use BaseParams etcd configuration for all components (#15012) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-07 16:15:26 +08:00
Xieql 8b06aac7a9
[skip e2e] Improve annotation (#14976) 
Signed-off-by: Xieql <qianglong.xie@zilliz.com>
 2022-01-07 13:57:49 +08:00
zhenshan.cao a5f8bdeb42
Fix bug: data race in MsgFactory (#14839) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-01-06 14:57:20 +08:00
Xieql 25b00230c8
[skip e2e] Improve annotation (#14908) 
Signed-off-by: Xieql <qianglong.xie@zilliz.com>
 2022-01-06 14:15:20 +08:00
Xieql c488c41dfb
[skip e2e] Improve annotation (#14912) 
Signed-off-by: Xieql <qianglong.xie@zilliz.com>
 2022-01-06 13:38:08 +08:00
congqixia ecdc3f7bf7
[skip e2e] Improve comment for DataNode delete node (#14825) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-01-05 14:29:35 +08:00
godchen 98232b4942
[skip e2e]Sort imports (#14820) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-01-05 14:25:48 +08:00
godchen e1eb721532
[skip e2e]Sort imports (#14819) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-01-05 14:23:52 +08:00
godchen 7031a511ce
[skip e2e]Sort imports (#14806) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-01-05 14:00:38 +08:00
zhenshan.cao b10c361822
Fix bug: should use Lock instead of RLock (#14797) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-01-04 23:33:24 +08:00
zhenshan.cao 76a9687310
Adjust usage of lock in datanode (#14631) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Co-authored-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-04 19:35:32 +08:00
zhenshan.cao 7abebe576c
Add more logs on write path (#14600) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-01-04 15:47:18 +08:00
XuanYang-cn f0a40f1569
Make FieldBinlog in a segment unique for a field (#14632) 
See also: #14620

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-30 20:29:22 +08:00
Jiquan Long ebb9b24b47
Improve name of flowgraph node (#14538) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-12-30 10:33:46 +08:00
XuanYang-cn 9d0bc136e0
Add vChannelName in IBN timetick log (#14527) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-30 10:09:48 +08:00
Xiaofan 69087ff8dd
Support embedded etcd (#14333) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-12-29 14:35:21 +08:00
congqixia a8c7e0c827
[skip e2e] Add comment for Compaction of DataNode (#14350) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-28 09:38:07 +08:00
XuanYang-cn 3cfa88d6f2
Fix delete node not aware of merged segments error (#14190) 
This PR made merge gracefully remove merged segment by moving
them into `compactedSegments`

And everytime when delete_node operates, if compactedSegments
have contents, delete_node will replace the latest segment in delBuf
merged from the merged segments, and truely remove the compactedSegments

Fixes: #14085

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-27 11:02:18 +08:00
zhenshan.cao 9a2c91d326
Fix bug: refresh config after session init (#14194) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-12-24 22:22:17 +08:00
XuanYang-cn 3d24007efa
Add more logs for load sealed segment (#14079) 
Add logs before&after flowgraph_ddnode forwarding delete msg

See also: #13879

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-24 10:50:07 +08:00
Cai Yudong 891c202b73
Use GlobalParamTable for all components (#13885) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-12-23 18:39:11 +08:00
Jiquan Long 0871bd3d63
Add log when update pk range (#14006) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-12-23 11:57:08 +08:00
Bingyi Sun 6b91e5e9d1
Add warning log to trace old messages (#13939) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-12-22 16:59:05 +08:00
congqixia 2b352bcde5
Fix compact result segment log size always zero (#13953) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-22 14:27:06 +08:00
Xiangyu Wang 2a6168e937
Improve import statement (#13734) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-12-21 19:15:32 +08:00
JackLCL 87071f3f41
[skip e2e]Fix error log output format in segment_replica.go (#13806) 
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
 2021-12-20 22:51:25 +08:00
Xiangyu Wang 5d4c4f38c8
Improve import statement (#13735) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-12-20 15:49:06 +08:00
Xiangyu Wang cabc40cb09
Improve import statement (#13737) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-12-20 14:39:12 +08:00
Xiangyu Wang a7dfd53f53
Improve import statement (#13736) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-12-20 14:38:25 +08:00
Xiangyu Wang 08be6888d5
Improve import statement (#13733) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-12-20 14:33:47 +08:00
congqixia 2de5cf7762
[skip e2e] Improve log in flush manager (#13689) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-20 12:22:43 +08:00
congqixia d4afc812d1
Normalize DataNode term in errors.go (#13688) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-20 12:20:50 +08:00
congqixia 1dcd06cc78
Add log file info for all kind of segment binlogs (#13606) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-19 20:00:42 +08:00
JackLCL 20c956b675
[skip e2e]Fix error log output format in segment_replica.go (#13676) 
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
 2021-12-19 12:32:42 +08:00
godchen 154f11e3a1
[skip e2e]Fix redundant conversion (#13648) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2021-12-17 22:28:50 +08:00
godchen e191b23249
[skip e2e]Fix error string (#13647) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2021-12-17 19:22:42 +08:00
godchen fefeaa2e1d
[skip e2e]Fix compaction executor typo (#13646) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2021-12-17 19:20:46 +08:00
cai.zhang 9f23fc7f2a
Register the service when the component state is healthy (#13248) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2021-12-15 11:47:10 +08:00
Bingyi Sun 0c39e0370c
Move segment statistics update to datanode tt loop (#13233) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-12-15 10:53:16 +08:00
JackLCL 30afcf2311
[skip ci]Fix error log output format in segment_replica.go (#13350) 
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
 2021-12-14 16:51:09 +08:00
JackLCL 5f02a3738a
[skip ci]Fix error log output format in meta_service.go (#13349) 
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
 2021-12-14 16:43:25 +08:00
Xiangyu Wang 405b3cd932
Use the same lint rules with golint in revive (#13324) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-12-14 15:31:07 +08:00
congqixia 8f87cc7ef1
Fix JoinIDPath logic problem (#13039) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-09 12:05:04 +08:00
XuanYang-cn dbf757e08b
Change genKey into JoinIDPath (#12959) 
1. Simplify genKey(), make genKey a rpc call
2. Unify the usage of JoinIDPath

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-09 11:09:06 +08:00
XuanYang-cn a9a332dbcf
Fix empty segment compact and load (#12710) 
1. Fix compaction save empty segment bug
2. Fix load empty segment bug
3. Add UT for dmlchannels to 100%

Resolves: #12450

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-09 11:03:08 +08:00
congqixia 1195f441b2
Update DataNode state before stop (#12946) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-08 14:19:04 +08:00
JackLCL 73b63b9a02
[skip ci]Fix error log output format in segment_replica.go (#12929) 
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
 2021-12-08 13:55:04 +08:00
JackLCL a6ce7d7f53
[skip ci]Fix error log output format in meta_service.go (#12928) 
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
 2021-12-08 13:51:09 +08:00
JackLCL e51ef53632
[skip ci]Fix error log output format in flush_manager.go (#12857) 
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
 2021-12-07 14:38:15 +08:00
congqixia b20a238c38
Fix newDataSyncService panicking with normal segment has no dml pos (#12771) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-06 16:21:35 +08:00
congqixia 58c5b6093d
Fix close10000 ut has chances to fail (#12740) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-06 09:59:35 +08:00
Bingyi Sun 8bbaae04ff
Decrease compaction parallel (#12707) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-12-03 18:57:33 +08:00
congqixia ffd05eb140
Fix deadlock in mergedTimeTickSender (#12697) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-03 16:39:33 +08:00
edward.zeng 45cbec1f7b
[skip ci] Refine datanode binlog io comment (#12638) 
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
 2021-12-02 19:11:32 +08:00
edward.zeng c2b73e1b55
[skip ci] Refine data node flow graph comment (#12637) 
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
 2021-12-02 18:59:33 +08:00
congqixia 8f9e62fa18
Apply DropVirtualChannel and FlushManager drop mode (#12563) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-02 16:39:33 +08:00
godchen 5e3e3b3b41
Remove grpc returned error (#12537) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-12-01 22:17:46 +08:00
edward.zeng 271aaa7537
[skip ci] Refine data node param table comment (#12560) 
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
 2021-12-01 19:31:36 +08:00
congqixia e813867652
Implement Drop mode for flush manager (#12469) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-01 10:11:39 +08:00
godchen f31ed089b5
Add timeout for reader next (#12308) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-26 22:45:24 +08:00
XuanYang-cn 48b45d82e5
Add ut for binlog_io to 100 coverage (#12283) 
Make DN ut coverage upto 90%
Resolves: #8058

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-26 17:43:17 +08:00
dragondriver b6d0150dc5
Fix health check when disconnect to etcd (#12292) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-11-26 11:39:16 +08:00
bigsheeper e245ba87a9
Add excluded segments for dropped segemnts (#12247) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-11-25 14:25:15 +08:00
XuanYang-cn 8a8ebed23d
Fix DN GC bug (#12258) 
Fixes: #12249

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-25 09:43:15 +08:00
congqixia 59d19384dc
Fix flush manager injection logic with multiple segments (#12260) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-24 18:41:16 +08:00
godchen e3f49858c5
Add subscription prefix (#12251) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-24 17:47:15 +08:00
dragondriver 6c4c0ef6b5
Add more deploy metrics (#12248) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-11-24 15:43:15 +08:00
XuanYang-cn e72c5b9351
Filter dropped segments in DN (#12242) 
See also: #12131, #12230

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-24 14:11:15 +08:00
XuanYang-cn 6bf0c4326d
Fix compaction generates wrong insert binlogs (#12204) 
This PR
- changes Get() to GetRow() in BinlogIterator
- removes Get() and Length() of FieldData
- adds ut in BinlogIterator for floatvector and binaryvector
- fix a log bug

Fixes: #12146

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-23 17:09:15 +08:00
congqixia 4121e31df1
Send SIGINT to runner goroutine after etcd disconnects (#12163) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-22 16:23:17 +08:00
godchen 51b353b52c
Add msgstream reader (#12130) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-19 15:57:12 +08:00
congqixia 7e91bcf115
Normalize datanode dminput node log (#12128) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-19 14:21:13 +08:00
dragondriver ee0f753f7a
Fix datarace between GetComponentStates and Register (#11935) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-11-19 13:57:12 +08:00
Bingyi Sun 038f42851e
Add more log for debug (#11893) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-18 10:37:41 +08:00
XuanYang-cn 7dbb4d10e5
Fix DeltaLogs empty yet appending a deltaInfo bug (#11984) 
- Add more logs for startpositions

Fixes: #11970

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-17 17:57:11 +08:00
congqixia 5e550a8793
Add close for flushManager (#11874) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-17 10:39:15 +08:00
XuanYang-cn 9bd3664672
Fix compaction bug (#11816) 
This PR fixes:
- timeout in compaction
- BinlogIterator using rawID as PKField bug

See also: #11757

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-17 10:07:13 +08:00
congqixia 5edbb82610
Add session revoke (#11908) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-16 22:31:14 +08:00
Xiangyu Wang 12f50cb22c
[skip ci]Update OWNERS files (#11898) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-11-16 15:41:11 +08:00
congqixia fa6314393a
Fix compactor loop closure captured (#11814) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-15 19:44:10 +08:00
congqixia 579416e3d3
Apply retry for flush manager execution (#11808) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-15 17:19:10 +08:00
XuanYang-cn f3852c1db0
Add more specific logs (#11724) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-15 15:25:09 +08:00
Bingyi Sun 5c8978e5d6
Fix drop segment bug (#11709) 
issue: #11708
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-12 17:31:11 +08:00
XuanYang-cn c4bb73dc74
Complete drop collection procedure in DN (#11552) 
drop, cancel compaction, flush all buffers and release fg

See also: #11426

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-11 20:56:49 +08:00
godchen 5357e301f4
Fix retrieve search error (#11501) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-11 00:54:45 +08:00
godchen 14d2891a69
[skip ci]Fix download typo (#11577) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-10 20:11:35 +08:00
godchen 0fc1e12faf
[skip ci]Fix download typo (#11575) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-10 20:09:33 +08:00
godchen 038d0f1adf
[skip ci]Fix download typo (#11573) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-10 20:07:39 +08:00
congqixia 4c37e94a39
Add type assertion for compactTask (#11468) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-10 19:16:40 +08:00
Xiaofan 5f08b67156
Move node.yaml to component.yaml (#11363) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-11-10 14:27:37 +08:00
godchen 863f1bb34e
Fix multi delete data not effect (#11422) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-09 15:01:17 +08:00
Xiangyu Wang 0eb71f7ee5
Logging traceID in process of processing delete request (#11374) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-11-09 11:01:02 +08:00
godchen 639800241d
Make flow graph for delta channel (#11379) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-09 09:27:04 +08:00
XuanYang-cn 3cd0584a04
Add compaction in datanode (#11353) 
See also: #9530

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-08 19:49:07 +08:00
congqixia 3ac7d7a9de
Add detailed comments for dataSyncService in DataNode (#11285) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-05 22:59:04 +08:00
sunby c26cd79532
Add compaction (#9906) 
issue: #9904
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-11-05 22:25:00 +08:00
congqixia 0e5931a3da
Add mergedTimetickSender for datanode (#11297) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-05 14:59:32 +08:00
godchen 9186d5527e
Forward delete msg (#11210) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-05 11:59:02 +08:00
XuanYang-cn 8863c01ff7
Make DN fail faster when watching an invalid chan (#11102) 
See also: #11098

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-04 15:56:28 +08:00
congqixia 0ef95c5df1
Add timeticker logger for insert buffer node (#11216) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-04 15:40:14 +08:00
godchen b63a9521c2
Make gen msg private (#10115) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-04 15:36:19 +08:00
godchen a0a3a889e7
Add common endian for global use (#11092) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-02 18:16:32 +08:00
XuanYang-cn 225b2c66a0
Fix datanode ut using register bug (#10885) 
Resolves: #10881
See also: #8058

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-01 11:01:49 +08:00
XuanYang-cn 1e3fc5076f
Change etcdkv clientv3 into MetaKv interface (#10903) 
This pr:
- changed etcdkv clientv3 into MetaKv interface
- replaced fmt.Sprintf with path.Join for kv key

See also: #8058

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-01 10:19:55 +08:00
Xiaofan 2df2d488e1
Fix Unit test fatal fail (#10933) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-10-30 10:24:38 +08:00
congqixia 756be81bce
Add injectFlush method for flushManager interface (#10659) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-28 19:16:31 +08:00
Xiaofan 6fddcb3925
Fix Session checker for proxy (#10737) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-10-27 21:58:33 +08:00
XuanYang-cn 42ce9b18d1
Add methods in segment replica (#10582) 
see also: #8058

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-27 17:14:28 +08:00
congqixia 169af67eb1
Fix segments not removed from flushingCache (#10738) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-27 16:06:24 +08:00
cai.zhang aff5cd9598
Supprot to read the configuration of minio from environment variables (#10534) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-10-26 15:10:03 +08:00
godchen 095a2378ca
Add datanode ut (#10433) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-25 22:52:27 +08:00
congqixia ae2a301662
Add injection logic for FlushManager (#10580) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-25 20:17:34 +08:00
XuanYang-cn 21d98c2825
Add binlog io for compaction (#10536) 
See also: #9530

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-25 20:13:51 +08:00
godchen d2e0943f21
Fix delete node flush error (#10549) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-25 18:49:20 +08:00
congqixia 2713988ff2
Merge auto manual flush with same segment id (#10550) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-25 18:03:42 +08:00
congqixia bd90a3831c
Allow flushed segments trigger flushe process (#10539) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-25 18:01:50 +08:00
XuanYang-cn cd06f50645
Remove schema in delete codec (#10517) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-24 09:59:10 +08:00
godchen 31393d8801
[skip ci]Fix datanode comment typo error (#10502) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-23 16:57:35 +08:00
godchen ffc0c07610
Change delete data primary key to int64 (#10438) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-22 15:37:12 +08:00
godchen d6bf0c928f
Get stats delta path from datacoord (#10362) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-22 14:31:13 +08:00
congqixia 7ce7cb7a5e
Add a timed flush trigger mechanism (#10197) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-20 15:02:36 +08:00
godchen 6279fb817a
Fix stats log load with prefix error (#10216) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-19 20:50:35 +08:00
godchen 8daeb0d519
Change bloom filter use pk (#10193) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-19 20:18:47 +08:00
sunby 4031b8746f
Fix panic when flushing segment (#10182) 
issue: #10180
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-19 19:06:37 +08:00
congqixia ed4e516ce7
Save stats and delta logs from DataNode (#10159) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-19 14:34:35 +08:00
congqixia 5737e0075f
Apply flush manager logic in datanode (#10142) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-19 11:04:34 +08:00
congqixia 3711fcd82f
Add more info for Delta log compaction (#10083) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-18 18:24:45 +08:00