milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
21,884 Commits
174 Branches
147 Tags
569 MiB
Commit Graph

309 Commits (eb046863485fdf3e130fc60484485c901b81276b)

Author SHA1 Message Date
yiwangdr c7b851f870
add interface for non-watch metakv (#25092) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-06-26 09:20:44 +08:00
jaime 18df2ba6fd
[Cherry-Pick] Support Database (#24769) 
Support Database(#23742)
Fix db nonexists error for FlushAll (#24222)
Fix check collection limits fails (#24235)
backward compatibility with empty DB name (#24317)
Fix GetFlushAllState with DB (#24347)
Remove db from global meta cache after drop database (#24474)
Fix db name is empty for describe collection response (#24603)
Add RBAC for Database API (#24653)
Fix miss load the same name collection during recover stage (#24941)

RBAC supports Database validation (#23609)
Fix to list grant with db return empty (#23922)
Optimize PrivilegeAll permission check (#23972)
Add the default db value for the rbac request (#24307)

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-06-25 17:20:43 +08:00
Xiaofan 72c5e2a41a
Fix channel reassigned to other datanodes (#25015) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-21 21:26:42 +08:00
wayblink bfae6b49af
Remove datanode timetick mq, use rpc to report instead (#23156) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-06-14 14:16:38 +08:00
congqixia 41af0a98fa
Use go-api/v2 for milvus-proto (#24770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
yiwangdr 37c02c9927
clean up meta interface (#24729) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-06-08 14:18:35 +08:00
smellthemoon 146050db82
Fix some wrong ut (#23990) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-05-10 09:31:19 +08:00
MrPresent-Han b517bc9e6a
refine balance mechanism including:(#23454) (#23763) (#23791) 
1. balance granuity to replica to avoid influence unrelated replicas
2. avoid balance back and forth

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2023-05-04 12:22:40 +08:00
Xiaofan 016311ad48
Fix Has Collection Failed on datacoord (#23710) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-04-27 09:50:34 +08:00
zhenshan.cao 4a32b842e8
Improve the check logic of channel remove (#23473) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-04-18 02:58:30 +08:00
cai.zhang 43a9e175a3
Exit component process when session key is deleted (#21658) (#22164) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-04-12 20:12:28 +08:00
zhenshan.cao dc5abe086c
Revert "Restrain gc files by collection (#22504)" (#23380) 
This reverts commit fa86de530d.

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-04-12 19:38:28 +08:00
jaime c9d0c157ec
Move some modules from internal to public package (#22572) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-04-06 19:14:32 +08:00
yihao.dai 17d3190fcc
Fix DescribeCollection failed with undefined msg type (#23192) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-04-03 19:52:24 +08:00
smellthemoon ce91f61e5a
Pre-create topics (#21907) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-04-03 16:44:23 +08:00
congqixia 127867b873
Add ratedgroup for some info/warning log (#23095) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-31 15:22:23 +08:00
yah01 73f1f15c1c
Fix DataCoord's meta may be broken with DataNode of prior version (#23030) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-03-29 16:58:02 +08:00
Bingyi Sun d6bfe8fbb9
Change frequent logs to debug level (#23012) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2023-03-28 07:12:01 +08:00
MrPresent-Han 77c9e33e70
support dml channel balancer on datacoord (#22324) (#22377) (#22692) 
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2023-03-20 10:01:56 +08:00
SimFG b57e476089
Fix the nil point about the session (#22748) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-03-14 20:07:54 +08:00
wayblink 28390dfe09
Add bulkinsert integration test (#22621) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-03-10 11:03:52 +08:00
yah01 efdc77bce6
Fix not fill timestamp when use collection created ts (#22629) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-03-08 18:53:51 +08:00
jaime d126f06946
Decouple mq module from internal proto definition (#22536) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-03-04 23:21:50 +08:00
yah01 319f1773af
Fix segment not found if forward delete to empty segment (#22528) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-03-03 14:13:49 +08:00
congqixia fa86de530d
Restrain gc files by collection (#22504) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-02 11:49:47 +08:00
Xiaofan 3d6bded115
Fix Row Number Mismatch (#22307) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-02-27 10:41:46 +08:00
Enwei Jiao 697dedac7e
Use cockroachdb/errors to replace other error pkg (#22390) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-02-26 11:31:49 +08:00
cai.zhang 66b3566ac1
Update component state to healthy after start (#22118) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-02-10 17:40:32 +08:00
yah01 b1f31da77a
Fix activate standby server ignores all errors (#22073) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-02-09 15:24:31 +08:00
cai.zhang bcae97b125
Register after start to prevent there are tow coordinator at the same time (#21641) (#21707) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-30 11:11:50 +08:00
wayblink 6a722396bd
Integration test framework (#21283) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-01-12 19:49:40 +08:00
cai.zhang 5d660f16d6
Forbid IndexCoord start with version<2.3 (#21597) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-10 14:33:38 +08:00
jaime 73d8509600
Improve ListSegments performance for datacoord catalog (#21477) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-01-06 14:33:36 +08:00
cai.zhang e5f408dceb
Merge IndexCoord and DataCoord (#21267) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-04 19:37:36 +08:00
Jiquan Long ff2a68e65a
Fix collection not exist when tried to do recovery (#21471) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-01-04 16:37:35 +08:00
Enwei Jiao 89b810a4db
Refactor all params into ParamItem (#20987) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-07 18:01:19 +08:00
jaime 039e9ce4bb
Refine msgstream interface (#20832) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-12-05 20:55:17 +08:00
Bingyi Sun 2390095232
Fix load uses compacted segments' binlogs (#20655) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-11-17 20:37:10 +08:00
Enwei Jiao c05b9ad539
Add event dispatcher for config (#20393) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-17 18:59:09 +08:00
Enwei Jiao 19524a5344
Fix nodeID mismatch at standalone mode (#20648) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-17 17:15:08 +08:00
zhenshan.cao c5dc076b04
Fix start of DataCoord (#20581) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-11-15 14:19:06 +08:00
smellthemoon a7ba416b5a
Remove invalid timestamp (#20557) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-11-14 16:21:07 +08:00
bigsheeper cd19d99ad7
Add channel level checkpoint (#20350) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-11-10 22:13:04 +08:00
SimFG 2aba154ad3
Make the subName unique for the datanode tt channel (#20331) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-11-08 20:15:04 +08:00
Enwei Jiao 7e56e7a976
Fix datarace at Setlogger (#20376) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-08 14:33:03 +08:00
jaime 709a0a94e9
Add some msgstream metrics (#20296) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-11-07 10:15:02 +08:00
Enwei Jiao 956c5e1b9d
Make Params singleton (#20088) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-04 14:25:38 +08:00
Xiaofan 11efa0bb5f
Fix compacted segment not dropped (#20128) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-10-27 17:15:32 +08:00
Xiaofan def5972e01
fix duplicated handoff request (#20108) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-10-27 15:15:32 +08:00
Xiaofan 192151bc11
Refine Data Log (#19996) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-10-25 19:31:30 +08:00
Xiaofan 1f170ce1ba
Refine kafka consumer (#19846) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-10-25 13:23:30 +08:00
congqixia 2e6ddd7f2a
Add datanode&channel match check for Flush (#19985) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-10-22 12:09:28 +08:00
smellthemoon 581e1d8c35
Fill MsgBase ALL (#19912) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-10-21 15:57:28 +08:00
SimFG a55f739608
Separate public proto files (#19782) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-10-16 20:49:27 +08:00
yah01 89c9cb3680
Fix scan may break GC limitation (#19670) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-10-15 15:13:24 +08:00
Ten Thousand Leaves be4b0189b1
Fix potential long log (#19757) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-10-13 19:21:23 +08:00
congqixia 2a2d32c4fc
Fix datacoord panics when collection info not found (#19706) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-10-11 21:39:24 +08:00
jaime 31db01b6ae
Support TTL for collection level (#19383) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-10-10 20:31:22 +08:00
wayblink 85e04d845c
Move GetComponentStates to milvus.proto, make it an external interface (#19594) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-10-10 15:55:22 +08:00
wayblink 54e2c079c4
MEP-30:Support basic coordinators active-standby mechanism #16901 (#19166) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-09-29 18:35:02 +08:00
yah01 2dca3688a0
Fix data race in DataCoord unit test (#19469) 
Refine the DataCoord initialization

Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-09-27 18:04:53 +08:00
Ten Thousand Leaves b30c9d4f60
Support Bulk Load in Milvus 2.2 (#18982) 
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-09-26 18:06:54 +08:00
jaime 91df8f2d6a
Use log id instead of log path of the binlog in metastore (#19123) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-09-25 15:56:51 +08:00
Xiaofan 928a213e31
Fix Chunk Manager Path mis use (#19141) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-09-23 14:40:51 +08:00
SimFG d7f38a803d
Separate some proto files (#19218) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-09-16 16:56:49 +08:00
yah01 11b352c3ae
Guarantee flushed segments are all indexed (#19062) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-09-16 11:32:48 +08:00
congqixia 29c72cad15
Use segment statslog instead of separate statslog (#18775) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-02 10:34:59 +08:00
cai.zhang c924f73105
Refactor for IndexCoord to support cloud (#18643) 
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
 2022-08-25 15:48:54 +08:00
codeman 70254c3ec5
Unified catalog interface for segment (#18289) (#18290) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-08-20 10:24:51 +08:00
Ten Thousand Leaves fc42ee5bb3
Fix # of rows of recovering segment (#18736) 
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-08-19 19:50:50 +08:00
xing.zhao d90313a1a8
Use chunkManager rather than minio.Client in datanode garbage collection (#18092) 
Signed-off-by: xingzhao <xing.zhao@zilliz.com>

Co-authored-by: xingzhao <xing.zhao@zilliz.com>
 2022-07-22 22:10:28 +08:00
cai.zhang 0ad15a22c8
IndexCoord handle events correctly (#17878) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-07-07 14:44:21 +08:00
Ten Thousand Leaves d4a1e94f32
Fix flush failure caused by empty stale segments (#18097) 
/bug

issue: #18096
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-07-06 18:10:21 +08:00
zhenshan.cao 0baeb609dc
Fix bugs in monitoring metrics (#17894) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-06-30 17:26:19 +08:00
Ten Thousand Leaves f0b036a35a
Move bulk load segment lock happen early (#17612) 
issue: #17600
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-06-27 13:56:17 +08:00
Enwei Jiao 746aeea35b
fix typo (#17668) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-06-21 18:54:13 +08:00
cai.zhang ed94ecf847
Segments can't be compacted when they have reference lock (#17649) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-20 21:56:12 +08:00
Ten Thousand Leaves 65ab847815
Add segment reference lock for bulk load (#17457) 
So we can support DataQueryable and DataIndexed states

issue: #16607
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-06-15 12:20:10 +08:00
shaoyue 76eaa3fc50
Support access s3 through AWS IAM role (#17292) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-06-02 19:42:03 +08:00
cai.zhang bcf3b7426a
Add distributed lock for segment refgerence (#16782) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-05-31 16:36:03 +08:00
Ten Thousand Leaves 5e1e7a6896
Fix an issue that causes flush hanging (#17169) 
issue: #16628
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-05-25 14:34:00 +08:00
cai.zhang 2be46a01c0
Reduce the number of retries and add error log (#16754) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-05-05 09:31:51 +08:00
zhenshan.cao 0a953948af
Simplify monitoring metrics (#16687) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-04-27 23:03:47 +08:00
Ten Thousand Leaves e66ac6a77c
Resolve an issue with bulk load where segments in flow graph replica are not updated (#16609) 
issue: #15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-25 11:07:47 +08:00
Xiaofan 89b4a34892
Fix Data race in NodeID (#16603) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-04-24 22:03:44 +08:00
jaime ed3c29eae7
Support Kafka (#16149) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-04-12 19:47:33 +08:00
godchen bb7a0766fe
Add dependency factory (#16204) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-04-07 22:05:32 +08:00
Ten Thousand Leaves 1e6f6bd072
Implement set segment state call from RootCoord to DataCoord. (#16374) 
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-06 15:33:32 +08:00
XuanYang-cn a77dd1034c
Adopt the extended state in DataCoord (#16200) 
ChannelWatchInfo with ToWatch, ToRelease will trigger a timer.
ChannelManager now reacts to different ChannelWatch states.

- WatchSuccess > log this info

- WatchFailure/WatchTimeout > ToRelease

- ReleaseSuccess > Delete, reassign if not from DropCollection

- ReleaseFailure/ReleaseTimeout > Cleanup subscription and Delete,
  reassgin if not from DropCollection.

Some Notes:
1. Reassignment will add this channel to buffer if there's only one node.

See also: #15846

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-03-28 22:33:27 +08:00
xiyichan ca129d4308
Support configurable msgstream (#16131) 
Signed-off-by: xiyichan <2863768433@qq.com>
 2022-03-24 10:15:25 +08:00
Ji Bin 3cd28420f1
Support compile under windows (#15786) 
This patch makes compile milvus under windows(MSYS), including:
- some cpp adaptation for compile under msys/gcc-10.3
- install toolchain scripts for setup from MinGW/MSYS `scripts/install_deps_msys.sh`
- adaptation for POSIX API use in golang
  * using gofrs/flock instead of syscall.Flock
  * using x/exp/mmap instead of syscall.Mmap
- introducing github actions for build milvus.exe under windows/MSYS
- rocksdb's patch for MSYS
- adaptation for compile knowhere under windows
- a windows package script for pack zip file, `scripts/package_windows.sh`

issue #7706

Signed-off-by: Ji Bin <matrixji@live.com>
 2022-03-17 17:17:22 +08:00
zhenshan.cao f3eeecf146
Reduce the number of metrics (#16030) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-03-15 21:51:21 +08:00
jaime 29975a7a26
Remove Consume() API from mqstream interface (#15886) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-11 20:09:59 +08:00
Cai Yudong 3e862ecdfb
Merge msgChannelConfig and knowhereConfig into commonConfig (#15843) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-03-04 11:17:56 +08:00
jaime 7a3a721380
Reconstruct mqstream module (#15784) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-03 21:57:56 +08:00
Xiaofan bd7d294c6d
Refine log level in data (#15821) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-03-02 15:35:55 +08:00
congqixia b0923f1299
Fix DataCoord panics if message queue service quits before it (#15702) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-02-23 11:09:52 +08:00
Nemo 3eb074a9b0
Update some comments for data coord with some minor code refactoring. (#15686) 
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-02-22 13:15:51 +08:00
congqixia c7f55c2e34
Make SessionWatch keep watch even no Rewatch func when find ErrCompacted (#15497) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-02-15 15:07:48 +08:00
Xiaofan be42627ab2
Move Logutil to Util dir (#15447) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-02-09 18:55:46 +08:00
Cai Yudong 1f337891e1
Rename base_param to service_param, rename global_param to component_param (#15463) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-08 20:57:47 +08:00
Cai Yudong f1ea9613ae
Add EtcdConfig and move PulsarConfig/RocksdbConfig/MinioConfig into BaseParamTable (#15434) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-07 10:09:45 +08:00
Cai Yudong 4bccc85534
Use MsgChannelConfig in GlobalParams for all components (#15163) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-02 00:35:43 +08:00
Xiaofan 885ecaf222
Refine time sync logic log (#15251) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-01-27 11:25:41 +08:00
Bingyi Sun 3987574448
Unsubscribe topic in DataCoord (#15353) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-01-24 19:47:38 +08:00
Cai Yudong b6a48817b0
Add MinioConfig in GlobalParams for all components (#15099) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-10 17:29:34 +08:00
Cai Yudong 7efb02a4c5
Use PulsarConfig in GlobalParams for all components (#15046) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-09 23:45:33 +08:00
Cai Yudong f8f61fb116
Use BaseParams etcd configuration for all components (#15012) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-07 16:15:26 +08:00
godchen b0357360d2
[skip e2e]Sort imports (#14807) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-01-05 14:04:46 +08:00
Bingyi Sun 37c9419707
Use logutil in DataCoord (#14737) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-01-04 17:31:23 +08:00
Bingyi Sun 75b0e9935b
[skip e2e]Use logutil in DataCoord (#14739) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-01-04 13:53:39 +08:00
Jiquan Long 00b3fcb98b
Add log for flowgraph (#14441) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-12-29 14:55:21 +08:00
Xiaofan 69087ff8dd
Support embedded etcd (#14333) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-12-29 14:35:21 +08:00
Bingyi Sun 02e568603b
Add log with context in DataCoord (#14431) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-12-28 19:42:19 +08:00
Bingyi Sun 098516f3d7
Improve log (#13696) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-12-27 16:52:06 +08:00
congqixia 7c83b52de0
Add retry logic for DataCoord GC minio bucket (#14095) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-23 21:48:25 +08:00
Cai Yudong 891c202b73
Use GlobalParamTable for all components (#13885) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-12-23 18:39:11 +08:00
Bingyi Sun ca71e134d1
[skip e2e]Improve log (#13700) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-12-22 15:15:05 +08:00
Xiangyu Wang 1341dcd3f8
Improve import statement (#13730) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-12-21 15:21:25 +08:00
Bingyi Sun 9d44ea034c
[skip e2e]Improve log (#13704) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-12-21 09:17:00 +08:00
edward.zeng 38add97685
[skip e2e] Refine datacoord server comment (#13797) 
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
 2021-12-20 19:54:58 +08:00
Bingyi Sun 7ee0a2bf9f
[skip e2e]Improve log (#13707) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-12-20 11:18:49 +08:00
Bingyi Sun 229a727b3a
[skip e2e]Improve log (#13706) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-12-20 11:16:52 +08:00
Bingyi Sun 9e0219ec3f
[skip e2e]Improve log (#13702) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-12-20 11:14:51 +08:00
Bingyi Sun 06c4662e97
[skip e2e]Improve log (#13705) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-12-20 11:06:43 +08:00
Bingyi Sun 279316e09b
[skip e2e]Improve log (#13701) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-12-20 11:02:45 +08:00
cai.zhang 9f23fc7f2a
Register the service when the component state is healthy (#13248) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2021-12-15 11:47:10 +08:00
Bingyi Sun 0c39e0370c
Move segment statistics update to datanode tt loop (#13233) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-12-15 10:53:16 +08:00
congqixia 3b4d114b71
Add comment for DataCoord startFlushLoop (#13379) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-15 09:45:09 +08:00
Bingyi Sun a78b2010e9
[skip ci]Improve log style (#13008) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-12-08 20:27:06 +08:00
Bingyi Sun 48c5632277
[skip ci]Improve log style (#13007) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-12-08 20:25:15 +08:00
Bingyi Sun 320cfbb606
[skip ci]Improve log style (#13006) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-12-08 20:19:26 +08:00
congqixia de454956fe
Refactor session WatchSessions to allow rewatch when Rev Compacted (#12880) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-08 10:11:04 +08:00
congqixia 4121e31df1
Send SIGINT to runner goroutine after etcd disconnects (#12163) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-22 16:23:17 +08:00
congqixia b99164f43d
Add datacoord GC configuration (#12061) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-18 22:29:40 +08:00
Bingyi Sun 175a656ff1
Unwatch dropped channel when init channel manager (#11986) 
issue: #11558
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-17 23:25:12 +08:00
congqixia 5edbb82610
Add session revoke (#11908) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-16 22:31:14 +08:00
Bingyi Sun e3e4dc7153
Improve GetVChanPosistion and fix unit tests (#11888) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-16 21:41:50 +08:00
congqixia d38043f235
Filter with partition id in GetVChanPosition (#11746) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-13 08:45:09 +08:00
Xiaofan 71e814f796
Remove common.yaml and component.yaml (#11661) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-11-12 21:25:08 +08:00
bigsheeper 93149c5ad9
Load growing segment in query node (#11664) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-11-12 18:27:10 +08:00
Bingyi Sun 54b40da4c5
Add segment drop logic (#11621) 
issue: #11558
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-12 00:22:42 +08:00
Bingyi Sun 7a3873960b
Fix watch channel with nil seek position (#11650) 
issue: #11649
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-11 19:32:43 +08:00
congqixia 3c9610cb1b
Add garbage collector for DataCoord (#11554) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-10 19:03:38 +08:00
congqixia 0c4e09e341
Add comment for datacoord Server.startDataNodeTtLoop (#11033) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-05 23:01:02 +08:00
sunby c26cd79532
Add compaction (#9906) 
issue: #9904
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-11-05 22:25:00 +08:00
sunby a55019825b
Fix stale segments selector (#11051) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-11-01 22:30:04 +08:00
Xiaofan 2df2d488e1
Fix Unit test fatal fail (#10933) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-10-30 10:24:38 +08:00
congqixia 390fad4eb7
Fix LatestPosition option conflict with earliest patch (#10907) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-29 21:30:49 +08:00
congqixia c019e80dbb
Change SeekPosition to earliest of all segments (#10771) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-28 14:30:26 +08:00