milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
16,855 Commits
133 Branches
115 Tags
559 MiB
Commit Graph

6666 Commits (383915cfcd88fd7246d1a1e19b4bf218ba2521a8)

Author SHA1 Message Date
wei liu 137e620a8c
fix search failed msg (#22036) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-02-08 15:44:31 +08:00
groot c21e11eb8a
Fix datanode crash bug for bulkinsert (#22041) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-02-08 14:06:31 +08:00
congqixia 81fbb885eb
Remove redundant optimization code (#22044) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-02-07 20:12:30 +08:00
wei liu 6fb60629be
fix describe rg api (#21998) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-02-07 19:30:31 +08:00
congqixia 7adabe09f1
Fix Alloc timstamp failure blocks compaction queue forever (#22039) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-02-07 19:02:31 +08:00
wei liu aced41d5d8
add resource group name rule (#21992) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-02-07 17:52:31 +08:00
Cai Yudong fc1d2d18cf
Optimize knowhere CMakeLists.txt (#22000) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2023-02-07 16:12:04 +08:00
Jiquan Long a548ce1cf1
Change default max shards num to 64 (#22016) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-02-07 14:43:57 +08:00
yah01 5abb8a0853
Improve search task merge policy (#22006) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-02-07 14:27:54 +08:00
XuanYang-cn a8db19fbdb
Fix ut multi-remove the timer (#22021) 
See also: #22019

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-02-07 14:15:54 +08:00
Jiquan Long 004a8c25ca
Remove hardcode max shards num (#22004) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-02-07 10:17:53 +08:00
jaime baa035885d
Modify default value of some configurations (#22013) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-02-07 10:05:56 +08:00
smellthemoon ffdb652911
Fix kafka ut (#21943) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-02-06 16:33:54 +08:00
Bingyi Sun 4005508880
Fix load collection blocks when memory is limited. (#21990) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2023-02-06 16:07:53 +08:00
wei liu 5da6aade02
fix default capacity of default rg (#21965) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-02-06 15:55:56 +08:00
congqixia 73b6c13245
Expand singleflight to Get and make writeEvent sync (#21993) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-02-06 15:27:54 +08:00
yah01 73ce87dfe5
Fix reduce decreasing recall (#21981) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-02-06 11:23:53 +08:00
cai.zhang e25f987a5c
Fix bug that calculated an extra DataNode of register policy (#21927) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-02-06 10:33:54 +08:00
cai.zhang ae305a54bd
Set scheduler duration of index task can be configurable (#21944) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-02-05 17:21:53 +08:00
Ten Thousand Leaves f9495fef00
Minor fix on segment metric (#21976) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2023-02-03 19:33:52 +08:00
zhenshan.cao 5316743a00
Reduce memory usage of scalar index (#21970) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-02-03 19:01:53 +08:00
Enwei Jiao 6d05f102c1
Upgrade boost to 1.81 (#21954) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-02-03 17:21:53 +08:00
jaime 748a935a11
Modify segment seal policy with number of statslog files (#21957) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-02-03 17:07:52 +08:00
congqixia 787ce7c121
Add Rated group for checkpoint gc skip case (#21949) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-02-03 14:23:56 +08:00
Ten Thousand Leaves 74e3cc64fb
Add refresh path to proxy (#21933) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2023-02-02 19:29:51 +08:00
wayblink d41cc0b21b
Revoke session to only delete session key created by this node (#21935) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-02-02 16:37:52 +08:00
wayblink 88293b8b79
Add log to analysis when etcd txn operation failed (#21558) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-02-02 15:57:51 +08:00
aoiasd 6a6818ac32
Fix rocksmq retention test is not stable (#21925) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-02-02 14:09:51 +08:00
Xiaofan 2213ed7bed
Fix bulkload segment can not be flushed (#21919) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-02-02 13:11:51 +08:00
yah01 bdcff5b921
Avoid counting all bits to improve query performance (#21909) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-02-01 18:49:51 +08:00
yah01 9b491858a5
Fix datanode unit test not stable (#21911) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-02-01 18:19:50 +08:00
Enwei Jiao 53ae40b8c6
Make ratelimiter's config refreshable (#21757) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-02-01 16:03:51 +08:00
Bingyi Sun b12740e108
Remove a goroutine when creating a proxy client (#21904) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2023-02-01 15:49:50 +08:00
Cai Yudong e3d1624e97
Handle distance Inf correctly (#21828) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2023-02-01 14:59:51 +08:00
congqixia ec282b1958
Apply lifetime control for indexnode (#21892) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-01-31 18:37:50 +08:00
congqixia 01e59f8ec2
Fix master semver value (#21873) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-01-31 17:41:50 +08:00
wayblink 45ff568470
Fix probable ut fail caused by session key confict (#21883) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-01-31 17:39:49 +08:00
congqixia 11a3d0a50b
Use embed ETCD for datacoord unit test (#21890) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-01-31 14:47:49 +08:00
yah01 61c2f29ab3
Fix miss to throw exception when failed to create dir (#21886) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-01-31 13:49:50 +08:00
congqixia d329cf33a0
Use embed etcd for ConnectManager unit test (#21884) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-01-31 13:35:49 +08:00
Xiaofan f1daef22a1
Fix failed bulkload segment marked as sealed (#21799) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-01-31 12:41:53 +08:00
cai.zhang a04238c25f
Keep the configuration of indexCoord (#21879) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-31 12:07:50 +08:00
congqixia 8af2aebb66
Fix msg channel prefix config priority for backward compatibility (#21871) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-01-31 10:09:51 +08:00
Jiquan Long 270c5b0502
Refine log when initialize dml channels (#21863) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-01-30 14:05:54 +08:00
jaime 5ed6ae2f27
Remove repeated etcd endpoint configuration (#21860) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-01-30 12:07:48 +08:00
jaime e14f96a8e4
[skip e2e]Fix missed logger with information (#21859) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-01-30 11:51:49 +08:00
cai.zhang bcae97b125
Register after start to prevent there are tow coordinator at the same time (#21641) (#21707) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-30 11:11:50 +08:00
smellthemoon bb086f47e4
Swap execution flow order in enqueue (#21845) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-01-30 10:27:50 +08:00
wei liu 73c44d4b29
resource group impl (#21609) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-01-30 10:19:48 +08:00
congqixia 66027790a2
Implement detailed lifetime control for querynode (#21851) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-01-29 17:45:49 +08:00
yah01 43d633cfed
Fix fill the string field twice when string index exists (#21852) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-01-29 17:43:48 +08:00
congqixia 0480719e1a
Fix useasan format not correct (#21829) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-01-28 11:43:47 +08:00
Xiaofan 949d5d078f
Fix memory calculation in dataCodec (#21800) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-01-28 11:09:52 +08:00
jaime d7bb908148
Fix rename collection data race (#21827) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-01-24 10:01:46 +08:00
Xiaofan 9258f7059d
Fix Bulkload ut print too much result (#21821) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-01-19 21:31:44 +08:00
congqixia 54ec22e879
Allow empty AK/SK for minio cfg (#21815) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-01-19 15:49:44 +08:00
congqixia 5986106037
Make paramtable init only once (#21782) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-01-19 14:53:44 +08:00
jaime 1dcd9eeb79
Support rename collection (#21693) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-01-19 14:13:43 +08:00
Bingyi Sun c82b5d15b6
Add unsubscribe dmchannel task (#21513) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2023-01-18 17:45:44 +08:00
Bingyi Sun b0b0929c6c
Change CreateAt to UpdatedAt when checking timeout (#21769) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2023-01-18 17:29:43 +08:00
Ten Thousand Leaves defb7660c8
Add refresh option for LoadCollection/LoadPartitions interfaces (#21776) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2023-01-18 16:41:44 +08:00
Enwei Jiao cecd59d940
Disable time-travel by default (#21775) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-01-18 13:57:43 +08:00
Xiaofan 3e21d25de5
Add slow gc warning log (#21734) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-01-18 11:41:43 +08:00
groot 0af53d1e2e
Fix bulkinsert report bug (#21778) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-01-18 10:33:43 +08:00
yah01 78fd1987eb
Fix data race caused by no clone while updating index (#21772) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-01-17 18:11:42 +08:00
smellthemoon ad6cbc990f
Combine similar function (#21752) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-01-17 17:53:42 +08:00
Bingyi Sun b91bb5a729
Fix load timeout after next target updates (#21759) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2023-01-17 15:15:46 +08:00
presburger e64f55cd27
downgrade curl for compiled on aarch64 platform (#21739) 
Signed-off-by: Yusheng.Ma <Yusheng.Ma@zilliz.com>
 2023-01-17 14:53:42 +08:00
wayblink de584b508e
Fix active-standby switch fail bug (#21755) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-01-17 11:43:43 +08:00
yah01 c8f89907b6
Fix current target may be updated to an invalid target (#21742) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-01-17 11:41:51 +08:00
jaime 81326ca0a1
[skip e2e]Fix cpp lint errors (#21750) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-01-17 09:43:42 +08:00
yah01 6e66c10114
Fix check time tick lag with no channel (#21737) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-01-16 18:49:43 +08:00
Xiaofan e7f947ed0a
Check ScalarIndex deserialization (#21724) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-01-16 17:29:43 +08:00
yah01 836773f1a0
Optimize performance of reducing segments (#21722) 
- Improve the performance of reducing from O(knlogn) to O(nlogk)

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-01-16 15:25:42 +08:00
congqixia 5f3d3dc4fc
generate-mockery for querycoordv2, querynode and rootcoort (#21714) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-01-16 13:59:42 +08:00
yah01 837e3162d7
Fix ready notifiers leak when collection released (#21712) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-01-14 21:55:41 +08:00
yah01 32fb409e57
Fix may update the current target to an unavailable target when node down (#21698) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-01-13 17:11:41 +08:00
yah01 c66ce4aeba
Add warning comments (#21697) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-01-13 16:43:41 +08:00
groot c0e7a122d4
Fix bulkinsert progress bug (#21680) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2023-01-13 16:25:40 +08:00
Enwei Jiao 90d9e165d4
Fix some configs not shown (#21653) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-01-13 15:31:41 +08:00
smellthemoon 4620aebd46
Fix data race (#21706) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-01-13 14:29:41 +08:00
congqixia 782f942b6f
Add MLogger.WithRateGroup for logger (#21703) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-01-13 14:11:40 +08:00
Enwei Jiao 564173dcbc
Upgrade protobuf to 3.21.9 (#21679) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-01-13 14:05:41 +08:00
yah01 f4d80b4211
Fix reading config from etcd may block forever (#21688) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-01-13 11:29:41 +08:00
Xiaofan f8e1566b24
Support zstd compression in grpc (#21689) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-01-13 10:55:41 +08:00
congqixia 204db183f3
Prevent panicking when index segment info not exists (#21664) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-01-12 21:39:40 +08:00
wayblink 6a722396bd
Integration test framework (#21283) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-01-12 19:49:40 +08:00
Enwei Jiao 49f53003e9
upgrade libcurl to fix mac build failed (#21684) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-01-12 19:03:40 +08:00
wei liu c67070ab1f
add resource group interface (#21676) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-01-12 18:31:41 +08:00
Enwei Jiao fb42466c65
Use opentelemetry (#21509) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-01-12 16:09:39 +08:00
Enwei Jiao 9fccbbb92b
Remove factory in querycoord (#21659) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-01-12 15:59:39 +08:00
SimFG b307ee8c07
Make the rbac error message clearer (#21645) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-01-12 13:55:42 +08:00
Xiaofan 8f202274a7
Fix lag behind error with 0 servicetime (#21637) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-01-12 10:53:39 +08:00
Jiquan Long d0b974de07
Make rootcoord client compatible with older version (#21651) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-01-12 10:09:39 +08:00
Jiquan Long 9fd9bed2b9
Remove collection meta after GC finished (#21595) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-01-12 09:55:42 +08:00
Jiquan Long 6d09bbed68
[skip e2e] Fix load meta migration (#21584) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-01-11 19:31:39 +08:00
Ten Thousand Leaves 8c2c04517e
Fix an issue where flushed segments can turn into sealed (#21566) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2023-01-11 18:57:43 +08:00
groot 9f8095996b
Report bulkinsert progress (#21612) (#21638) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2023-01-11 17:37:44 +08:00
cai.zhang f02ccfb422
Set GetIndexBuildProgress is compatible with SDK version 2.1 (#21622) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-11 16:59:42 +08:00
cai.zhang e127cf7b99
Reset indexpb for upgrade (#21620) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-11 14:35:40 +08:00
groot df0ffd08ce
Update segment id for import task (#21621) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2023-01-11 11:55:39 +08:00
liliu-z cc5ecabac6
Skip search GRPC call for standalon (#21383) 
Signed-off-by: Li Liu <li.liu@zilliz.com>
 2023-01-10 21:09:38 +08:00
bigsheeper 2146af1fb2
Return insufficient memory error when load failed (#21574) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-01-10 20:35:39 +08:00
congqixia c550427e81
Use singleflight in localCache.load (#21606) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-01-10 16:21:39 +08:00
yah01 3b41a6931d
Add QueryCoord OWNERS file (#21552) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-01-10 16:15:37 +08:00
groot aab20a4031
Streaming read numpy (#21579) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-01-10 14:53:38 +08:00
cai.zhang 5d660f16d6
Forbid IndexCoord start with version<2.3 (#21597) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-10 14:33:38 +08:00
Ten Thousand Leaves 23ceb9c638
Remove index checking phase for bulk insert (#21578) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2023-01-09 21:15:38 +08:00
congqixia 42ab1b0a48
Add segment gc restraint of dml position before channel cp (#21564) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-01-06 21:33:36 +08:00
Xiaofan dc2bfb0901
Add stack trace for core dump (#21555) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-01-06 16:53:35 +08:00
Xiaofan 4b4944ecee
Support sync all segments while close (#21421) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-01-06 14:49:36 +08:00
MrPresent-Han 6fb3542f2a
enable auto balance paramter(#21504) (#21507) 
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2023-01-06 14:45:35 +08:00
jaime 73d8509600
Improve ListSegments performance for datacoord catalog (#21477) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-01-06 14:33:36 +08:00
bigsheeper 79535931ab
Return error code when deny to read/write (#21561) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-01-06 14:31:37 +08:00
cai.zhang aa203acfb3
Low IndexCoord weight (#21548) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-06 14:21:37 +08:00
yah01 e8e2696430
avoid checking search params many times (#21516) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-01-06 13:15:35 +08:00
Ten Thousand Leaves b3da554705
Increase bulk insert file size limit (#21526) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2023-01-06 13:09:36 +08:00
jaime 6d53a99171
Improve LoadWithPrefix performance for SuffixSnapshot (#21524) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-01-06 10:11:36 +08:00
jaime 09e1d4fc83
Remove redundant LoadPrefix requests for Catalog ListCollections (#21551) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-01-06 10:09:36 +08:00
Enwei Jiao 24d5faf7aa
Fix show configuration bug (#21544) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-01-06 09:55:36 +08:00
congqixia 3d0f516fd7
Add With method for MLogger (#21530) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-01-05 13:13:34 +08:00
cai.zhang efefba4da4
Fix checkpoint will not update after delete and compact (#21495) (#21499) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-05 11:31:35 +08:00
cai.zhang 9dd2e1dcd8
Use tt msg stream for consume delete msg (#21478) (#21500) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-05 11:03:35 +08:00
jaime 58b79eb74c
Add based on timetravel GC for snapshot KV (#21417) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-01-04 21:37:35 +08:00
Jiquan Long 6e521dd4b3
Optimize GetLoadingProgress to save RPC (#21493) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-01-04 20:33:35 +08:00
cai.zhang e5f408dceb
Merge IndexCoord and DataCoord (#21267) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-04 19:37:36 +08:00
yah01 7b39873ae0
limit the frequency of GetMetrics() log (#21514) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-01-04 17:39:35 +08:00
smellthemoon bf3c02155a
Support update (#20875) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-01-04 17:21:36 +08:00
Jiquan Long ff2a68e65a
Fix collection not exist when tried to do recovery (#21471) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-01-04 16:37:35 +08:00
Ten Thousand Leaves 3f6d139ad2
Fix bad import task expiration default (#21235) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2023-01-04 14:11:33 +08:00
congqixia 2844f9b1e9
Setup first tsafe to avoid false maxLag error (#21490) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-01-03 20:41:33 +08:00
Bingyi Sun 72184eaeaf
Upgrade pulsar client (#21456) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2023-01-03 16:27:34 +08:00
wei liu e735485bb8
fix empty auth config unmarshal (#21473) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-01-03 10:11:34 +08:00
SimFG 289df1cc3d
Make the `UnsubDmChannel` rpc unsuccessful when the releaseCollectionTask fails (#21469) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-01-02 16:39:32 +08:00
bigsheeper c187de7754
Show trigger reason when deny to read/write (#21368) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-12-30 18:35:32 +08:00
bigsheeper cf40bb9b5c
Improve datacoord log (#21453) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-12-30 15:37:31 +08:00
Jiquan Long de9ed42623
Forbid deleting entities by non-pk field (#21459) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-12-30 14:17:32 +08:00
SimFG df6ccfa104
Fix wrong bool param in the `getDistribution` method (#21463) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-12-30 14:15:31 +08:00
congqixia daa4492e95
Invalidate cache first when shard leader list contains error (#21451) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-29 23:29:32 +08:00
SimFG e6d2849c9a
Check whether the node is stopping when using `load balance` api (#21438) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-12-29 15:47:31 +08:00
Jiquan Long d7156812c1
Try using ASAN in ci ut (#21089) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-12-29 15:29:31 +08:00
congqixia 8b4d60b073
Add guarantee and serviceable lag too large check (#21441) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-29 15:17:30 +08:00
congqixia 3a58ac9fba
Change node stop to TearDownTest (#21422) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-28 20:11:30 +08:00
SimFG 41bbfbec6a
Fix the pulsar error when datanode is removed (#21415) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-12-28 16:45:30 +08:00
Ten Thousand Leaves 30b86275ac
Make BulkInsert free DataNode workload when import segments are flushed (#21253) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-12-28 11:11:30 +08:00
Xiaofan 3667bb09b8
Avoid Sync when the segment open (#21397) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-12-28 10:51:34 +08:00
wayblink 0bcedbd2bf
quick fix queryCoordV2 active-standby (#21395) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-12-28 10:49:29 +08:00
SimFG 2aed48c433
Fix query node panic when watching dm channels (#21402) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-12-28 10:19:31 +08:00
SimFG 6a29a964df
Fix queryCoord panic during query node down (#21400) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-12-28 10:17:30 +08:00
congqixia c5ff8bf7c8
Fix QueryNode may panics when stopped (#21406) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-27 18:11:30 +08:00
Xiaofan d16b7c3c2d
Add more logs for debugging load balance (#21389) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-12-27 15:57:30 +08:00
jaime a8a1b0b3e8
Remove logs that print insert results (#21394) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2022-12-27 12:49:29 +08:00
wayblink 28b44b781f
Update healthz to support standby state and fix rootcoord active-standby (#21078) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-12-27 12:47:34 +08:00
smellthemoon cded2ce77c
Fix offset panic (#21292) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-12-27 11:11:29 +08:00
Enwei Jiao 264f08953d
Refactor Rocksmq's config (#21304) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-26 19:11:30 +08:00
congqixia 3c9840d6d3
Fix some datanode instance not stopped in unit tests (#21385) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-26 16:59:30 +08:00
Xiaofan 5d044c4f29
Add check for stats log and bin log number (#21264) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-12-26 16:35:30 +08:00
smellthemoon 1e5cd9dc6e
Modify comment (#21362) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-12-26 15:27:30 +08:00
Enwei Jiao 77de8fc746
Fix boost compile error (#21374) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-26 09:57:29 +08:00
yah01 396a85c926
Fix the number of executing tasks may break the limit (#21318) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-12-20 20:33:27 +08:00
jaime d401608899
Add a segment seal policy by number of binlog files (#21263) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-12-20 14:09:25 +08:00
Enwei Jiao ae41a6583f
Fix compile on M1 (#21298) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-19 15:05:24 +08:00
jaime e05ac56283
Revert build arrow with Conan (#21258) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-12-16 18:27:24 +08:00
congqixia 74db56dece
Fix cache refetch logic not applied to Query (#21281) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-16 16:49:23 +08:00
yah01 9fa58968b0
Fix proxy doesn't update shard leaders cache for some types of error (#21185) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-12-16 16:05:24 +08:00
Enwei Jiao 166e9f0da5
Refactor GrpcConfig (#21142) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-16 15:59:23 +08:00
SimFG 63cd4132a6
Implement the `GetLoadState` api (#21257) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-12-16 14:39:24 +08:00
SimFG 4df91ddf60
Support the graceful stop for the indexnode (#21021) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-12-16 10:03:27 +08:00
Ten Thousand Leaves ef3fcdf65b
Small candidate compaction should only happen with >1 segments (#21249) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-12-16 10:01:23 +08:00
jaime 0954014c6d
[skip e2e] Fix broken default parameters for seal segment policy (#21207) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-12-15 18:31:23 +08:00
Enwei Jiao e6374901f9
Make TopK Limit configurable (#21151) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-15 14:03:22 +08:00
aoiasd 0011b65338
Fix unit test of local cache cause data race (#21237) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-12-15 13:41:22 +08:00
liliu-z 6188960e85
Upgrade Knowhere to 1.3.6 to fix bugs (#21240) 
Signed-off-by: Li Liu <li.liu@zilliz.com>

Signed-off-by: Li Liu <li.liu@zilliz.com>
 2022-12-15 12:15:22 +08:00
zhenshan.cao ce3e0fcf9e
Fixbug: proxy meta_cache partially update (#21232) (#21242) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-12-14 20:45:23 +08:00
congqixia 1518baacd1
Make sure querynode stop only once (#21203) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-14 18:21:23 +08:00
yah01 cc56d58052
Fix panic while concurrent releasing payload writer/reader (#21218) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-12-14 16:23:22 +08:00
Ten Thousand Leaves 9d8f422cb8
Reduce maxFieldNum from 256 to 64 (#21104) 
issue: #21077

/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-12-14 10:39:22 +08:00
SimFG f31d5facff
Fix the unsafe casbin `Model` (#21129) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-12-14 10:29:22 +08:00
SimFG 655db658c8
Empty implement of the `GetLoadState` api (#21205) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-12-14 10:23:22 +08:00
Ten Thousand Leaves b01546edd3
Guard bad distance/score from knowhere (#21034) 
/kind bug

issue: #21138
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-12-13 17:03:21 +08:00
bigsheeper fc10c74005
Use channel-cp as seekPosition when FromDmlCPLoadDelete (#21110) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-12-13 16:17:22 +08:00
bigsheeper f595500383
Move sync delete policy from deleteNode to insertBufferNode (#21152) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-12-13 16:15:21 +08:00
Ten Thousand Leaves a81c9cc11c
Recalculate segment max size during compaction (#21077) 
issue: #21077

/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-12-13 15:39:21 +08:00
Enwei Jiao d553b5108e
Fix jemalloc.so not found (#21106) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-13 11:27:21 +08:00
zhenshan.cao 14e737ee2d
Fix metric with wrong label (#21127) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-12-13 09:51:21 +08:00
XuanYang-cn 71cab36e37
Fix channelmanager ut (#21140) 
See also: #17199

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-12-12 19:31:22 +08:00
congqixia 53782ab83e
Fill collection schema after DataCoord restarts (#21161) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-12 19:05:21 +08:00
smellthemoon b04eb17088
Fix load timeout (#21156) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-12-12 18:43:23 +08:00
congqixia b0afdb2180
Fix access log panicking for nil response (#21149) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-12 17:23:22 +08:00
yah01 5ba1a94858
Fix observers may update current target to a unfinished next target (#21107) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-12-12 15:15:22 +08:00
XuanYang-cn b850c4ae2c
Refine search logs in querynode (#21134) 
- Make sure all logs logging with ctx
- Add an info log for searching on not-indexed segments
- Separate log prefix of shardleader and non-shardleaders

See also: #20975

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-12-12 14:33:22 +08:00
XuanYang-cn a7b24cbc53
Move APIs of types.DataNode to services.go (#21042) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-12-12 10:57:22 +08:00
Xiaofan e977e014a9
Fix flush didn't respect binaryvector and other schemas (#21120) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-12-12 10:33:26 +08:00
cai.zhang 0faf24cc54
Forbid gc index meta when creating index (#21061) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-12-12 09:51:20 +08:00
zhenshan.cao abc2adb1f8
Add indexCountLabel for metric QueryNodeNumEntities (#21081) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-12-10 23:45:20 +08:00
Ten Thousand Leaves 699b63fb42
Fix metric with wrong label (#21108) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-12-10 10:55:20 +08:00
liliu-z d326c385bb
Upgrade Knowhere to 1.3.5 to fix DiskANN aio issue (#21109) 
Signed-off-by: Li Liu <li.liu@zilliz.com>

Signed-off-by: Li Liu <li.liu@zilliz.com>
 2022-12-09 22:53:20 +08:00
aoiasd 9079b1d5a4
Rename default accesslog dir name (#21086) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-12-09 18:09:19 +08:00
SimFG 4a5c282b62
Add the superuser config (#21090) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-12-09 16:13:20 +08:00
Ten Thousand Leaves d67db2ed9b
Make compaction target able to exceed segment max size by a bit (#21068) 
issue: #21053
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-12-09 16:03:20 +08:00
aoiasd de0ab9e2cf
Refactor showConfigurations to allow return global config rather than only return config of this component (#21063) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-12-09 14:31:21 +08:00
Xiaofan 908023a06e
Add more test on file reader (#21082) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-12-09 11:55:19 +08:00
SimFG d67e878f96
Fix the concurrent write issue when init privilege policy (#21071) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-12-08 19:43:23 +08:00
aoiasd 60abe08f85
Add retention for access log on minio (#20681) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-12-08 19:39:19 +08:00
yah01 9ebaa10dec
Add more logs for GetShardLeaders (#21046) 
Also increase the heartbeatAvailableInterval from 2.5s to 10s

Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-12-08 19:09:18 +08:00
Bingyi Sun 27219ce9c0
Fix load progress bug (#21069) 
issue: #21023

Signed-off-by: sunby <bingyi.sun@zilliz.com>

Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-12-08 18:41:20 +08:00
smellthemoon 18cad3a1fb
Optimization of delete and insert (#20990) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-12-08 18:37:19 +08:00
cai.zhang eb7ef01b9a
Fix bug for ListWithPrefix from minio with recursive is false (#21059) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-12-08 15:47:18 +08:00
Jiquan Long 6709ccf690
Fix empty retrieve result (#21032) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-12-08 10:15:19 +08:00
Enwei Jiao 89b810a4db
Refactor all params into ParamItem (#20987) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-07 18:01:19 +08:00
yah01 80a2a49681
Improve error message when target id not match (#21043) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-12-07 17:17:19 +08:00
Jiquan Long dee426a7a5
Fix group checker ut (#21038) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-12-07 16:11:22 +08:00
Ji Bin 6692c9999e
Using first unicastable address as local ip (#20674) 
Signed-off-by: Ji Bin <matrixji@live.com>

Signed-off-by: Ji Bin <matrixji@live.com>
 2022-12-07 15:03:19 +08:00
groot 33a102c6fc
Refine bulkinsert (#20986) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-12-07 14:53:18 +08:00
Xiaofan e131915207
Support 64M Rpc Limit (#20845) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-12-07 10:07:18 +08:00
SimFG f8cff79804
Support the graceful stop for the query node (#20851) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-12-06 22:59:19 +08:00
congqixia 5e49b095f5
Fix compaction handler may double release and block forever (#21015) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-06 20:09:18 +08:00
yah01 0297ab1a46
Make progress if any channel/segment was loaded on node (#20775) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-12-06 18:29:19 +08:00
yah01 11e4445ef7
Check whether segments are fully loaded while fetching shard leaders (#20991) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-12-06 18:05:18 +08:00
XuanYang-cn f844aa0cab
Remove not inuse stale segments in flushmsg (#20981) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-12-06 17:51:18 +08:00
smellthemoon c49f20ea94
Load should base on the accurate loaded number (#20999) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-12-06 17:13:18 +08:00
Ten Thousand Leaves 2f5b03fef7
Split small segment proportion and small segment compactable proportion (#20996) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-12-06 14:37:18 +08:00
jaime 039e9ce4bb
Refine msgstream interface (#20832) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-12-05 20:55:17 +08:00
jaime 548e90ec68
Support an analogous Java Guava cache implementation (#20831) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-12-05 20:29:18 +08:00
XuanYang-cn 243d8cff82
Provide different channelNames for channelmanager test (#20969) 
Fixes: #20790

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-12-05 15:39:22 +08:00
yah01 aec347e591
Check last heartbeat response time while fetching shard leaders (#20968) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-12-05 15:09:20 +08:00
yah01 47b76e13be
Fix shard leader misses to fill target ID while delegates query/search (#20973) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-12-05 14:09:17 +08:00
congqixia 2791f5526c
Fix BufferData limit calculation for BinaryVector (#20963) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-05 11:15:17 +08:00
XuanYang-cn 3a558aa04f
Add ut for rootcoord/kv/kv_catalog.go (#20909) 
Add ut for CreateCredential, GetCredential, Dropcredential,
ListCredentials, CreateRole, DropRole AlterUserRole, ListRole,
ListUser, ListUserRole, DeleteGrant, AlterGrant, ListGrant, and, ListPolicy

Fixes: #20833

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-12-05 11:05:17 +08:00
yah01 25a3b9ae19
Check target node ID for query/search (#20967) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-12-04 20:05:17 +08:00
congqixia 4238b5dbab
Make datanode unittest use embed etcd server (#20948) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-02 16:05:16 +08:00
yah01 482986cdf6
Revert "Fix fake segment is never handled (#20935)" (#20945) 
This reverts commit 8f449f2996.

Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-12-02 11:41:16 +08:00
yah01 ec3aef1f3e
Add time records for search path (#20924) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-12-02 11:39:17 +08:00
liliu-z aecf2e4f4b
Upgrade knowhere to 1.3.4 (#20914) 
Signed-off-by: Li Liu <li.liu@zilliz.com>

Signed-off-by: Li Liu <li.liu@zilliz.com>
 2022-12-02 00:33:21 +08:00
congqixia 40abb13413
Generate unique id with ts for sync task (#20928) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-01 20:35:18 +08:00
congqixia f745d7f489
Fix compaction target segment rowNum is always 0 (#20937) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-01 20:33:17 +08:00
zhenshan.cao 9724ae5e27
Refine log level (#20904) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-12-01 20:07:15 +08:00
yah01 8f449f2996
Fix fake segment is never handled (#20935) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-12-01 20:03:16 +08:00
Ten Thousand Leaves b238f4ee58
Remove bad & duplicate metric (#20905) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-12-01 16:39:16 +08:00
cai.zhang 9a42cff9b4
GetFlushedSegments should include flushing segment (#20859) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-12-01 16:35:20 +08:00
Ten Thousand Leaves c373b44c2c
Verify and adjust segment # of rows before saving and passing SegmentInfo (#20837) 
issue: #18120

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-12-01 16:31:16 +08:00
yah01 ddd29ea6ab
Optimize scheduler, increase merge tasks probability (#20922) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-12-01 16:25:16 +08:00
smellthemoon f7d603f947
Fix Group Checker test fail (#20911) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-12-01 13:39:15 +08:00
congqixia 82fe655032
Setup lastSyncTs when add segment to avoid sync next timetick (#20920) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-01 12:29:16 +08:00
xige-16 a20770c172
Delete logs that print sensitive information (#20889) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-12-01 10:35:16 +08:00
Enwei Jiao 2ecdb4ba4a
Etcd config source support TLS (#20874) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-30 18:23:15 +08:00
Cai Yudong 4c5ffc832c
Add param check for BruteForceSearch in segcore (#20838) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-11-30 14:53:15 +08:00
Jiquan Long 5a6a92d603
Refine log level (#20896) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-11-30 14:11:15 +08:00
cai.zhang 83244afcf3
Reduce some logs level which is frequent (#20894) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-11-30 14:07:15 +08:00
yah01 060649b8aa
Refactor task scheduler and executor (#20828) 
Make the performance able to scale out

Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-11-30 13:57:15 +08:00
yah01 18762f82aa
Refine TimeRecorder output (#20897) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-11-30 13:33:15 +08:00
Jiquan Long 8d78d9943a
Fix term filtering failed on indexed scalar field (#20840) (#20882) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-11-30 10:05:15 +08:00
smellthemoon a46a5ac900
Fix rocksmq incompatible (#20888) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-11-30 10:01:15 +08:00
SimFG c6fc1511aa
Update the error message for the rabc method (#20740) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-11-29 20:05:14 +08:00
Ten Thousand Leaves 0700e56008
Increase MaxWatchDuration and make it configurable (#20884) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-11-29 17:19:14 +08:00
yah01 5ffbb01a67
Fix IndexNode panic while failed to create index (#20861) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-11-29 15:31:15 +08:00
Jiquan Long 7f76e798e9
Fix metric ProxyApplyPrimaryKeyLatency (#20486) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-11-29 15:23:14 +08:00
congqixia a8e6d5d47e
Fix delete operation scope to vchannel (#20866) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-11-29 15:15:15 +08:00
congqixia 938c09679c
Fix LRU cache may panic on double close (#20854) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-11-29 09:47:14 +08:00
smellthemoon b5f178f22b
Fix kafka panic (#20860) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-11-28 18:17:14 +08:00
Ten Thousand Leaves 4f9a294b2e
Refine segment stat metrics on QueryNode (#20763) 
issue: #20760

/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-11-28 14:53:14 +08:00
congqixia dd7e309a29
Add log for transfer segment operation (#20829) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-11-28 09:53:13 +08:00
Enwei Jiao 9b61170bf0
Fix compile proto failed (#20820) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-28 09:47:14 +08:00
smellthemoon d9185485f1
Remove TraceID in mq (#20697) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-11-25 23:11:12 +08:00
MrPresent-Han b021d7c59c
[skip e2e] add log for querynode load and remove segment(#20510) (#20810) 
issue: #20510

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2022-11-25 17:33:13 +08:00
congqixia 49c8ce7561
Remove etcd dependency for querynode ut (#20830) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-11-25 14:45:12 +08:00
XuanYang-cn 7fbe0c4df2
Remove mockTestKv in metatable test (#20689) 
See also: #19289

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-11-25 11:07:12 +08:00
yah01 882ba6240b
Check whether CodecIndex is nil to avoid panic (#20816) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-11-25 10:45:12 +08:00
shaoyue 37725d3470
[restapi] Fix insert request lost precision for int64 fields (#20468) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-11-25 10:43:12 +08:00
Ten Thousand Leaves 3cde5b53ad
Fix a panic issue while reporting metrics (#20776) 
issue: #20771

/kind bug

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-11-24 17:37:13 +08:00
SimFG 44d45452fa
Make the sync segment request idempotent in the data node (#20707) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-11-24 10:21:13 +08:00
SimFG 6d9d24b4ca
Make memory and etcd data as consistent as possible (#20683) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-11-24 10:19:12 +08:00
yah01 1eabfdb199
Avoid allocating and coping when fill column with default values (#20721) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-11-23 15:15:11 +08:00
Enwei Jiao 958e94f6f0
Use Conan as c++ package manager (#19920) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-23 10:39:11 +08:00
congqixia e6c8f9cdee
Add collection delta channel in mutex protection (#20736) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-11-23 09:49:11 +08:00
Ten Thousand Leaves a8c7199a52
Fix segment_num, stored_rows_num and stored_rows_count metrics (#20704) 
TODO: better track segment with state other than flushed and dropped.

/kind bug

issue: #20146, #20145
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-11-22 19:21:13 +08:00
XuanYang-cn d1cb380ee0
Enable to gen compaction plan for stale segments (#20741) 
See also: #20502

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-11-22 17:33:16 +08:00
yah01 f76ea292d2
Change LRU to a generic type (#20744) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-11-22 12:11:11 +08:00
congqixia e35b523a99
Use embed etcd for allocator ut (#20749) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-11-22 09:47:16 +08:00
jaime c761de19bc
Fix broken kafka UT (#20743) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-11-21 20:43:13 +08:00
xige-16 1ad702cc40
Reduce the memory used by load disk index (#20475) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-11-21 19:35:11 +08:00
wei liu 67403fcb3b
fix mannual balance with empty segment list (#20738) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2022-11-21 19:29:12 +08:00
wayblink b32f99e2e4
Update datanode.segment.insertBufSize default value to 16777216 (#20713) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-11-21 19:19:11 +08:00
jaime d84cda22ad
Fix start fail with kafka model (#20726) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-11-21 17:49:12 +08:00
xige-16 34c88cea32
Fix GetDiskUsedSize not work (#20474) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-11-21 17:47:17 +08:00
xige-16 15badd0263
Fix cache disk index error (#20419) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-11-21 17:45:23 +08:00
xige-16 82570e057c
Limit MinDim to build disk index (#20724) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-11-21 17:43:10 +08:00
xige-16 81b6703201
Fix spell PQ error (#20725) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-11-21 16:35:11 +08:00
xige-16 ec617d4952
Avoid delete segment which has been loaded (#20730) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-11-21 16:33:12 +08:00
smellthemoon 8283d32ac4
Only balance segement in targets (#20709) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-11-21 16:21:11 +08:00
smellthemoon f5ab719f21
timestamp decided if the pks were the same (#20166) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-11-21 10:55:10 +08:00
MrPresent-Han d44d50e735
fix getting query segment info error during the period of loading and unloading segments (#20549) 
issue: #20281

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2022-11-21 10:41:10 +08:00
groot bf2107ecf5
Fix a regression of local storage (#20653) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-11-21 10:19:10 +08:00
groot c8022ea63c
Change import task timeout value (#20717) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-11-21 10:11:11 +08:00
jaime 0a5270e8c0
Fix alter collection hang (#20694) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-11-21 10:09:10 +08:00
SimFG dffcd974ff
Make sure that the segment path matches the segment id (#20677) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-11-18 15:35:09 +08:00
Xiaofan 633a749880
Recude IndexCodec Load Memory (#20621) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-11-18 10:47:08 +08:00
Bingyi Sun 2390095232
Fix load uses compacted segments' binlogs (#20655) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-11-17 20:37:10 +08:00
bigsheeper fef87f879c
Disable tt delay protection (#20657) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-11-17 20:21:14 +08:00
Enwei Jiao c05b9ad539
Add event dispatcher for config (#20393) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-17 18:59:09 +08:00