Commit Graph

1478 Commits (50672f0d609317898448888748678a2063a9978a)

Author SHA1 Message Date
jaime d126f06946
Decouple mq module from internal proto definition (#22536)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-03-04 23:21:50 +08:00
Enwei Jiao b21395f487
Forbid dmlChannelNum auto refresh (#22550)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-03-03 15:21:48 +08:00
congqixia ea4ab9e4a4
Replace ErrorList with multiErrors (#22532)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-03 14:59:49 +08:00
Gao aa19243c22
Fix diskann param parse bug (#22498)
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-03-02 19:59:50 +08:00
congqixia 51cb8590a1
Use errors.Wrap and Combine so retry error could check `errors.Is` (#22520)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-02 17:25:48 +08:00
Ten Thousand Leaves 74d158e149
Fix enableAutoCompaction default value (#22254)
/kind bug

issue: #22253

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2023-03-02 07:21:48 +08:00
yah01 7478e44911
Support using mmap to load data (#22052)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-01 18:07:49 +08:00
congqixia fa5c33975e
Fix reset connection data race (#22483)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-01 11:21:47 +08:00
yah01 21eea923cc
Add util to combine multiple errors so everyone checkable (#22473)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-02-28 19:35:47 +08:00
yah01 60cd548bf5
Rename concurrency package to conc (#22453)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-02-28 14:19:47 +08:00
congqixia 608615e5bd
Add background health check if ctx err returned (#22439)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-02-28 14:11:47 +08:00
Xiaofan 256aaa1944
Change Etcd session timeout to 20s (#22401)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-02-28 11:13:47 +08:00
Xiaofan fa101e9751
Change RockMQ page size to 256MB (#22432)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-02-28 10:45:47 +08:00
yah01 167581c08a
Refine Pool to improve the ease of use (#22412)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-02-27 18:07:48 +08:00
wayblink 416866e42f
Add memory usage too large sync policy (#22244)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-02-27 17:47:51 +08:00
yah01 b758c305a7
Make Future generic (#22411)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-02-27 12:07:47 +08:00
Enwei Jiao 697dedac7e
Use cockroachdb/errors to replace other error pkg (#22390)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
zhenshan.cao e768437681
Correct usage of Timer and Ticker (#22228)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-02-23 18:59:45 +08:00
SimFG bddda5bdb2
Only print the file content when log the hook.yaml (#22361)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-02-23 15:33:45 +08:00
Enwei Jiao 0851e05014
Gernate milvus.yaml by code (#22003)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-23 11:37:46 +08:00
SimFG 7e19a924b7
Add the rbac for the GetLoadingProgress and GetLoadState api (#22320)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-02-22 20:37:45 +08:00
Cai Yudong d4e0b6e91b
Remove unused index param check (#22271)
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-02-21 17:36:26 +08:00
wayblink b7c0d12d82
Skip file with prefix '.' may exist in filesystem (#22138)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-02-21 10:54:29 +08:00
bigsheeper f4e7b246f2
Fix msgdispatcher concurrency issue and add prometheus metrics (#22174)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-02-16 14:58:34 +08:00
congqixia f2575e5fa8
Add unconvert & durationcheck linters and fix issues (#22161)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-02-15 17:22:34 +08:00
aoiasd 148a024e05
Add tickle for datacoord watch event (#21193)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-02-15 16:20:34 +08:00
wayblink 77a27c6dfc
Support print log to file and console at the same time (#21946)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-02-14 16:56:34 +08:00
SimFG c2a49d5a0b
Add error log and metrics for the hook and privilege interceptor (#22111) (#22137)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-02-13 16:50:33 +08:00
bigsheeper d2667064bb
Add msgDispatcher to support sharing msgs for different vChannel (#21917)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-02-13 16:38:33 +08:00
jaime a2435cfc4f
[Cherry-Pick]Add a segment metrics for count number of binlog files (#22103)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-02-13 10:24:33 +08:00
wei liu d078441522
support rg api rbac (#22097)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-02-10 10:54:33 +08:00
Xiaofan d3f8948c17
Fix embedded milvus (#22077)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-02-09 22:06:32 +08:00
wei liu d085abbd56
fix load collection with rg (#22083)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-02-09 16:24:31 +08:00
yah01 b1f31da77a
Fix activate standby server ignores all errors (#22073)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-02-09 15:24:31 +08:00
Enwei Jiao 44154a828c
Forbid updating some configs (#22047)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-09 14:28:31 +08:00
groot c21e11eb8a
Fix datanode crash bug for bulkinsert (#22041)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-02-08 14:06:31 +08:00
Jiquan Long a548ce1cf1
Change default max shards num to 64 (#22016)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-02-07 14:43:57 +08:00
jaime baa035885d
Modify default value of some configurations (#22013)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-02-07 10:05:56 +08:00
congqixia 73b6c13245
Expand singleflight to Get and make writeEvent sync (#21993)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-02-06 15:27:54 +08:00
cai.zhang ae305a54bd
Set scheduler duration of index task can be configurable (#21944)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-02-05 17:21:53 +08:00
jaime 748a935a11
Modify segment seal policy with number of statslog files (#21957)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-02-03 17:07:52 +08:00
wayblink d41cc0b21b
Revoke session to only delete session key created by this node (#21935)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-02-02 16:37:52 +08:00
Enwei Jiao 53ae40b8c6
Make ratelimiter's config refreshable (#21757)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-01 16:03:51 +08:00
congqixia ec282b1958
Apply lifetime control for indexnode (#21892)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-01-31 18:37:50 +08:00
cai.zhang a04238c25f
Keep the configuration of indexCoord (#21879)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-31 12:07:50 +08:00
congqixia 8af2aebb66
Fix msg channel prefix config priority for backward compatibility (#21871)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-01-31 10:09:51 +08:00
jaime 5ed6ae2f27
Remove repeated etcd endpoint configuration (#21860)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-01-30 12:07:48 +08:00
wei liu 73c44d4b29
resource group impl (#21609)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-01-30 10:19:48 +08:00
congqixia 66027790a2
Implement detailed lifetime control for querynode (#21851)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-01-29 17:45:49 +08:00
Xiaofan 949d5d078f
Fix memory calculation in dataCodec (#21800)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-28 11:09:52 +08:00
Xiaofan 9258f7059d
Fix Bulkload ut print too much result (#21821)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-19 21:31:44 +08:00
congqixia 54ec22e879
Allow empty AK/SK for minio cfg (#21815)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-01-19 15:49:44 +08:00
congqixia 5986106037
Make paramtable init only once (#21782)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-01-19 14:53:44 +08:00
jaime 1dcd9eeb79
Support rename collection (#21693)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-01-19 14:13:43 +08:00
Enwei Jiao cecd59d940
Disable time-travel by default (#21775)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-01-18 13:57:43 +08:00
Xiaofan 3e21d25de5
Add slow gc warning log (#21734)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-18 11:41:43 +08:00
wayblink de584b508e
Fix active-standby switch fail bug (#21755)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-01-17 11:43:43 +08:00
groot c0e7a122d4
Fix bulkinsert progress bug (#21680)
Signed-off-by: groot <yihua.mo@zilliz.com>
2023-01-13 16:25:40 +08:00
Enwei Jiao 90d9e165d4
Fix some configs not shown (#21653)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-01-13 15:31:41 +08:00
Xiaofan f8e1566b24
Support zstd compression in grpc (#21689)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-13 10:55:41 +08:00
wayblink 6a722396bd
Integration test framework (#21283)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-01-12 19:49:40 +08:00
Enwei Jiao fb42466c65
Use opentelemetry (#21509)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-01-12 16:09:39 +08:00
Jiquan Long 9fd9bed2b9
Remove collection meta after GC finished (#21595)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-01-12 09:55:42 +08:00
Jiquan Long 6d09bbed68
[skip e2e] Fix load meta migration (#21584)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-01-11 19:31:39 +08:00
groot 9f8095996b
Report bulkinsert progress (#21612) (#21638)
Signed-off-by: groot <yihua.mo@zilliz.com>
2023-01-11 17:37:44 +08:00
cai.zhang e127cf7b99
Reset indexpb for upgrade (#21620)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-11 14:35:40 +08:00
congqixia c550427e81
Use singleflight in localCache.load (#21606)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-01-10 16:21:39 +08:00
groot aab20a4031
Streaming read numpy (#21579)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-01-10 14:53:38 +08:00
Xiaofan dc2bfb0901
Add stack trace for core dump (#21555)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-06 16:53:35 +08:00
Xiaofan 4b4944ecee
Support sync all segments while close (#21421)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-06 14:49:36 +08:00
cai.zhang aa203acfb3
Low IndexCoord weight (#21548)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-06 14:21:37 +08:00
Ten Thousand Leaves b3da554705
Increase bulk insert file size limit (#21526)
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2023-01-06 13:09:36 +08:00
Enwei Jiao 24d5faf7aa
Fix show configuration bug (#21544)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-01-06 09:55:36 +08:00
cai.zhang e5f408dceb
Merge IndexCoord and DataCoord (#21267)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-04 19:37:36 +08:00
Jiquan Long ff2a68e65a
Fix collection not exist when tried to do recovery (#21471)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-01-04 16:37:35 +08:00
Ten Thousand Leaves 3f6d139ad2
Fix bad import task expiration default (#21235)
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2023-01-04 14:11:33 +08:00
wei liu e735485bb8
fix empty auth config unmarshal (#21473)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-01-03 10:11:34 +08:00
congqixia 8b4d60b073
Add guarantee and serviceable lag too large check (#21441)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-12-29 15:17:30 +08:00
SimFG 2aed48c433
Fix query node panic when watching dm channels (#21402)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-12-28 10:19:31 +08:00
smellthemoon cded2ce77c
Fix offset panic (#21292)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-12-27 11:11:29 +08:00
Enwei Jiao 264f08953d
Refactor Rocksmq's config (#21304)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-26 19:11:30 +08:00
jaime d401608899
Add a segment seal policy by number of binlog files (#21263)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-12-20 14:09:25 +08:00
Enwei Jiao 166e9f0da5
Refactor GrpcConfig (#21142)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-16 15:59:23 +08:00
SimFG 4df91ddf60
Support the graceful stop for the indexnode (#21021)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-12-16 10:03:27 +08:00
jaime 0954014c6d
[skip e2e] Fix broken default parameters for seal segment policy (#21207)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-12-15 18:31:23 +08:00
Enwei Jiao e6374901f9
Make TopK Limit configurable (#21151)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-15 14:03:22 +08:00
aoiasd 0011b65338
Fix unit test of local cache cause data race (#21237)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2022-12-15 13:41:22 +08:00
Ten Thousand Leaves 9d8f422cb8
Reduce maxFieldNum from 256 to 64 (#21104)
issue: #21077

/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-12-14 10:39:22 +08:00
SimFG f31d5facff
Fix the unsafe casbin `Model` (#21129)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-12-14 10:29:22 +08:00
XuanYang-cn b850c4ae2c
Refine search logs in querynode (#21134)
- Make sure all logs logging with ctx
- Add an info log for searching on not-indexed segments
- Separate log prefix of shardleader and non-shardleaders

See also: #20975

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-12-12 14:33:22 +08:00
Xiaofan e977e014a9
Fix flush didn't respect binaryvector and other schemas (#21120)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-12-12 10:33:26 +08:00
SimFG 4a5c282b62
Add the superuser config (#21090)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-12-09 16:13:20 +08:00
Ten Thousand Leaves d67db2ed9b
Make compaction target able to exceed segment max size by a bit (#21068)
issue: #21053
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-12-09 16:03:20 +08:00
aoiasd de0ab9e2cf
Refactor showConfigurations to allow return global config rather than only return config of this component (#21063)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2022-12-09 14:31:21 +08:00
aoiasd 60abe08f85
Add retention for access log on minio (#20681)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2022-12-08 19:39:19 +08:00
yah01 9ebaa10dec
Add more logs for GetShardLeaders (#21046)
Also increase the heartbeatAvailableInterval from 2.5s to 10s

Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-12-08 19:09:18 +08:00
Jiquan Long 6709ccf690
Fix empty retrieve result (#21032)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-12-08 10:15:19 +08:00
Enwei Jiao 89b810a4db
Refactor all params into ParamItem (#20987)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
Jiquan Long dee426a7a5
Fix group checker ut (#21038)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-12-07 16:11:22 +08:00
Ji Bin 6692c9999e
Using first unicastable address as local ip (#20674)
Signed-off-by: Ji Bin <matrixji@live.com>

Signed-off-by: Ji Bin <matrixji@live.com>
2022-12-07 15:03:19 +08:00
groot 33a102c6fc
Refine bulkinsert (#20986)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-12-07 14:53:18 +08:00
Xiaofan e131915207
Support 64M Rpc Limit (#20845)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-12-07 10:07:18 +08:00
SimFG f8cff79804
Support the graceful stop for the query node (#20851)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-12-06 22:59:19 +08:00
Ten Thousand Leaves 2f5b03fef7
Split small segment proportion and small segment compactable proportion (#20996)
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-12-06 14:37:18 +08:00
jaime 039e9ce4bb
Refine msgstream interface (#20832)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-12-05 20:55:17 +08:00
jaime 548e90ec68
Support an analogous Java Guava cache implementation (#20831)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-12-05 20:29:18 +08:00
yah01 aec347e591
Check last heartbeat response time while fetching shard leaders (#20968)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-12-05 15:09:20 +08:00
congqixia 4238b5dbab
Make datanode unittest use embed etcd server (#20948)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-12-02 16:05:16 +08:00
yah01 ec3aef1f3e
Add time records for search path (#20924)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-12-02 11:39:17 +08:00
zhenshan.cao 9724ae5e27
Refine log level (#20904)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-12-01 20:07:15 +08:00
Ten Thousand Leaves c373b44c2c
Verify and adjust segment # of rows before saving and passing SegmentInfo (#20837)
issue: #18120

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-12-01 16:31:16 +08:00
smellthemoon f7d603f947
Fix Group Checker test fail (#20911)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-12-01 13:39:15 +08:00
Enwei Jiao 2ecdb4ba4a
Etcd config source support TLS (#20874)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-30 18:23:15 +08:00
yah01 060649b8aa
Refactor task scheduler and executor (#20828)
Make the performance able to scale out

Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-30 13:57:15 +08:00
yah01 18762f82aa
Refine TimeRecorder output (#20897)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-30 13:33:15 +08:00
Ten Thousand Leaves 0700e56008
Increase MaxWatchDuration and make it configurable (#20884)
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-11-29 17:19:14 +08:00
yah01 5ffbb01a67
Fix IndexNode panic while failed to create index (#20861)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-29 15:31:15 +08:00
congqixia 938c09679c
Fix LRU cache may panic on double close (#20854)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-11-29 09:47:14 +08:00
Ten Thousand Leaves 4f9a294b2e
Refine segment stat metrics on QueryNode (#20763)
issue: #20760

/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-11-28 14:53:14 +08:00
smellthemoon d9185485f1
Remove TraceID in mq (#20697)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-11-25 23:11:12 +08:00
yah01 f76ea292d2
Change LRU to a generic type (#20744)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-22 12:11:11 +08:00
xige-16 1ad702cc40
Reduce the memory used by load disk index (#20475)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-11-21 19:35:11 +08:00
wayblink b32f99e2e4
Update datanode.segment.insertBufSize default value to 16777216 (#20713)
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-11-21 19:19:11 +08:00
jaime d84cda22ad
Fix start fail with kafka model (#20726)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-11-21 17:49:12 +08:00
xige-16 34c88cea32
Fix GetDiskUsedSize not work (#20474)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-11-21 17:47:17 +08:00
xige-16 82570e057c
Limit MinDim to build disk index (#20724)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-11-21 17:43:10 +08:00
xige-16 81b6703201
Fix spell PQ error (#20725)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-11-21 16:35:11 +08:00
smellthemoon f5ab719f21
timestamp decided if the pks were the same (#20166)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-11-21 10:55:10 +08:00
groot bf2107ecf5
Fix a regression of local storage (#20653)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-11-21 10:19:10 +08:00
jaime 0a5270e8c0
Fix alter collection hang (#20694)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-11-21 10:09:10 +08:00
SimFG dffcd974ff
Make sure that the segment path matches the segment id (#20677)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-11-18 15:35:09 +08:00
Xiaofan 633a749880
Recude IndexCodec Load Memory (#20621)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-11-18 10:47:08 +08:00
bigsheeper fef87f879c
Disable tt delay protection (#20657)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-11-17 20:21:14 +08:00
Enwei Jiao c05b9ad539
Add event dispatcher for config (#20393)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-17 18:59:09 +08:00
Enwei Jiao 19524a5344
Fix nodeID mismatch at standalone mode (#20648)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-17 17:15:08 +08:00
groot 8da44638ac
Refine json parser for import (#20545)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-11-16 19:05:08 +08:00
Xiaofan 58ae746f69
Fix GC Tuner Overflow (#20497)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-11-15 10:43:06 +08:00
Xiaofan af66a0b621
Fix Time Tick session2ChanTs map use the same nodeID (#20537)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-11-14 14:41:11 +08:00
groot e217b951ee
Fix bulkinsert bug for int64 value (#20491)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-11-14 10:15:10 +08:00
bigsheeper fc15789da9
Ensure compatibility of channel seek position and move syncPeriod to config (#20504)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-11-12 21:09:04 +08:00
wei liu 701e7b5a6e
support pulsar tenant and auth (#19727)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2022-11-11 10:55:05 +08:00
bigsheeper cd19d99ad7
Add channel level checkpoint (#20350)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-11-10 22:13:04 +08:00
yah01 663bdf3399
Add RootPath field back (#20488)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-10 19:43:03 +08:00
aoiasd fca2b71e28
add access log for proxy (#19927)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2022-11-10 17:09:06 +08:00
groot 53850e340c
Dont expire pending import tasks (#20431)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-11-10 10:25:08 +08:00
wei liu f925fa7661
fix check segment stale task (#20401)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2022-11-08 19:43:05 +08:00
Enwei Jiao 7e56e7a976
Fix datarace at Setlogger (#20376)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-08 14:33:03 +08:00
wayblink a7bed1c927
Pass backup flag through Import request options (#20334)
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-11-08 11:33:03 +08:00
wei liu c5cd92d36e
update target (#19296)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2022-11-07 19:37:04 +08:00
MrPresent-Han 21b54709a2
enable delbuf auto flush function to avoid OOM when del msg accumulating (#20213)
issue: #19713

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
Co-authored-by: MrPresent-Han <zilliz@zillizdeMacBook-Pro-2.local>
2022-11-07 18:49:02 +08:00
groot b847c425e1
Fix listImportTasks bug (#20301)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-11-07 17:11:02 +08:00
Xiaofan 0d18336b21
Fix GCTuner calculation overflow (#20354)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-11-07 12:17:05 +08:00
jaime 709a0a94e9
Add some msgstream metrics (#20296)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-11-07 10:15:02 +08:00
xige-16 4a66965df4
Delete RAW_DATA copy when load IVF_FLAT index data (#20274)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-11-05 17:33:05 +08:00
Xiaofan ffd52f0d08
Add GC Tuner (#20321)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-11-04 22:25:01 +08:00
Enwei Jiao 956c5e1b9d
Make Params singleton (#20088)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-04 14:25:38 +08:00
zhagnlu 15b2d1cace
Change default thread core coefficient (#20120) (#20328)
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
2022-11-04 12:41:35 +08:00
zhagnlu cb2591d1fe
Parallel operations of disk-ann file manager (#20120) (#20122)
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
2022-11-03 14:39:40 +08:00
Ten Thousand Leaves 5708352a80
Improve DataCoord and DataNode logs (#20265)
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-11-03 12:15:35 +08:00
groot cd3b3d3da6
Only allow one file for row-based import (#20234)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-11-02 10:23:35 +08:00
shaoyue 9fac1476f3
Add support for GCS(GoogleCloudStorage) with IAM (#20164)
go mod tidy

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-11-01 11:07:35 +08:00
bigsheeper a2d5947fe4
Change limiter burst type to float64 (#20215)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-31 20:59:34 +08:00
groot c6151ad351
Parse utf32 string of numpy file (#20176)
Signed-off-by: groot <yihua.mo@zilliz.com>

Signed-off-by: groot <yihua.mo@zilliz.com>
2022-10-31 17:39:34 +08:00
Xiaofan 4136009a9a
Support load segments/channels in parallel (#20036)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-31 13:55:33 +08:00
Jiquan Long 8d0cc4226c
Fix IVF_SQ nbits check (#20183)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-10-31 10:13:34 +08:00
bigsheeper bddd47c60f
Prevent limiter burst overflow (#20153)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-28 19:05:32 +08:00
bigsheeper b949fa0ea9
Increase default ttProtection max delay (#20125)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-27 17:37:32 +08:00
groot bee66631e3
Refine bulkload (#19671)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-10-27 16:21:34 +08:00
bigsheeper f0c616050a
Reduce and improve quota center log (#20058)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-25 19:39:30 +08:00
Xiaofan 1f170ce1ba
Refine kafka consumer (#19846)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-25 13:23:30 +08:00
smellthemoon 2f19e6595a
Fill SourceID (#20000)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-10-24 23:43:30 +08:00
Xiaofan 1b5e765307
Block Flowgraph before handle compacted segments (#19988)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-24 15:09:29 +08:00
Xiaofan 7d2ba60412
If autoID enabled, check to make sure user didn't specify autoID field (#19943)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-21 19:03:27 +08:00
smellthemoon 581e1d8c35
Fill MsgBase ALL (#19912)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-10-21 15:57:28 +08:00
wayblink dbd60757c9
Support pass start_ts and end_ts through Import API (#19810)
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-10-20 12:15:27 +08:00
yah01 470e8ef419
Store index ID to filter out invalid handoff event (#19703)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-20 11:29:27 +08:00
jaime b79687687d
Replace segment index file path with index file id within segment index meta (#19866)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-19 16:55:27 +08:00
bigsheeper 99df2d9b54
Add binlogSize and ttDelay metrics monitor (#19872)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-19 15:13:26 +08:00
yah01 7cf26011e6
Add more logs for checkpoint of channel (#19885)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-19 13:19:27 +08:00
smellthemoon 108e51b2f0
[test]Create index before load and fix error message (#19874) (#19857)
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: zhuwenxing <wenxing.zhu@zilliz.com>
2022-10-19 10:01:26 +08:00
wayblink 4cc57adc5d
Initiatively release session in etcd when service close (#19825)
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-10-18 19:13:26 +08:00
jaime 59bcbf0cf6
Support check health for the service level (#19779)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-18 13:39:26 +08:00
Enwei Jiao 0a83cdfe1b
Refactor grpclient, make it generic (#19791)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-10-17 20:05:26 +08:00
bigsheeper 760e9c5df2
Enable rate limit and update quota unit (#19816)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-17 16:23:25 +08:00
Jiquan Long b0a12116c8
Fix meta migration tool (#19814)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-10-17 15:07:25 +08:00
xige-16 158787811e
Move assemble/disassemble func to core (#19420)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-16 21:17:25 +08:00
SimFG a55f739608
Separate public proto files (#19782)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
xige-16 a1db9038fb
Move disk index params to config file (#19714)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-14 17:51:24 +08:00
Ten Thousand Leaves 8b8df0a5e9
Do not use DataCoord context when DataNode is handling import task (#19732)
So that when DataCoord is done, DataNode can still proceed.

/kind bug

issue: #19730
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-10-14 15:15:24 +08:00
xige-16 8c9c1672ae
Assign different storage config for indexes (#19517)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-14 14:45:23 +08:00
bigsheeper 22b32465e7
Make bulkLoad failed if quota exceeds and remove autoindex config (#19770)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-14 14:27:25 +08:00
Enwei Jiao 82dd1bba1c
Refactor healthz API (#19747)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-10-14 14:19:24 +08:00
jaime 8e7a4733ec
Add GetVersion API for the client (#19694)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-14 09:59:24 +08:00
bigsheeper efdce464f5
Fix ttProtection.enabled and memProtection (#19756)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-13 16:15:25 +08:00
bigsheeper f1e330a997
Add read result rate protection (#19728)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-13 14:57:24 +08:00
XuanYang-cn d73186b2eb
Fix timetravel error msg (#19722)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-10-12 15:17:23 +08:00
Jiquan Long a8a074162f
Add meta migration tool (#19709)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-10-12 11:37:23 +08:00
groot a561614d48
Fix bulkload failed bug (#19702)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-10-12 10:27:23 +08:00
bigsheeper e6c76aba0e
Force deny writing if disk quota exceeded (#19680)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-12 10:03:22 +08:00
jaime 10c03de628
Refine method name from BroadCastXx to BroadcastXx (#19700)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-11 21:07:23 +08:00