Commit Graph

401 Commits (0c4321cf579f1cd40ceb5efcfbd60bfc9fd23177)

Author SHA1 Message Date
congqixia 2f201c25e2
Remove deprecated io/ioutil usage (#27747)
`io/ioutil` package is deprecated, use `io`,`os` package replacement
also added golangci-lint rule to block future reference

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: guoguangwu <guoguangwu@magic-shield.com>
2023-10-17 20:32:09 +08:00
jaime ec1fe3549e
Add a stop hook to clean session (#27564)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-10-16 10:24:10 +08:00
SimFG 26f06dd732
Format the code (#27275)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
yihao.dai bb6711f28c
Add ChunkCache: support get vector from storage (#26142)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-15 10:21:20 +08:00
congqixia f7b2ad6650
Fix component pointer to `component` interface never be nil caused panic (#27072)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-14 13:37:18 +08:00
cai.zhang 4fa878d03c
Remove unused log (#27032)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-09-13 16:07:18 +08:00
Enwei Jiao fb0705df1b
Decouple basetable and componentparam (#26725)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-05 10:31:48 +08:00
yah01 3349db4aa7
Refine errors to remove changes breaking design (#26521)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-04 09:57:09 +08:00
congqixia 4bf26dd4ed
Refine standalone components stop order (#26742)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-31 16:51:02 +08:00
congqixia 073ac8350b
Refine signal handler for whole milvus role lifetime (#26642)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-29 19:28:28 +08:00
jaime c603f1c244
Remove mysql metastore (#26633)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-08-29 14:36:26 +08:00
Enwei Jiao 533f0ddf6d
Add amazonlinux image, prepare to remove Centos7 image, remove openblas image (#26368)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-08-18 18:32:19 +08:00
SimFG 8b54914287
Add go.mod file for the milvus/pkg package (#26359)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-08-16 14:49:32 +08:00
Jiquan Long fb5d809599
Fix components' setup (#26182)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-08-09 14:33:15 +08:00
xige-16 33c2012675
Add more metrics (#25081)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-26 17:52:44 +08:00
jaime 18df2ba6fd
[Cherry-Pick] Support Database (#24769)
Support Database(#23742)
Fix db nonexists error for FlushAll (#24222)
Fix check collection limits fails (#24235)
backward compatibility with empty DB name (#24317)
Fix GetFlushAllState with DB (#24347)
Remove db from global meta cache after drop database (#24474)
Fix db name is empty for describe collection response (#24603)
Add RBAC for Database API (#24653)
Fix miss load the same name collection during recover stage (#24941)

RBAC supports Database validation (#23609)
Fix to list grant with db return empty (#23922)
Optimize PrivilegeAll permission check (#23972)
Add the default db value for the rbac request (#24307)

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-06-25 17:20:43 +08:00
SimFG 31c08a8135
Improve the panic code about the milvus component #24783 (#24919)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-06-17 10:32:39 +08:00
congqixia 561021ec5a
Refine sigpipe handling (#24872)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-14 10:10:38 +08:00
yiwangdr 4387f36897
make etcdKV private (#24778)
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2023-06-13 10:52:38 +08:00
congqixia 41af0a98fa
Use go-api/v2 for milvus-proto (#24770)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
chyezh f97127ae55
add nats mq wrappers (#24445)
bug fixup, configurable natsmq, add unittest, pass e2e.



move natsmq to pkg project

Signed-off-by: chyezh <ye.zhen@zilliz.com>
Co-authored-by: yiwangdr <yiwangdr@gmail.com>
2023-06-07 10:00:37 +08:00
xige-16 fe18109aab
Clean tmp disk data when start milvus (#24404)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-05-26 15:07:26 +08:00
cai.zhang 43a9e175a3
Exit component process when session key is deleted (#21658) (#22164)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-04-12 20:12:28 +08:00
Cai Yudong 55202ce7d9
Support show knowhere prometheus metrics (#23102)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2023-04-07 15:34:29 +08:00
jaime c9d0c157ec
Move some modules from internal to public package (#22572)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
aoiasd 1a485044a9
fix NumConsumers metric not work (#23167)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-04-05 08:18:28 +08:00
yah01 7da870f512
Remove useCustomConfig and simpilify the session type (#23166)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-04-03 20:10:24 +08:00
congqixia 8f847884c2
Fix misc copylocks and remove exclude rule (#22962)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-24 11:17:58 +08:00
congqixia 0317e0458b
Use Set[string] instead of map[string]interface{} ServerType (#22853)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-20 16:21:57 +08:00
jaime d126f06946
Decouple mq module from internal proto definition (#22536)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-03-04 23:21:50 +08:00
Enwei Jiao 697dedac7e
Use cockroachdb/errors to replace other error pkg (#22390)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
Enwei Jiao 0851e05014
Gernate milvus.yaml by code (#22003)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-23 11:37:46 +08:00
Xiaofan d3f8948c17
Fix embedded milvus (#22077)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-02-09 22:06:32 +08:00
cai.zhang d09d1e835a
Reset indexcoord flag for running (#21898)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-02-02 15:49:51 +08:00
wayblink e9f99081e4
Start management API before roles run (#21874)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-02-02 12:03:58 +08:00
Enwei Jiao fb42466c65
Use opentelemetry (#21509)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-01-12 16:09:39 +08:00
Jiquan Long 6d09bbed68
[skip e2e] Fix load meta migration (#21584)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-01-11 19:31:39 +08:00
Xiaofan dc2bfb0901
Add stack trace for core dump (#21555)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-06 16:53:35 +08:00
cai.zhang aa203acfb3
Low IndexCoord weight (#21548)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-06 14:21:37 +08:00
cai.zhang e5f408dceb
Merge IndexCoord and DataCoord (#21267)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-04 19:37:36 +08:00
Enwei Jiao 264f08953d
Refactor Rocksmq's config (#21304)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-26 19:11:30 +08:00
Enwei Jiao 166e9f0da5
Refactor GrpcConfig (#21142)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-16 15:59:23 +08:00
Enwei Jiao 89b810a4db
Refactor all params into ParamItem (#20987)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
SimFG f8cff79804
Support the graceful stop for the query node (#20851)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-12-06 22:59:19 +08:00
Enwei Jiao 2ecdb4ba4a
Etcd config source support TLS (#20874)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-30 18:23:15 +08:00
Enwei Jiao c05b9ad539
Add event dispatcher for config (#20393)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-17 18:59:09 +08:00
Enwei Jiao 19524a5344
Fix nodeID mismatch at standalone mode (#20648)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-17 17:15:08 +08:00
Xiaofan af66a0b621
Fix Time Tick session2ChanTs map use the same nodeID (#20537)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-11-14 14:41:11 +08:00
Enwei Jiao b37a9c5fe2
Fix log to file not work (#20541)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-14 14:23:07 +08:00
Enwei Jiao 7e56e7a976
Fix datarace at Setlogger (#20376)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-08 14:33:03 +08:00
Enwei Jiao 956c5e1b9d
Make Params singleton (#20088)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-04 14:25:38 +08:00
Jiquan Long d9081980e2
[skip e2e] Exit migration after session expired (#20282)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-11-03 11:47:36 +08:00
Jiquan Long f2f10cb114
[skip e2e] Expire migration session asap and fix exit code (#20224)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-11-01 11:47:34 +08:00
Jiquan Long e9cd2cb42a
[skip e2e] Fix empty index build meta migration (#20206)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-10-31 17:33:35 +08:00
Jiquan Long 3625b02d50
[skip e2e] Clean load meta of 2.2 when rollback (#20195)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-10-31 13:01:33 +08:00
Jiquan Long 977947e224
[skip e2e] Print detailed migration info (#20178)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-10-28 21:49:33 +08:00
cai.zhang 53289a63e6
Specification log (#20119)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-10-28 17:01:32 +08:00
cai.zhang 1be4d1e267
Fill the WriteHandoff field in the upgrade script (#20139)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-10-27 20:25:35 +08:00
cai.zhang 4848edda8d
Append field indexID for index name (#20046)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-10-27 20:23:38 +08:00
yah01 e54b6181b6
Ignore the collections without index from 2.1 (#20064)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-25 17:45:31 +08:00
cai.zhang a60db370e6
Compatibility with handling index params (#19997)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-10-25 16:59:30 +08:00
yah01 aa6b8af80d
Migrate metadata of QueryCoord from 2.1 to 2.2 (#19986)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-24 23:41:35 +08:00
jaime b79687687d
Replace segment index file path with index file id within segment index meta (#19866)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-19 16:55:27 +08:00
Jiquan Long 34e04c27ad
[skip e2e] Fix missing num_rows after meta migration (#19841)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-10-17 19:40:51 +08:00
Jiquan Long b0a12116c8
Fix meta migration tool (#19814)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-10-17 15:07:25 +08:00
SimFG a55f739608
Separate public proto files (#19782)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
xige-16 a1db9038fb
Move disk index params to config file (#19714)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-14 17:51:24 +08:00
Enwei Jiao 82dd1bba1c
Refactor healthz API (#19747)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-10-14 14:19:24 +08:00
Enwei Jiao 4df3552f74
Prepare http admin API (#19676)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-10-13 10:37:23 +08:00
Jiquan Long a8a074162f
Add meta migration tool (#19709)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-10-12 11:37:23 +08:00
yah01 2925d8a1ac
Remove QueryCoordV1 (#19681)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-11 11:11:23 +08:00
wayblink 85e04d845c
Move GetComponentStates to milvus.proto, make it an external interface (#19594)
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-10-10 15:55:22 +08:00
congqixia 838a633584
Add ctx parameter for ChunkManager methods (#19546)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-29 16:18:56 +08:00
Xiaofan 928a213e31
Fix Chunk Manager Path mis use (#19141)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-09-23 14:40:51 +08:00
xige-16 0ba47bd0d0
Mix start with more role (#19251)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-09-16 20:38:49 +08:00
SimFG d7f38a803d
Separate some proto files (#19218)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
jaime 816967c000
Add metrics for etcd (#18962)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-09-02 17:46:59 +08:00
Zach 0c9a10e8f8
1. refine logging interfaces (#18692)
2. adjust logs for query/search requests

Signed-off-by: Zach41 <zongmei.zhang@zilliz.com>

Signed-off-by: Zach41 <zongmei.zhang@zilliz.com>
2022-08-23 10:44:52 +08:00
codeman 6c3dbf0a07
Support meta storing on database (#17236) (#18541)
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-08-11 12:12:38 +08:00
Xiaofan b427450593
Remove Load Delta Channel Request (#18497)
Fix #18393

remove watch delta channel task related logic

Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2022-08-04 17:50:37 +08:00
jaime 6e7831470c
Revert settings of Pulsar address and refine config code (#18494)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-08-02 21:26:33 +08:00
SimFG eb1f0bc805
Add the consistency check tool (#17136)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-06-21 10:28:12 +08:00
bigsheeper c77bc81057
Set jemalloc LD_PRELOAD env to linux image (#17494)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-10 21:58:08 +08:00
shaoyue 7527285ec4
Fix mixture (#17332)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-06-02 10:34:03 +08:00
Ten Thousand Leaves 6f75d02c65
Disable knowhere logging for embedded Milvus and some other tweaks (#16496)
/kind enhancement

issue: #15711
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-20 17:23:46 +08:00
Ten Thousand Leaves cf9b9aba79
Fix a bug where RPC calls before proxy finishes initialization crashes Milvus (#16474)
/kind bug

issue: #16473
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-14 20:01:34 +08:00
jaime ed3c29eae7
Support Kafka (#16149)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-04-12 19:47:33 +08:00
godchen bb7a0766fe
Add dependency factory (#16204)
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-04-07 22:05:32 +08:00
xiyichan ca129d4308
Support configurable msgstream (#16131)
Signed-off-by: xiyichan <2863768433@qq.com>
2022-03-24 10:15:25 +08:00
Xiaofan 787d4f8bfd
Refine prometheus metrics and etcd log (#16084)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-03-18 15:51:22 +08:00
Ji Bin 3cd28420f1
Support compile under windows (#15786)
This patch makes compile milvus under windows(MSYS), including:
- some cpp adaptation for compile under msys/gcc-10.3
- install toolchain scripts for setup from MinGW/MSYS `scripts/install_deps_msys.sh`
- adaptation for POSIX API use in golang
  * using gofrs/flock instead of syscall.Flock
  * using x/exp/mmap instead of syscall.Mmap
- introducing github actions for build milvus.exe under windows/MSYS
- rocksdb's patch for MSYS
- adaptation for compile knowhere under windows
- a windows package script for pack zip file, `scripts/package_windows.sh`

issue #7706

Signed-off-by: Ji Bin <matrixji@live.com>
2022-03-17 17:17:22 +08:00
Ten Thousand Leaves a3a29fed06
Add entry point for embedded Milvus. (#16008)
/kind feature

issue: #15711
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-03-16 16:51:22 +08:00
Ten Thousand Leaves d5bae6710f
Make all logs configurable for standalone/embedded Milvus. (#15926)
Resolves: #15919, #15708
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-03-10 10:35:59 +08:00
jaime 7a3a721380
Reconstruct mqstream module (#15784)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-03-03 21:57:56 +08:00
Nemo d1b8c9850b
Fix a bug for standalone where it appears that DEPLOY_MODE is get/used before set. (#15712)
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-02-24 14:41:52 +08:00
Xiaofan be42627ab2
Move Logutil to Util dir (#15447)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-02-09 18:55:46 +08:00
Cai Yudong 1f337891e1
Rename base_param to service_param, rename global_param to component_param (#15463)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-08 20:57:47 +08:00
Cai Yudong f1ea9613ae
Add EtcdConfig and move PulsarConfig/RocksdbConfig/MinioConfig into BaseParamTable (#15434)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-07 10:09:45 +08:00
Cai Yudong 72a17339f5
[skip e2e] Remove SIGSEGV handling (#15341)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-21 20:03:38 +08:00
Cai Yudong 7b48dc7b84
[skip e2e] Add SIGSEGV handling (#15248)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-17 17:35:35 +08:00