a0e1a1eb31
feat: support enable/disable mmap for index ( #29005 )
...
support enable/disable mmap for index, the user could alter the index's
mode by `AlterIndex` method
related: https://github.com/milvus-io/milvus/issues/21866
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-21 18:07:24 +08:00
ad866d2889
feat: integrate storagev2 into index build process ( #28995 )
...
issue: https://github.com/milvus-io/milvus/issues/28994
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-13 17:24:38 +08:00
b1e0a27f31
enhance: Add logs for each step during service initialization ( #28624 )
...
/kind improvement
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-11-27 16:30:26 +08:00
7445d3711c
feat: trigger compaction to handle index version ( #28442 )
...
issue: https://github.com/milvus-io/milvus/issues/28441
---------
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-11-21 09:26:22 +08:00
90e2c63d9e
Fix getting incorrect CPU num ( #28146 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-11-06 06:02:16 +08:00
6b1a106a31
Moving etcd client into session ( #27069 )
...
Signed-off-by: Filip Haltmayer <filip.haltmayer@zilliz.com>
2023-10-27 07:36:12 +08:00
6060dd7ea8
Add chunk manager request timeout ( #27692 )
...
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-10-23 20:08:08 +08:00
2f201c25e2
Remove deprecated io/ioutil usage ( #27747 )
...
`io/ioutil` package is deprecated, use `io`,`os` package replacement
also added golangci-lint rule to block future reference
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: guoguangwu <guoguangwu@magic-shield.com>
2023-10-17 20:32:09 +08:00
ec1fe3549e
Add a stop hook to clean session ( #27564 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-10-16 10:24:10 +08:00
be980fbc38
Refine state check ( #27541 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-11 21:01:35 +08:00
e4f73cc805
Add host & enable_disk to session ( #27507 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-10-08 20:05:31 +08:00
63ac43a3b8
Refine errors for import ( #27379 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-30 10:31:28 +08:00
370fdaf50d
Record engine version for segment index ( #27384 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-09-28 18:03:28 +08:00
6539a5ae2c
Refine DataCoord status ( #27262 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-26 17:15:27 +08:00
7f7c71ea7d
Decoupling client and server API in types interface ( #27186 )
...
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-09-26 09:57:25 +08:00
5db4a0489e
dynamic index version control ( #27335 )
...
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-25 21:39:27 +08:00
370b6fde58
milvus support multi index engine ( #27178 )
...
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-22 09:59:26 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
a6b98740b7
Replace all status with only error string ( #27125 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-18 14:05:28 +08:00
168e82ee10
Fix panic while handling with the nil status ( #27040 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-15 10:09:21 +08:00
22d7fa4e1c
Use GetXXX instead of directly using dots to get value ( #27045 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-09-15 10:07:19 +08:00
24a04effe9
Delete failed index tasks ASAP ( #27093 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-09-14 19:19:20 +08:00
af5619e73e
Fix misusing chunk manager ( #27051 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-09-13 18:11:20 +08:00
ea2b909c6d
Fix historm metrics buckets range for index ( #26976 )
...
See also: #26743
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-13 17:21:18 +08:00
9dae896395
Fix indexnode log ( #26927 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-09-08 16:57:16 +08:00
9166011c4a
Add float16 vector ( #25852 )
...
Signed-off-by: Writer-X <1256866856@qq.com>
2023-09-08 10:03:16 +08:00
8b5b137207
Should set index task retry not failed when task canceled ( #26878 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-09-06 19:01:14 +08:00
fb0705df1b
Decouple basetable and componentparam ( #26725 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-05 10:31:48 +08:00
3349db4aa7
Refine errors to remove changes breaking design ( #26521 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-04 09:57:09 +08:00
e8f1b1736e
Remove log.Error(err.error())-style log ( #26783 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-01 13:09:01 +08:00
fd37860e57
init knowhere build/search thread pool; ( #26449 )
...
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-08-18 14:00:19 +08:00
78bc688d16
Remove QueryMsgStream in MqFactory interface ( #26374 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-08-16 10:29:33 +08:00
83910593ca
Fix IndexNode panic if storage config empty ( #26302 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-12 21:43:30 +08:00
2770ac4df5
Fix nilness linter errors ( #26218 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-09 11:31:15 +08:00
b9850ce5c0
Fix copylocks linter errors ( #26217 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-08 20:41:15 +08:00
5634ba777d
add new threadpool with various priority to avoid deadlock( #25781 ) ( #26028 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-08-03 09:31:07 +08:00
833674c1cb
add glog configurable function and redirect aws log to segcore log ( #25664 )
...
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-07-27 19:49:02 +08:00
3c503afe7c
Use typeutil.ConcurrentMap instead of sync.Map ( #25846 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-24 10:23:01 +08:00
0f3350ded4
set knowhere thread pool size in index node ( #25804 )
...
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-07-21 18:50:59 +08:00
66fdc71479
Refactor logs in DataCoord & DataNode ( #25574 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-07-14 15:56:31 +08:00
28200acb67
Fix indexnode memory leakage when update index fails ( #25460 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-11 11:18:28 +08:00
33c2012675
Add more metrics ( #25081 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-26 17:52:44 +08:00
04082b3de2
Migrate the ability to upload and download binlog to cpp ( #22984 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-25 14:38:44 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
ebd0279d3f
Check error by Error() and NoError() for better report message ( #24736 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-08 15:36:36 +08:00
37c02c9927
clean up meta interface ( #24729 )
...
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2023-06-08 14:18:35 +08:00
c84bdcea49
merge stats log when segment flushing or compacting ( #23570 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-05-29 10:21:28 +08:00
fe18109aab
Clean tmp disk data when start milvus ( #24404 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-05-26 15:07:26 +08:00
0b72cf2c2e
log out error ( #24260 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-05-19 18:25:24 +08:00
73a181d226
Fix get vector it timeout and improve some string const usage ( #24141 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-16 17:41:22 +08:00
43a9e175a3
Exit component process when session key is deleted ( #21658 ) ( #22164 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-04-12 20:12:28 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
127867b873
Add ratedgroup for some info/warning log ( #23095 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-31 15:22:23 +08:00
1287ca699a
Refine usage of TimeRecorder.Record ( #23142 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-03-30 18:56:22 +08:00
8aebc6f3b7
Remove faiss GPU index support ( #22966 )
...
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-03-24 17:53:58 +08:00
8f847884c2
Fix misc copylocks and remove exclude rule ( #22962 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-24 11:17:58 +08:00
ab3cbdfc61
Partial change to prepare for GPU index type support ( #22591 )
...
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-03-14 23:21:56 +08:00
9e17e2660e
Fix lifetime race in indexnode ( #22645 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-09 16:19:52 +08:00
5cae49e28d
Store index files in slices to reduce memory peaks ( #22369 ) ( #22455 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-03-01 19:43:47 +08:00
7478e44911
Support using mmap to load data ( #22052 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-01 18:07:49 +08:00
697dedac7e
Use cockroachdb/errors to replace other error pkg ( #22390 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
e768437681
Correct usage of Timer and Ticker ( #22228 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-02-23 18:59:45 +08:00
ec282b1958
Apply lifetime control for indexnode ( #21892 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-01-31 18:37:50 +08:00
949d5d078f
Fix memory calculation in dataCodec ( #21800 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-28 11:09:52 +08:00
5986106037
Make paramtable init only once ( #21782 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-01-19 14:53:44 +08:00
90d9e165d4
Fix some configs not shown ( #21653 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-01-13 15:31:41 +08:00
6a722396bd
Integration test framework ( #21283 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-01-12 19:49:40 +08:00
fb42466c65
Use opentelemetry ( #21509 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-01-12 16:09:39 +08:00
aa203acfb3
Low IndexCoord weight ( #21548 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-06 14:21:37 +08:00
e5f408dceb
Merge IndexCoord and DataCoord ( #21267 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-04 19:37:36 +08:00
7b39873ae0
limit the frequency of GetMetrics() log ( #21514 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-01-04 17:39:35 +08:00
4df91ddf60
Support the graceful stop for the indexnode ( #21021 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-12-16 10:03:27 +08:00
de0ab9e2cf
Refactor showConfigurations to allow return global config rather than only return config of this component ( #21063 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2022-12-09 14:31:21 +08:00
89b810a4db
Refactor all params into ParamItem ( #20987 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
aecf2e4f4b
Upgrade knowhere to 1.3.4 ( #20914 )
...
Signed-off-by: Li Liu <li.liu@zilliz.com>
Signed-off-by: Li Liu <li.liu@zilliz.com>
2022-12-02 00:33:21 +08:00
882ba6240b
Check whether CodecIndex is nil to avoid panic ( #20816 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-11-25 10:45:12 +08:00
c05b9ad539
Add event dispatcher for config ( #20393 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-17 18:59:09 +08:00
19524a5344
Fix nodeID mismatch at standalone mode ( #20648 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-17 17:15:08 +08:00
889eb02172
Add goroutine pool for creating index ( #20391 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-11-08 19:41:02 +08:00
7e56e7a976
Fix datarace at Setlogger ( #20376 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-08 14:33:03 +08:00
8ea8f91334
Make index log better ( #20327 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-11-07 14:23:02 +08:00
956c5e1b9d
Make Params singleton ( #20088 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-04 14:25:38 +08:00
39212bc04c
Fix get cpu num for segcore ( #20120 ) ( #20307 )
...
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
2022-11-03 19:29:36 +08:00
e7103310b8
Fix log when building index ( #20302 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-11-03 18:19:35 +08:00
cb2591d1fe
Parallel operations of disk-ann file manager ( #20120 ) ( #20122 )
...
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
2022-11-03 14:39:40 +08:00
5bb2626466
Set diskUsageRatio to 4.0 when build disk index ( #20066 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-26 16:11:31 +08:00
e7d58b4fb7
Set index state as retry when error occurred ( #19978 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-10-25 11:09:30 +08:00
b79687687d
Replace segment index file path with index file id within segment index meta ( #19866 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-19 16:55:27 +08:00
158787811e
Move assemble/disassemble func to core ( #19420 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-16 21:17:25 +08:00
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
a1db9038fb
Move disk index params to config file ( #19714 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-14 17:51:24 +08:00
8c9c1672ae
Assign different storage config for indexes ( #19517 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-14 14:45:23 +08:00
85e04d845c
Move GetComponentStates to milvus.proto, make it an external interface ( #19594 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-10-10 15:55:22 +08:00
0ac4bc32a5
Add logic of autoindex ( #19570 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-10-08 15:38:58 +08:00
838a633584
Add ctx parameter for ChunkManager methods ( #19546 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-29 16:18:56 +08:00
c58a8d8fd6
Support bind IndexNode mode ( #19247 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-09-27 22:38:55 +08:00
2442335aa2
Fix indexnode ut bug ( #19468 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-09-27 14:56:52 +08:00
3fb8e80147
Rename serializeDiskIndex to getIndexFileInfo ( #19416 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-09-27 10:32:53 +08:00
9e18645363
Optimize IndexCoord scheduler loop ( #19375 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-09-24 13:40:52 +08:00
928a213e31
Fix Chunk Manager Path mis use ( #19141 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-09-23 14:40:51 +08:00
yah01
Bingyi Sun
jaime
Enwei Jiao
Filip Haltmayer
zhagnlu
congqixia
yah01
Jiquan Long
foxspy
SimFG
cai.zhang
XuanYang-cn
Xu Tong
cqy123456
MrPresent-Han
xige-16
yiwangdr
aoiasd
smellthemoon
zhenshan.cao
Cai Yudong
Xiaofan
wayblink
liliu-z