milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
18,743 Commits
174 Branches
147 Tags
630 MiB
Commit Graph

436 Commits (7b5110d73bc7f043ce631daf5864a95d51a2430d)

Author SHA1 Message Date
zhagnlu 79c417b14e
fix: pass active count to query context instead of timestamp (#29541) 
#29319

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-12-31 16:08:48 +08:00
Jiquan Long 6f4791da0b
fix: panic in concurrent insert/query scenario (#29408) 
issue: https://github.com/milvus-io/milvus/issues/29405

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-12-26 15:10:48 +08:00
cqy123456 4c979538a4
enhance: update cagra index params in config and add params check (#29045) 
issue:https://github.com/milvus-io/milvus/issues/29230
this pr do two things about cagra index:
 a.milvus yaml config support gpu memory settings

 b.add cagra-params check

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
Co-authored-by: yusheng.ma <yusheng.ma@zilliz.com>
 2023-12-26 11:04:47 +08:00
zhagnlu a6eb7e5f9a
enhance: skip segment when using pk in (..) expr (#29394) 
#29293

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-12-21 20:06:42 +08:00
Gao 9b52cb6417
enhance: improve reducing results when many segments are filtered (#29073) 
Do not fill the invalid ids for the empty results, it will incur useless
memory overhead and reduce overhead when nq and topk is large.

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-12-20 12:56:42 +08:00
zhagnlu a602171d06
enhance: Refactor runtime and expr framework (#28166) 
#28165

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-12-18 12:04:42 +08:00
congqixia dcb662d9ed
enhance: Refine C.NewSegment response and handle exception (#28952) 
See also #28795

Orignal `C.NewSegment` may panic if some condition is not met, this pr
changes response struct to `CNewSegmentResult`, which contains
`C.CStatus` and may return catched exception

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-07 13:34:35 +08:00
cai.zhang fb089cda8b
enhance: Load raw data while scalar index doesn't have raw data (#28888) 
issue: #28886

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-12-06 20:36:36 +08:00
Bingyi Sun 36f69ea031
feat: integrate storagev2 in building index of segcore (#28768) 
issue: https://github.com/milvus-io/milvus/issues/28655

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-12-05 16:48:54 +08:00
PowderLi 20fc90c591
enhance: find collection schema from cache (#28782) 
issue: #28781 #28329

1. There is no need to call `DescribeCollection`, if the collection's
schema is found in the globalMetaCache
2. did `GetProperties` to check the access to Azure Blob Service while
construct the ChunkManager

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-12-03 19:22:33 +08:00
yah01 d69440524b
fix: bypass growing index if no index meta (#28791) 
we shouldn't panic if no index meta, just skip building it
fix #28022

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-11-30 14:10:27 +08:00
congqixia 1dc086496f
fix: schema->size() check logic with system field (#28802) 
Now segcore load system field info as well, the growing segment
assertion shall not pass with "+ 2" value
This will cause all growing segments load failure
Fix #28801
Related to #28478
See also #28524

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-29 22:40:28 +08:00
zhagnlu 0d9d098186
enhance: Add precheck when chunk manager init (#28330) 
#28329

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-11-23 19:56:32 +08:00
Bingyi Sun e8c5e75786
enhance: ignore proto generated files (#28537) 
ignore proto generated files
issue: https://github.com/milvus-io/milvus/issues/28566

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-11-22 10:20:22 +08:00
PowderLi a1c505dbd5
add internal storage metrics (#28278) 
/kind improvement
issue: #28277

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-19 17:22:25 +08:00
yah01 f7d2ab6677
enhance: reduce 1x copy for variable length field while retrieving (#28345) 
- Reduce 1x copy for varchar/string/JSON/array types while retrieving
- Reduce 1x copy for int8/int16 while retrieving

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-15 18:08:20 +08:00
MrPresent-Han 836f300536
support skip-index based on chunk-metrics to accelerate expr filter(#27925) (#28297) 
related: #27925

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-11-15 11:20:19 +08:00
Xu Tong 8ec85f5f4c
Add template for VectorMemIndex (#28324) 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-11-11 13:20:22 +08:00
yah01 267c67dfee
enhance: reduce 1x copy while retrieving data from growing segment (#28323) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-10 15:44:22 +08:00
cai.zhang 01c5484817
No hit when the index exceeds the array length (#28301) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-10 10:04:19 +08:00
PowderLi 9f9726f8b7
print azure sdk log (#28240) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-08 17:50:18 +08:00
cai.zhang 2b5f632fa4
Fix bug for constructing ArrayView with fixed-length type (#28185) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-07 23:38:21 +08:00
PowderLi 7bb0fa9c70
reduce useless ObjectExists (#28156) 
replace ListBlobs() with GetProperties()
unified style std::string& / char*
config azure requestTimeoutMs

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-07 16:32:20 +08:00
yah01 863e26969a
Reduce 1x memory copy for retrieving data (#28106) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-06 17:48:19 +08:00
yah01 dc89730a50
Support collection-level mmap control (#26901) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-02 23:52:16 +08:00
Enwei Jiao f8dd589755
Refactor collection's cgo call (#28055) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-11-02 13:02:13 +08:00
Enwei Jiao 8ae9c947ae
Use OpenDAL to access object store (#25642) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-11-01 09:00:14 +08:00
cqy123456 4fbe3c9142
replace loaded binlog with binlog index for search performance (#27673) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-11-01 02:20:15 +08:00
yihao.dai ab6b0103a3
Get vector concurrently (#27838) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-10-30 15:44:12 +08:00
Enwei Jiao 4a33391b8f
rename createindex (#27903) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-10-27 10:12:14 +08:00
Alexander Guzhva fbaec4c1a2
Fix non-removed 2 GB temporary file in test_utils.cpp (#27651) 
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
 2023-10-18 02:24:09 +08:00
PowderLi 09d8b76048
[restful] new context with grpc metadata (#27668) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-10-17 20:00:14 +08:00
Xiaofan d83869aaeb
Refine minio chunks manager (#27510) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-10-13 14:15:35 +08:00
Gao 7a65b6fb85
Limit faiss ivf index build thread num and fix ut (#27567) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-10-11 10:33:33 +08:00
yihao.dai 106c17f304
Make read ahead policy in ChunkCache configurable (#27291) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-28 15:47:27 +08:00
Enwei Jiao b80a3e19d3
Add code for PanicInfo (#27364) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-27 12:01:28 +08:00
foxspy 5db4a0489e
dynamic index version control (#27335) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-25 21:39:27 +08:00
foxspy 370b6fde58
milvus support multi index engine (#27178) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-22 09:59:26 +08:00
yah01 93e2eb78c9
Delete only if primary keys exist (#25292) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-09-20 19:03:25 +08:00
cai.zhang a362bb1457
Support array datatype (#26369) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-09-19 14:23:23 +08:00
PowderLi 4feb3fa7c6
support azure (#26398) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-09-19 10:01:23 +08:00
yihao.dai bb6711f28c
Add ChunkCache: support get vector from storage (#26142) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-15 10:21:20 +08:00
Enwei Jiao 0afdfdb9af
Remove other Exceptions, keeps SegcoreError only (#27017) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-14 14:05:20 +08:00
yah01 3d05ddf505
Reduce cpp test time (#27043) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-13 15:41:18 +08:00
cqy123456 0ff4ddc76c
remove VectorMemNMIndex (#27000) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-09-12 17:13:18 +08:00
Enwei Jiao c3f15c6b95
Refactor duplicate error class into one place (#26985) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-11 20:43:17 +08:00
Xu Tong 9166011c4a
Add float16 vector (#25852) 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-09-08 10:03:16 +08:00
Cai Yudong 8dc16b599b
Add binary metric types SUBSTRUCTURE/SUPERSTRUCTURE back (#26766) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-08-31 20:07:00 +08:00
yah01 ba882b49b6
Optimize query/search on growing segment while output vector field (#26542) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-24 09:46:24 +08:00
zhagnlu 0cb3f88e73
Fix unit test case of thread pool (#26501) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-08-21 21:50:22 +08:00