49b8657f95
enhance: Support implicit type conversion for parquet ( #29046 )
...
issue: #29019
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-12 16:14:44 +08:00
0e65e90338
enhance: Support otlp with insecure ( #29115 )
...
issue: https://github.com/milvus-io/milvus/issues/28914
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-12-12 11:14:37 +08:00
9d54d6f590
fix: change Abseil to shared library to solve macos compilation issue ( #28986 )
...
fix the compilation error on macos 14.0 with x86 arch processor
related to #28985
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2023-12-11 21:54:37 +08:00
464bc9e8f4
fix: fix reduce precision for search( #27325 ) ( #29031 )
...
related: #27325
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-08 10:04:37 +08:00
dcb662d9ed
enhance: Refine C.NewSegment response and handle exception ( #28952 )
...
See also #28795
Orignal `C.NewSegment` may panic if some condition is not met, this pr
changes response struct to `CNewSegmentResult`, which contains
`C.CStatus` and may return catched exception
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-07 13:34:35 +08:00
fb089cda8b
enhance: Load raw data while scalar index doesn't have raw data ( #28888 )
...
issue: #28886
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-06 20:36:36 +08:00
36f69ea031
feat: integrate storagev2 in building index of segcore ( #28768 )
...
issue: https://github.com/milvus-io/milvus/issues/28655
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-05 16:48:54 +08:00
f01e507b15
[automated] Update Knowhere Commit ( #28965 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-12-05 15:56:35 +08:00
9b6cbe956a
[automated] Update Knowhere Commit ( #28917 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-12-04 15:42:34 +08:00
c8b1a4618a
enhance: Resolve libunwind requirement conflict using 1.7.2 ( #28929 )
...
Try to resolve libunwind dependency requirement conflict between glog &
folly
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-04 14:06:40 +08:00
20fc90c591
enhance: find collection schema from cache ( #28782 )
...
issue: #28781 #28329
1. There is no need to call `DescribeCollection`, if the collection's
schema is found in the globalMetaCache
2. did `GetProperties` to check the access to Azure Blob Service while
construct the ChunkManager
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-03 19:22:33 +08:00
342635ed61
enhance: enable assert method to format arguments ( #28812 )
...
for now the assert method in segcore could accept a string information,
too many codes don't print the value they assert.
make it happy
related #28811
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-01 18:04:33 +08:00
f5856812a2
fix: Fix get binary vector from chunk cache ( #28866 )
...
The way of getting binary vector size is wrong. This PR will fix it.
issue: https://github.com/milvus-io/milvus/issues/28865
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-12-01 14:40:32 +08:00
7206795e91
fix: update folly to resolve simd issue ( #28878 )
...
related #27552 , after this, milvus could run successfully on sse4.2
only machine
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-12-01 13:50:32 +08:00
8036ee13fa
feat: avoid dereferencing nullptr ( #28862 )
...
issue: #28793
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-01 10:20:32 +08:00
ecc3ca374c
[automated] Update Knowhere Commit ( #28882 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-12-01 02:28:31 +08:00
cac802ef7f
enhance: use already installed vcpkg ( #28703 )
...
issue #28686
1. Update Builder gpu image changes, see changes #28505
2. update azure-identity-cpp from beta to release
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-30 15:58:32 +08:00
d69440524b
fix: bypass growing index if no index meta ( #28791 )
...
we shouldn't panic if no index meta, just skip building it
fix #28022
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-11-30 14:10:27 +08:00
1dc086496f
fix: schema->size() check logic with system field ( #28802 )
...
Now segcore load system field info as well, the growing segment
assertion shall not pass with "+ 2" value
This will cause all growing segments load failure
Fix #28801
Related to #28478
See also #28524
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-29 22:40:28 +08:00
3b1b14dd78
fix: update binlog index memory uasge before loading segments ( #28528 )
...
issue: #27678
when interimIndex = true, memory predict should be update with the
memory usage of binlog index build process.
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-11-29 16:42:27 +08:00
86ccb8e146
[automated] Update Knowhere Commit ( #28704 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-11-24 16:56:24 +08:00
6f7a9264d5
enhance: Handle knowhere error for creare diskann index ( #28690 )
...
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-24 11:58:23 +08:00
0d9d098186
enhance: Add precheck when chunk manager init ( #28330 )
...
#28329
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-11-23 19:56:32 +08:00
e9e9b30e8e
enhance: Disable stdout buffer ( #28391 )
...
#28390
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-11-23 19:26:25 +08:00
e5ce385ffd
enhance: remove -inl.h files ( #28674 )
...
issue: https://github.com/milvus-io/milvus/issues/28673
Move template implementations from -inl.h to .cpp file and make explicit
instantiation
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-23 17:20:25 +08:00
c96d07682e
enhance: enhance the accuracy of memory usage ( #28554 )
...
before this, Milvus use container/system's memory info to get the memory
usage, which could be inaccurate.
we allocates the memory by private anon mmap,
then `rss - shared` would be the accurate memory usage
resolve #28553
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-23 15:12:23 +08:00
e8c5e75786
enhance: ignore proto generated files ( #28537 )
...
ignore proto generated files
issue: https://github.com/milvus-io/milvus/issues/28566
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-22 10:20:22 +08:00
bfccfcd0ca
enhance: refine error messages ( #28424 )
...
- Split the simple reason and full detail
- Refine existing error messages
related: #28422
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 17:02:24 +08:00
02c5a649cf
enhance: store system fields in segcore ( #28524 )
...
we need the system fields info for some usacase
fix : #28523
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 09:28:22 +08:00
a1c505dbd5
add internal storage metrics ( #28278 )
...
/kind improvement
issue: #28277
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-19 17:22:25 +08:00
cc713931e9
fix: Fix rocksdb compile failed ( #28484 )
...
Fix : #27675
/kind improvement
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-11-16 16:36:19 +08:00
f7d2ab6677
enhance: reduce 1x copy for variable length field while retrieving ( #28345 )
...
- Reduce 1x copy for varchar/string/JSON/array types while retrieving
- Reduce 1x copy for int8/int16 while retrieving
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-15 18:08:20 +08:00
836f300536
support skip-index based on chunk-metrics to accelerate expr filter( #27925 ) ( #28297 )
...
related: #27925
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-11-15 11:20:19 +08:00
8ec85f5f4c
Add template for VectorMemIndex ( #28324 )
...
Signed-off-by: Writer-X <1256866856@qq.com>
2023-11-11 13:20:22 +08:00
267c67dfee
enhance: reduce 1x copy while retrieving data from growing segment ( #28323 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-10 15:44:22 +08:00
3920bbc55f
Force set aliyun use_virtual_host to true for all ( #28158 )
...
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-11-10 10:32:20 +08:00
01c5484817
No hit when the index exceeds the array length ( #28301 )
...
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-10 10:04:19 +08:00
30847cad3e
Handle exception while loading ( #28304 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-09 17:59:12 +08:00
b7b31ce0bc
Update knowhere commit ( #28285 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-11-09 14:24:19 +08:00
9f9726f8b7
print azure sdk log ( #28240 )
...
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-08 17:50:18 +08:00
2b5f632fa4
Fix bug for constructing ArrayView with fixed-length type ( #28185 )
...
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-07 23:38:21 +08:00
7bb0fa9c70
reduce useless ObjectExists ( #28156 )
...
replace ListBlobs() with GetProperties()
unified style std::string& / char*
config azure requestTimeoutMs
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-07 16:32:20 +08:00
b1df3ead0e
Update knowhere commit ( #28176 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-11-06 22:44:19 +08:00
863e26969a
Reduce 1x memory copy for retrieving data ( #28106 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-06 17:48:19 +08:00
8011054a2a
Check length before comparing strings ( #28110 )
...
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-04 10:04:29 +08:00
501cba90ce
Use IVFFLAT_CC instead of IVFFLAT for sealed segment tmp index ( #28164 )
...
See also #28152
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-03 23:50:15 +08:00
7f28e9d2f3
Update knowhere commit ( #28087 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-11-03 10:24:16 +08:00
fc2df9514f
Refine code for fixed-length types array ( #28108 )
...
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-03 00:40:14 +08:00
dc89730a50
Support collection-level mmap control ( #26901 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-02 23:52:16 +08:00
f8dd589755
Refactor collection's cgo call ( #28055 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-11-02 13:02:13 +08:00
cai.zhang
Enwei Jiao
Xiaofan
MrPresent-Han
congqixia
Bingyi Sun
sre-ci-robot
PowderLi
yah01
yihao.dai
Gao
cqy123456
zhagnlu
Xu Tong