c4ae9f4ece
feat: introduce third-party milvus-storage ( #39418 )
...
related: https://github.com/milvus-io/milvus/issues/39173
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-01-24 17:21:13 +08:00
5730b69e56
feat: Enable more VECTOR_INT8 unittest ( #39569 )
...
Issue: #38666
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2025-01-24 17:03:07 +08:00
8117d59f85
fix:fix GetValueFromConfig for bool type ( #39526 )
...
#39525
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-01-24 16:17:05 +08:00
341d6c1eb7
feat: Update segcore for VECTOR_INT8 ( #39415 )
...
Issue: #38666
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2025-01-21 11:03:03 +08:00
140c5a0a75
enhance: add unit test for string pk ( #39329 )
...
https://github.com/milvus-io/milvus/issues/39107
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-01-20 19:03:04 +08:00
5b35fc700d
enhance: [skip-e2e] Use template to remove duplicate unittest ( #39396 )
...
Issue: #38666
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2025-01-18 10:33:01 +08:00
64feeb0e2b
enhance: Rename API GenDataset to GenFieldData in unittest ( #39386 )
...
Issue: #38666
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2025-01-17 15:55:03 +08:00
8c4ba70a4c
fix: enable to build index with single segment ( #39233 )
...
fix https://github.com/milvus-io/milvus/issues/39232
---------
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
2025-01-16 11:01:06 +08:00
3e788f0fbd
enhance: record memory size (uncompressed) item for index ( #38770 )
...
issue: #38715
- Current milvus use a serialized index size(compressed) for estimate
resource for loading.
- Add a new field `MemSize` (before compressing) for index to estimate
resource.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-14 10:33:06 +08:00
5e38f01e5b
enhance: update knowhere version ( #39212 )
...
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2025-01-14 10:21:05 +08:00
3447ff7310
enhance: [bitset] extend op_find() to be able to search both 0 and 1 ( #39176 )
...
issue: #39124
`bitset::find_first()` and `bitset::find_next()` now accept one more
parameter, which allows to search for `0` bit instead of `1` bit
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2025-01-14 09:50:58 +08:00
2a02bbe3ee
enhance: Use template to remove unittest duplication ( #39144 )
...
Issue: #38666
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2025-01-13 09:58:57 +08:00
032292a432
feat: support phrase match query ( #38869 )
...
The relevant issue: https://github.com/milvus-io/milvus/issues/38930
---------
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
2025-01-12 20:24:58 +08:00
8abf6c9149
fix: build text index when loading field data ( #39070 )
...
fix: https://github.com/milvus-io/milvus/issues/39053
may fix https://github.com/milvus-io/milvus/issues/38644 which could be
caused by https://github.com/milvus-io/milvus/issues/39053
---------
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
2025-01-09 15:24:56 +08:00
f0dae81494
fix: set iterative filter hint to false when no expr specified ( #39033 )
...
issue: https://github.com/milvus-io/milvus/issues/39013
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2025-01-08 12:56:56 +08:00
3739446a33
enhance: refine array view to optimize memory usage( #38736 ) ( #38808 )
...
related: #38736
700m data, array_length=10
non-mmap_offsets_uint64: 2.0G
mmap_offsets_uint64: 1.1G
mmap_offsets_uint32: 880MB
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2025-01-07 13:26:55 +08:00
907fc24f85
enhance: support null expr ( #38772 )
...
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-01-02 14:16:54 +08:00
2557e3f2a9
enhance: Initialize field id to avoid negative number ( #38789 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-12-27 18:00:50 +08:00
85f462be1a
enhance: speed up search iterator stage 1 ( #37947 )
...
issue: #37548
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-12-26 10:32:49 +08:00
acc8fb7af6
enhance: eliminate compile warnings (part2) ( #38535 )
...
See #38435
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-12-25 15:30:50 +08:00
b537a72309
fix: interted index out of range ( #38577 )
...
issue: #38546 , #38486
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-19 15:20:47 +08:00
9afcc5bc5c
fix:fix incorrect dir operations when create or load inverted index ( #38359 )
...
#37944
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-17 20:06:45 +08:00
dd4f33ae19
fix: Fix chunked segment can not warmup using mmap ( #38492 )
...
issue: #38410
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-12-17 13:42:45 +08:00
33aecb0655
fix: build break on target test_cpp under OSX ( #38479 )
...
See: #38434
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-12-17 13:38:45 +08:00
3e2a2f278b
enhance: Handle rust error in c++ ( #38113 )
...
https://github.com/milvus-io/milvus/issues/37930
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-12-16 19:40:45 +08:00
01de0afc4e
enhance: refactor delete mvcc function ( #38066 )
...
#37413
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-15 18:02:43 +08:00
3038383e36
fix: UT compile broken under osx ( #38432 )
...
See: #38434
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-12-13 16:24:43 +08:00
efbfa1cc3e
fix:fix ut failed for debug ( #38384 )
...
#38382
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-12-13 14:38:43 +08:00
994fc544e7
enhance: support iterative filter execution ( #37363 )
...
issue: #37360
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-12-11 11:32:44 +08:00
8216345b07
enhance: reduce copy of bitset and id conversion of brurtforce search ( #37675 )
...
issue: https://github.com/milvus-io/milvus/issues/37798
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-11-19 15:48:40 +08:00
6b82320953
fix: Fix using wrong upperbound when searching by pk ( #37769 )
...
issue: https://github.com/milvus-io/milvus/issues/37649
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-19 10:16:31 +08:00
3d28d99411
fix: to use the correct offset in span ( #37780 )
...
#37734
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-18 21:56:30 +08:00
e9391acf80
fix: bm25 brute force search need index params k1 and b ( #37721 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-18 15:44:31 +08:00
3f1614e9d9
enhance: add trace_id into segcore logs ( #37656 )
...
issue: #37655
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-18 10:20:30 +08:00
e4b6773d0a
fix: fix create text index dir conflict bug ( #37693 )
...
#37623
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-11-15 18:26:30 +08:00
65d3c6622a
enhance: Optimize GetChunkIDByOffset and add ut ( #37704 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-15 14:16:31 +08:00
d1596297d9
fix: Fix query failure with inverted index ( #37686 )
...
https://github.com/milvus-io/milvus/issues/37649
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-15 10:28:31 +08:00
1b4f7e3ac1
enhance: Add more expr ut for chunked segment ( #37600 )
...
related pr: #37570
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-14 18:40:32 +08:00
3389a6b500
enhance: support null in text match index ( #37517 )
...
#37508
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-13 11:08:29 +08:00
2d29dcd30c
enhance:refine group_strict_size parameter( #37482 ) ( #37483 )
...
related: #37482
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-11-12 09:56:28 +08:00
12951f0abb
enhance: rename tokenizer to analyzer and check analyzer params ( #37478 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-10 16:12:26 +08:00
40ba5a3414
fix: fix chunked segment term filter expression and add ut ( #37392 )
...
issue: https://github.com/milvus-io/milvus/issues/37143
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-07 11:04:19 -08:00
d67853fa89
feat: Tokenizer support build with params and clone for concurrency ( #37048 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
https://github.com/milvus-io/milvus/issues/36751
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-06 17:48:24 +08:00
625b6176cd
fix: Search for pk using raw data to reduce the overhead caused by views ( #37202 )
...
issue: #37152
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-05 20:36:24 +08:00
bd04cac4b3
fix: fix group by on chunked segment ( #37292 )
...
https://github.com/milvus-io/milvus/issues/37244
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-05 17:12:23 +08:00
63843dce33
fix: Fix conan gdal building problem ( #37338 )
...
issue:https://github.com/milvus-io/milvus/issues/27576
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-10-31 21:04:16 +08:00
67c4340565
feat: Geospatial Data Type and GIS Function Support for milvus server ( #35990 )
...
issue:https://github.com/milvus-io/milvus/issues/27576
# Main Goals
1. Create and describe collections with geospatial fields, enabling both
client and server to recognize and process geo fields.
2. Insert geospatial data as payload values in the insert binlog, and
print the values for verification.
3. Load segments containing geospatial data into memory.
4. Ensure query outputs can display geospatial data.
5. Support filtering on GIS functions for geospatial columns.
# Solution
1. **Add Type**: Modify the Milvus core by adding a Geospatial type in
both the C++ and Go code layers, defining the Geospatial data structure
and the corresponding interfaces.
2. **Dependency Libraries**: Introduce necessary geospatial data
processing libraries. In the C++ source code, use Conan package
management to include the GDAL library. In the Go source code, add the
go-geom library to the go.mod file.
3. **Protocol Interface**: Revise the Milvus protocol to provide
mechanisms for Geospatial message serialization and deserialization.
4. **Data Pipeline**: Facilitate interaction between the client and
proxy using the WKT format for geospatial data. The proxy will convert
all data into WKB format for downstream processing, providing column
data interfaces, segment encapsulation, segment loading, payload
writing, and cache block management.
5. **Query Operators**: Implement simple display and support for filter
queries. Initially, focus on filtering based on spatial relationships
for a single column of geospatial literal values, providing parsing and
execution for query expressions.
6. **Client Modification**: Enable the client to handle user input for
geospatial data and facilitate end-to-end testing.Check the modification
in pymilvus.
---------
Signed-off-by: tasty-gumi <1021989072@qq.com>
2024-10-31 20:58:20 +08:00
b8492498ac
fix: mask with valid data when preCheckOverflow ( #37221 )
...
#37175
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-31 10:44:26 +08:00
90948e9444
fix: add SearchOnSealed unit test and fix a bug ( #37241 )
...
issue: https://github.com/milvus-io/milvus/issues/37244
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-10-30 10:26:19 +08:00
44ddcb5a63
fix: not check has_value before get value in JSON ( #37128 )
...
https://github.com/milvus-io/milvus/issues/36236
also: https://github.com/milvus-io/milvus/issues/37113
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-25 17:19:28 +08:00
sthuang
Cai Yudong
zhagnlu
Bingyi Sun
Spade A
Zhen Ye
Buqian Zheng
Alexander Guzhva
Gao
Chun Han
smellthemoon
Patrick Weizhi Xu
Ted Xu
cqy123456
aoiasd
cai.zhang
zhenshan.cao
Hao Tan