f3a36f8a29
fix: use global pool but not dedicated pool for every index ( #37852 )
...
issue: #37851
- make a global thread pool at tantivy temporally.
- set 1 but not 4 threads for inverted text index.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-20 20:44:32 +08:00
8216345b07
enhance: reduce copy of bitset and id conversion of brurtforce search ( #37675 )
...
issue: https://github.com/milvus-io/milvus/issues/37798
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-11-19 15:48:40 +08:00
6b82320953
fix: Fix using wrong upperbound when searching by pk ( #37769 )
...
issue: https://github.com/milvus-io/milvus/issues/37649
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-19 10:16:31 +08:00
3d28d99411
fix: to use the correct offset in span ( #37780 )
...
#37734
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-18 21:56:30 +08:00
16e206167c
enhance: analyzer length filter max should be close interval instead open interval ( #37770 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-18 19:30:31 +08:00
e9391acf80
fix: bm25 brute force search need index params k1 and b ( #37721 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-18 15:44:31 +08:00
3f1614e9d9
enhance: add trace_id into segcore logs ( #37656 )
...
issue: #37655
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-18 10:20:30 +08:00
3b5a0df159
enhance: Optimize chinese analyzer and support CnAlphaNumFilter ( #37727 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-16 10:28:30 +08:00
0ba868ae64
enhance: update knowhere version ( #37730 )
...
release note draft :
https://github.com/zilliztech/knowhere/releases/tag/v2.5.0
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-11-16 10:08:30 +08:00
7999367c0c
fix: use not retried err when get wrong parameter ( #37707 )
...
#37508
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-15 19:14:30 +08:00
e4b6773d0a
fix: fix create text index dir conflict bug ( #37693 )
...
#37623
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-11-15 18:26:30 +08:00
65d3c6622a
enhance: Optimize GetChunkIDByOffset and add ut ( #37704 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-15 14:16:31 +08:00
d1596297d9
fix: Fix query failure with inverted index ( #37686 )
...
https://github.com/milvus-io/milvus/issues/37649
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-15 10:28:31 +08:00
5ae347aba0
enhance: update knowhere version ( #37688 )
...
issue: #37665 #37631 #37620 #37587 #36906
knowhere has add default nlist value, so some invalid param test ut with
no nlist param will be valid.
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-11-15 10:10:31 +08:00
1b4f7e3ac1
enhance: Add more expr ut for chunked segment ( #37600 )
...
related pr: #37570
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-14 18:40:32 +08:00
993051bb49
fix: brute force bm25 search lack avgdl param ( #37650 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-14 14:58:31 +08:00
0565300b7f
fix: Sparse to use CC index as growing/temp index ( #37591 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-11-14 10:54:31 +08:00
1c5b5e1e3d
feat: Add chinese and english analyzer with refactor jieba tokenizer ( #37494 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-14 10:34:31 +08:00
cf883b114e
enhance: update knowhere version ( #37510 )
...
issue: #36925
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-11-13 16:36:27 +08:00
3389a6b500
enhance: support null in text match index ( #37517 )
...
#37508
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-13 11:08:29 +08:00
3c225e5c94
fix: data race when using fields_ ( #37612 )
...
issue: #37609
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-13 04:06:30 +08:00
2d29dcd30c
enhance:refine group_strict_size parameter( #37482 ) ( #37483 )
...
related: #37482
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-11-12 09:56:28 +08:00
c1eccce2fa
enhance: enable multiple chunked segment by default ( #37570 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-12 09:20:28 +08:00
12951f0abb
enhance: rename tokenizer to analyzer and check analyzer params ( #37478 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-10 16:12:26 +08:00
40ba5a3414
fix: fix chunked segment term filter expression and add ut ( #37392 )
...
issue: https://github.com/milvus-io/milvus/issues/37143
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-07 11:04:19 -08:00
5310d3469f
fix: Escape brace of dumped JSON for index err message ( #37504 )
...
Related to #37503
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-07 18:58:25 +08:00
9b6dd23f8e
fix: wrong path spelling when use rootpath in segcore ( #37453 )
...
#36532
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-07 11:22:25 +08:00
d67853fa89
feat: Tokenizer support build with params and clone for concurrency ( #37048 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
https://github.com/milvus-io/milvus/issues/36751
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-06 17:48:24 +08:00
625b6176cd
fix: Search for pk using raw data to reduce the overhead caused by views ( #37202 )
...
issue: #37152
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-05 20:36:24 +08:00
bd04cac4b3
fix: fix group by on chunked segment ( #37292 )
...
https://github.com/milvus-io/milvus/issues/37244
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-05 17:12:23 +08:00
9a0e1c82bc
fix: repeated error code in milvus and segcore ( #37359 )
...
issue: #37357
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-05 16:28:23 +08:00
0c4321cf57
fix: crash when startup if the milvus volume is on-operation concurrently ( #37312 )
...
issue: #37311
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-04 14:50:23 +08:00
c27f477b6c
enhance: Update Knowhere version ( #37333 )
...
issue: #37269
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-11-04 11:56:31 +08:00
51cb2fbf97
fix: parse fail in empty json ( #37294 )
...
#37200
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-03 19:00:21 +08:00
5aad000a93
enhance: [skip ci] remove tools/core_gen after #35251 ( #36306 )
...
`tools/core_gen` python scripts are useless after
https://github.com/milvus-io/milvus/pull/35251
fixes : #36305
Signed-off-by: Yinzuo Jiang <yinzuo.jiang@zilliz.com>
Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
2024-11-01 13:08:21 +08:00
cd2655c861
fix: fix wrong method is called to fetch variable valid data ( #37304 )
...
issue: https://github.com/milvus-io/milvus/issues/37147
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-01 01:52:20 +08:00
63843dce33
fix: Fix conan gdal building problem ( #37338 )
...
issue:https://github.com/milvus-io/milvus/issues/27576
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-10-31 21:04:16 +08:00
67c4340565
feat: Geospatial Data Type and GIS Function Support for milvus server ( #35990 )
...
issue:https://github.com/milvus-io/milvus/issues/27576
# Main Goals
1. Create and describe collections with geospatial fields, enabling both
client and server to recognize and process geo fields.
2. Insert geospatial data as payload values in the insert binlog, and
print the values for verification.
3. Load segments containing geospatial data into memory.
4. Ensure query outputs can display geospatial data.
5. Support filtering on GIS functions for geospatial columns.
# Solution
1. **Add Type**: Modify the Milvus core by adding a Geospatial type in
both the C++ and Go code layers, defining the Geospatial data structure
and the corresponding interfaces.
2. **Dependency Libraries**: Introduce necessary geospatial data
processing libraries. In the C++ source code, use Conan package
management to include the GDAL library. In the Go source code, add the
go-geom library to the go.mod file.
3. **Protocol Interface**: Revise the Milvus protocol to provide
mechanisms for Geospatial message serialization and deserialization.
4. **Data Pipeline**: Facilitate interaction between the client and
proxy using the WKT format for geospatial data. The proxy will convert
all data into WKB format for downstream processing, providing column
data interfaces, segment encapsulation, segment loading, payload
writing, and cache block management.
5. **Query Operators**: Implement simple display and support for filter
queries. Initially, focus on filtering based on spatial relationships
for a single column of geospatial literal values, providing parsing and
execution for query expressions.
6. **Client Modification**: Enable the client to handle user input for
geospatial data and facilitate end-to-end testing.Check the modification
in pymilvus.
---------
Signed-off-by: tasty-gumi <1021989072@qq.com>
2024-10-31 20:58:20 +08:00
4bac2eb13e
enhance: Update Knowhere version ( #37315 )
...
Signed-off-by: Li Liu <li.liu@zilliz.com>
2024-10-31 17:24:20 +08:00
b8492498ac
fix: mask with valid data when preCheckOverflow ( #37221 )
...
#37175
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-31 10:44:26 +08:00
2092dc0ba1
enhance: reserve vector space to reduce reallocate cost in Views() and StringViews() ( #37182 )
...
issue: #37152
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-10-31 10:02:21 +08:00
7961568223
fix: Rectify `OffsetOrderedArray` contain logic ( #37305 )
...
Related to #36887
Remove non-hit pk delete record logic does not work since
`insert_record_.contain` does not work due to logic problem.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-30 21:26:19 +08:00
90948e9444
fix: add SearchOnSealed unit test and fix a bug ( #37241 )
...
issue: https://github.com/milvus-io/milvus/issues/37244
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-10-30 10:26:19 +08:00
86687bd8ed
enhance: Refine code for get_deleted_bitmap ( #36819 )
...
issue: #33744
Check whether the PK is truly sorted in the debug model.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-10-28 15:19:30 +08:00
b81f162f6a
fix: fix several bugs and refactor some codes related with chunked segment ( #37168 )
...
issue: https://github.com/milvus-io/milvus/issues/37147
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-10-28 14:17:30 +08:00
d7b2ffe5aa
enhance: add an unify vector index config checker ( #36844 )
...
issue: #34298
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-10-28 10:11:37 +08:00
a2f0092e39
fix: check sparse float before calling get_dim ( #37145 )
...
https://github.com/milvus-io/milvus/issues/37146
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-10-26 16:25:29 +08:00
44ddcb5a63
fix: not check has_value before get value in JSON ( #37128 )
...
https://github.com/milvus-io/milvus/issues/36236
also: https://github.com/milvus-io/milvus/issues/37113
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-25 17:19:28 +08:00
ff0b7ea0ef
enhance: build interim index for mmapped vector in ChunkedSealedSegment ( #36993 )
...
issue:https://github.com/milvus-io/milvus/issues/36392
related pr: https://github.com/milvus-io/milvus/pull/36391
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-10-25 15:55:28 +08:00
3628593d20
feat: Implement custom function module in milvus expr ( #36560 )
...
OSPP 2024 project:
https://summer-ospp.ac.cn/org/prodetail/247410235?list=org&navpage=org
Solutions:
- parser (planparserv2)
- add CallExpr in planparserv2/Plan.g4
- update parser_visitor and show_visitor
- grpc protobuf
- add CallExpr in plan.proto
- execution (`core/src/exec`)
- add `CallExpr` `ValueExpr` and `ColumnExpr` (both logical and
physical) for function call and function parameters
- function factory (`core/src/exec/expression/function`)
- create a global hashmap when starting milvus (see server.go)
- the global hashmap stores function signatures and their function
pointers, the CallExpr in execution engine can get the function pointer
by function signature.
- custom functions
- empty(string)
- starts_with(string, string)
- add cpp/go unittests and E2E tests
closes : #36559
Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
2024-10-25 15:25:30 +08:00
Zhen Ye
cqy123456
Bingyi Sun
smellthemoon
aoiasd
foxspy
zhagnlu
Buqian Zheng
Chun Han
congqixia
cai.zhang
Yinzuo Jiang
zhenshan.cao
Hao Tan
liliu-z
Gao
Yinzuo Jiang