Commit Graph

3066 Commits (9b6dd23f8ede46cf874c97a1aacf2ce04ca35404)

Author SHA1 Message Date
cai.zhang aed3b94b5d
enhance: Refine error message for contains array (#37383)
issue: #36221

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-07 10:44:25 +08:00
wei liu 7cfd609ebc
fix: [skip e2e]unstable integration test TestNodeDownOnSingleReplica (#37480)
issue: #37289
cause pr #37116 introduce retry on get shard leader, which make search
won't fail during query node down.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-06 17:08:21 -08:00
nico cf66fbee8a
test: update test cases (#37215)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-11-06 17:50:25 +08:00
aoiasd d67853fa89
feat: Tokenizer support build with params and clone for concurrency (#37048)
relate: https://github.com/milvus-io/milvus/issues/35853
https://github.com/milvus-io/milvus/issues/36751

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-06 17:48:24 +08:00
congqixia c83b93946e
enhance: [GoSDK] Add alter collection API & expose options (#37365)
Related to #31293

This PR:

- Add `AlterCollection` API for collection property modification
- Expose hidden or missing option methods

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-06 14:36:25 +08:00
zhuwenxing 0fc6c634b0
test: fix tokenizer and monkey patch faker function (#37119)
/kind improvement

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Signed-off-by: zhuwenxing <wxzhuyeah@gmail.com>
2024-11-05 08:42:23 +08:00
cai.zhang ba9f36ba33
enhance: Increase the concurrency of index tasks in CI (#37335)
#37096

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-04 12:04:23 +08:00
foxspy eaf86f7649
fix: optimize invalid datatype error msg (#37376)
issue: #37151

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-11-04 11:46:28 +08:00
cai.zhang 50de122dc7
enhance: Rename textmatch to text_match (#37290)
issue: #36672

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-03 18:40:27 +08:00
congqixia be71b98146
enhance: Fix case error msg after knowhere upgrade (#37339)
Previous PR: #37315

Error message updated after knowhere behavior change, this PR update
corresponding test error message.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-01 14:00:22 +08:00
wei liu f190e5d802
fix: [skip e2e] TestNodeDownOnSingleReplica has unstable result (#37288)
issue: #37289
those test case use search to verify replica's status, but if the search
gap is 1s, the node down's effect may be fixed up by balance.

This PR remove the 1 second gap between search operation.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-01 10:50:21 +08:00
zhuwenxing 247f75180f
test: add restful cases for full text search and some minor fix (#37148)
/kind improvement

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-31 21:18:23 +08:00
zhuwenxing d24970c090
test: fix potential unstable cases of full-text search (#37328)
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-31 21:12:21 +08:00
zhenshan.cao 63843dce33
fix: Fix conan gdal building problem (#37338)
issue:https://github.com/milvus-io/milvus/issues/27576

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-10-31 21:04:16 +08:00
Hao Tan 67c4340565
feat: Geospatial Data Type and GIS Function Support for milvus server (#35990)
issue:https://github.com/milvus-io/milvus/issues/27576

# Main Goals
1. Create and describe collections with geospatial fields, enabling both
client and server to recognize and process geo fields.
2. Insert geospatial data as payload values in the insert binlog, and
print the values for verification.
3. Load segments containing geospatial data into memory.
4. Ensure query outputs can display geospatial data.
5. Support filtering on GIS functions for geospatial columns.

# Solution
1. **Add Type**: Modify the Milvus core by adding a Geospatial type in
both the C++ and Go code layers, defining the Geospatial data structure
and the corresponding interfaces.
2. **Dependency Libraries**: Introduce necessary geospatial data
processing libraries. In the C++ source code, use Conan package
management to include the GDAL library. In the Go source code, add the
go-geom library to the go.mod file.
3. **Protocol Interface**: Revise the Milvus protocol to provide
mechanisms for Geospatial message serialization and deserialization.
4. **Data Pipeline**: Facilitate interaction between the client and
proxy using the WKT format for geospatial data. The proxy will convert
all data into WKB format for downstream processing, providing column
data interfaces, segment encapsulation, segment loading, payload
writing, and cache block management.
5. **Query Operators**: Implement simple display and support for filter
queries. Initially, focus on filtering based on spatial relationships
for a single column of geospatial literal values, providing parsing and
execution for query expressions.
6. **Client Modification**: Enable the client to handle user input for
geospatial data and facilitate end-to-end testing.Check the modification
in pymilvus.

---------

Signed-off-by: tasty-gumi <1021989072@qq.com>
2024-10-31 20:58:20 +08:00
cai.zhang 2ef6cbbf59
feat: The expression supports filling elements through templates (#37033)
issue: #36672

The expression supports filling elements through templates, which helps
to reduce the overhead of parsing the elements.

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-10-31 14:20:22 +08:00
zhuwenxing 6e37372619
test: update checker (#37275)
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-31 09:50:20 +08:00
yanliang567 3a3404658e
test: Add a test for iterator enhancement (#37296)
related issue: https://github.com/milvus-io/milvus/issues/37084

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-10-30 16:54:20 +08:00
Zhen Ye 889434691c
enhance: enable asan for e2e (#37149)
issue: #35854

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-10-29 14:14:24 +08:00
zhuwenxing 4c108b1564
test: update jieba tokenizer in test (#37199)
/kind improvement

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-28 19:22:22 +08:00
zhuwenxing cdee149191
test: fix testcases for verification after chaos (#37153)
/kind improvement

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-28 10:33:29 +08:00
zhuwenxing c8dd665bf6
test: supplementing case for text match (#36693)
/kind improvement

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-28 10:31:40 +08:00
foxspy d7b2ffe5aa
enhance: add an unify vector index config checker (#36844)
issue: #34298

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-10-28 10:11:37 +08:00
yihao.dai ed37c27bda
fix: Fix collection leak in querynode (#37061)
Unref the removed L0 segment count.

issue: https://github.com/milvus-io/milvus/issues/36918

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-25 19:59:29 +08:00
Yinzuo Jiang 3628593d20
feat: Implement custom function module in milvus expr (#36560)
OSPP 2024 project:
https://summer-ospp.ac.cn/org/prodetail/247410235?list=org&navpage=org

Solutions:

- parser (planparserv2)
    - add CallExpr in planparserv2/Plan.g4
    - update parser_visitor and show_visitor
- grpc protobuf
    - add CallExpr in plan.proto
- execution (`core/src/exec`)
- add `CallExpr` `ValueExpr` and `ColumnExpr` (both logical and
physical) for function call and function parameters
- function factory (`core/src/exec/expression/function`)
    - create a global hashmap when starting milvus (see server.go)
- the global hashmap stores function signatures and their function
pointers, the CallExpr in execution engine can get the function pointer
by function signature.
- custom functions
    - empty(string)
    - starts_with(string, string)
- add cpp/go unittests and E2E tests

closes: #36559

Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
2024-10-25 15:25:30 +08:00
Buqian Zheng 088d5d7d76
fix: optimize BM25 err message (#37074)
issue: https://github.com/milvus-io/milvus/issues/37022

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-10-25 14:35:45 +08:00
zhuwenxing ac2858d418
test: add full text search checker in test (#37122)
/kind improvement

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-25 14:09:29 +08:00
qixuan 80aa9ab4d6
test: Add insert and upsert related cases for null and default value support (#37088)
issue: #36129

---------

Signed-off-by: qixuan <673771573@qq.com>
2024-10-25 11:03:29 +08:00
binbin c285853de8
test: Add test cases about expr for null and default support (#37121)
issue: #36129

Signed-off-by: binbin lv <binbin.lv@zilliz.com>
2024-10-25 11:01:30 +08:00
smellthemoon 2b3f5bec07
fix: panic when create index on all none data (#37046)
#37045

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-24 17:09:28 +08:00
sre-ci-robot 53836f320a
[automated] Update Pytest image changes (#37067)
Update Pytest image changes
See changes:
3b024f9b36
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-10-24 16:53:28 +08:00
yanliang567 0d376f1a8f
test: Add query iterator test and seperate a new test file (#37085)
related issue: #37084

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-10-24 11:47:29 +08:00
jaime 4746f47282
feat: management WebUI homepage (#36822)
issue: #36784
1. Implement an embedded web server for WebUI access.  
2. Complete the homepage development.

Home page demo:
<img width="2177" alt="iShot_2024-10-10_17 57 34"
src="https://github.com/user-attachments/assets/38539917-ce09-4e54-a5b5-7f4f7eaac353">

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-23 11:29:28 +08:00
zhuwenxing 3b024f9b36
test: Add full-text search test cases (#36998)
/kind improvement

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-23 09:51:27 +08:00
zhuwenxing 80d48f1e53
test: add text match checker in test (#37052)
/kind improvement

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-23 09:49:27 +08:00
qixuan 36147b10f5
test: Add insert and upsert related cases for null and default value support (#36932)
issue: #36129

Signed-off-by: qixuan <673771573@qq.com>
2024-10-22 10:13:27 +08:00
zhuwenxing ef923958bb
test: add restful param check testcases (#36931)
/kind improvement

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-19 19:13:25 +08:00
yanliang567 d17135d3bc
test: Add alias test with rename collection (#36978)
related issue: #36963 
1. add alias tests with rename collection for issue #36963
2. update some test levels
3. fix some xfail tests

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-10-18 16:27:24 +08:00
smellthemoon eb3e4583ec
enhance: all op(Null) is false in expr (#35527)
#31728

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-17 21:14:30 +08:00
wt 97ff012c67
test: check query response data (#36952)
Signed-off-by: wangting0128 <ting.wang@zilliz.com>
2024-10-17 19:13:25 +08:00
congqixia e5948bd039
enhance: [GoSDK] Use variadic params for options (#36912)
Use variadic parameter function for options make client options easier
to use.

Related to #31293

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-16 19:33:23 +08:00
smellthemoon 44d80c1355
fix: not return err if consistencyLevel is not set to a valid value (#36714)
https://github.com/milvus-io/milvus/issues/36444

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-16 13:23:22 +08:00
nico 2291d0c9f5
test: update test cases (#36878)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-10-15 19:13:24 +08:00
yanliang567 0ec92513a5
test: Update Diskann supporting group by (#36853)
related issue: #36813 

milvus diskann does not support group_search before, it supports now.

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-10-15 10:35:23 +08:00
ThreadDao d566b0ceff
test: add stats task feature cases and DefaultVectorSearchParams (#36768)
issue: #36767

---------

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2024-10-14 19:15:23 +08:00
nico 937ebec2ce
test: update test cases (#36841)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-10-14 19:13:23 +08:00
sre-ci-robot af2da8d8d6
[automated] Update Pytest image changes (#36845)
Update Pytest image changes
See changes:
c96bbe19ba
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-10-14 19:11:23 +08:00
yanliang567 c96bbe19ba
test: Add upsert in rows tests (#36820)
related issue: #36710

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-10-14 16:41:22 +08:00
yanliang567 621dbc9107
test: Add json key in [] test to verify issue 36718 (#36738)
related issue: #36718

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-10-11 14:29:21 +08:00
aoiasd db34572c56
feat: support load and query with bm25 metric (#36071)
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-11 10:23:20 +08:00