Commit Graph

21286 Commits (7c5a8012cf8e04b426ceb21aa797e944873a28cb)

Author SHA1 Message Date
SimFG 7c5a8012cf
enhance: remove the collectionBasicInfo class in the proxy metacache (#37874)
/kind improvement
- issue: #37928

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-22 16:10:33 +08:00
yellow-shine b48f531e63
enhance: [pipeline] limit compute resource (#37888)
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-11-22 14:14:38 +08:00
Bingyi Sun 700a448a54
fix: Escape prefix before search in inverted index (#37925)
issue: https://github.com/milvus-io/milvus/issues/37912

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-22 14:10:33 +08:00
Bingyi Sun 06d73cf2e2
enhance: Remove raw tokenizer register. (#37886)
tantivy already register raw tokenizer by default

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-22 12:02:32 +08:00
congqixia b34bfb98a0
enhance: Refine Replica manager colle2Replicas secondary index (#37906)
Related to #37630

This PR add a new util coll2Replicas secondary index to reduce map
access & iteration while get replicas by collection

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-22 11:54:32 +08:00
congqixia 1ed686783f
enhance: Use `PrimaryKeys` to replace interface slice for segment delete (#37880)
Related to #35303

Reduce temporary memory usage for PK interface for segment delete.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-22 11:52:33 +08:00
SimFG 923a661dfe
enhance: filter the fields instead of create a new response obj (#37845)
/kind improvement

Here you only need to filter out the system fields, and you don’t need
to recreate a response, because recreating the response will cause this
part to be easily missed when adding fields later.

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-22 11:42:32 +08:00
SimFG cef55eb093
enhance: add retry termination log (#37894)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-22 11:40:33 +08:00
ThreadDao bc131a911c
test: add cases for gosdk v2 load release (#37831)
issue: #33419

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2024-11-22 10:54:32 +08:00
congqixia cfa1f1f141
enhance: Add thread watcher to provide actual thread num (#37905)
Related to #37904

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-22 10:20:33 +08:00
congqixia 92e6ee6285
enhance: Use load pool for `CreateTextIndex` (#37898)
Related to #37895

Only resolves the starving issue which caused goroutine leakage

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-22 10:06:33 +08:00
congqixia 83df725146
enhance: Revert default l0 forward policy to FilterByBF (#37867)
Related to #37767

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-22 09:00:33 +08:00
sre-ci-robot 84259040f7 Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-11-22 00:01:33 +00:00
SimFG 54aaeda63f
fix: add the request ctx for stream pipeline interface (#37835)
- issue: #37834

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-22 03:10:34 +08:00
sthuang 19572f5b06
enhance: RBAC new grant/revoke privilege (#37785)
issue: https://github.com/milvus-io/milvus/issues/37031
also fix issues: https://github.com/milvus-io/milvus/issues/37843,
https://github.com/milvus-io/milvus/issues/37842,
https://github.com/milvus-io/milvus/issues/37887

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-21 22:20:34 +08:00
wei liu 5f3601a6a5
fix: unstable integration test caused by paramtable.GetNodeID (#37909)
issue: #37908
cause paramtable is global single instance, which cause
paramtable.GetNodeID may return wrong server id in integration test.

This PR use node.GetNodeID to replace paramtable.GetNodeID

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-21 22:16:33 +08:00
Zhen Ye 1dc1a97e65
fix: use different thread pool for scheduler and merger (#37911)
issue: #37895
related pr: https://github.com/milvus-io/tantivy/pull/2

Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-21 21:34:33 +08:00
MimoN 3cb9511c57
fix: update webui filters (#37885)
issue: #36621
- fix the empty segment source
- remove http request timeout limit
- support more filters in seg and channel table

Signed-off-by: mimo.oyn <ning.ouyang@zilliz.com>
2024-11-21 20:04:33 +08:00
tinswzy e247ff9ee7
enhance: refine rootcoord/metatable interfaces to ensure that each method includes a ctx parameter (#37846)
issue: #35917
Before enhancing log trace information, it's necessary to pass the
context to the method entry point.
This PR first refine the rootcoord/metatable interfaces to ensure that
each method includes a ctx parameter.

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-11-21 19:30:32 +08:00
wei liu 965bda6e60
enhance: Add channel name to shard leader log in meta cache (#37856)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-21 19:24:31 +08:00
wei liu 97a44b62fd
fix: Data race in datacoord channel manager (#37866)
issue: #37865

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-21 19:00:32 +08:00
XuanYang-cn d7dcc752f1
enhance: Increase task capacity and clean illegal task (#37896)
1. taskQueueCapacity 256 is too small for production when we want to
re-write the entire collection

2. tasks should be cleaned when unable to recover, or the meta will
remain in etcd forever later.

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-21 18:48:32 +08:00
wei liu 0a440e0d38
fix: Prevent simultaneous balance of segments and channels (#37850)
issue: #33550
balance segment and balance segment execute at same time, which will
cause bounch of corner case.

This PR disable simultaneous balance of segments and channels

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-21 17:56:55 +08:00
wei liu 2159526e80
fix: Deadlock in compaction handler (#37868)
issue: #37803

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-21 17:26:46 +08:00
XuanYang-cn 70e6a00ba1
fix: Replace outer lock with concurrent map (#37817)
See also: #37493

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-21 16:08:33 +08:00
wei liu 7817ea6b8b
enhance: Remove unnecessary stack trace in error (#37816)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-21 15:10:32 +08:00
cai.zhang c07f056b17
fix: Use the ID to retrieve the real name when collectionName is empty (#37859)
issue: #36989

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-21 14:28:32 +08:00
wei liu b983ef9fca
fix: Channel may be released after balance (#37862)
issue: #37830
casue dist handler doesn't set channel's version, so if channel checker
try to dedup channel, it may release the new delegator after balance
finished.

this PR fix the way to set proper version for channel.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-21 10:40:31 +08:00
Zhen Ye f3a36f8a29
fix: use global pool but not dedicated pool for every index (#37852)
issue: #37851

- make a global thread pool at tantivy temporally.
- set 1 but not 4 threads for inverted text index.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-20 20:44:32 +08:00
zhikunyao 714421997c
enhance: [skip e2e] update mergify to support macos 13 job (#37844)
Signed-off-by: Zhikun Yao <zhikun.yao@zilliz.com>
2024-11-20 16:52:31 +08:00
presburger c73ca14a9b
enhence: update git workflows macos (#37819)
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-11-20 14:51:55 +08:00
Buqian Zheng 511edd29fd
enhance: disallow get raw vector data of a BM25 Function output field (#37800)
issue: https://github.com/milvus-io/milvus/issues/35853

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-11-20 14:22:30 +08:00
sre-ci-robot 7ba85504ba
[automated] Bump milvus version to v2.4.16 (#37795)
Bump milvus version to v2.4.16
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-11-20 11:46:37 +08:00
congqixia b8d31ebed8
enhance: Remove unnecessary segment clone updating dist (#37797)
Related to #37630

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-20 11:26:31 +08:00
nish112022 484c6b5c44
feat: Added code for Internal-tls (#36865)
issue : https://github.com/milvus-io/milvus/issues/36864

I have a few questions regarding my approach.I will consolidate them
here for feedback and review.Thanks

---------

Signed-off-by: Nischay Yadav <nischay.yadav@ibm.com>
Signed-off-by: Nischay <Nischay.Yadav@ibm.com>
2024-11-20 06:00:32 +08:00
congqixia 83c902992c
enhance: Prevent generate "null" search params (#37812)
Preventing generating null search params in restful search request

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-19 19:54:32 +08:00
yihao.dai b3fc53015d
fix: Avoid memory copy in RESTful search and query (#37674)
Custom `jsonRender` that encodes JSON data directly into the response
stream, it uses less memory since it does not buffer the entire JSON
structure before sending it, unlike `c.JSON` in `HTTPReturn`, which
serializes the JSON fully in memory before writing it to the response.
Benchmark testing shows that, using the custom render incurs no
performance loss and reduces memory consumption by nearly 50%.

issue: https://github.com/milvus-io/milvus/issues/37671

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-19 17:24:31 +08:00
SimFG 3bbbf3cd95
enhance: improve the lock key of the create/drop collection task (#37759)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-19 15:50:32 +08:00
cqy123456 8216345b07
enhance: reduce copy of bitset and id conversion of brurtforce search (#37675)
issue: https://github.com/milvus-io/milvus/issues/37798

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-11-19 15:48:40 +08:00
yihao.dai b6612e02b4
enhance: Reduce GetIndexInfos calls (#37695)
Batch `GetIndexInfos` calls for segments to reduce RPC calls.

issue: https://github.com/milvus-io/milvus/issues/37634

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-19 14:24:31 +08:00
nico 866d512f6f
test: update test cases and sdk version (#37774)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-11-19 14:14:32 +08:00
cai.zhang dae4160466
enhance: Whether to enable mergeSort mode when performing mixCompaction (#37664)
issue: #37579

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-19 11:28:31 +08:00
Xiaofan 33bfb25c73
enhance: refine meta cache log and logic (#37318)
related to #36989
add more logs in proxy meta cache and make it clearer

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-11-19 10:50:33 +08:00
congqixia 175f49ea60
enhance: [GoSDK] Use generic column base and add unit tests (#37768)
Related to #31293

Previous column is written using go before 1.18, which does not support
generic type. After migrating client pkg to main repo, generic base
could be used to reduce lots of duplicated code.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-19 10:40:31 +08:00
congqixia c79fbd5eab
fix: Load l0 delta for growings when using `RemoteLoad` (#37771)
Related to #37574

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-19 10:38:31 +08:00
wei liu 113c72c02e
fix: [skip e2e] unstable ut TestResourceManager (#37761)
issue: #37760

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-19 10:24:31 +08:00
Bingyi Sun 6b82320953
fix: Fix using wrong upperbound when searching by pk (#37769)
issue: https://github.com/milvus-io/milvus/issues/37649

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-19 10:16:31 +08:00
wei liu fceff6ed7f
enhance: Optimize param cost in search (#37738)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-19 10:14:40 +08:00
congqixia 6d86b9022e
enhance: Provide secondary index critria when filter leaderview (#37777)
Related to #37630

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-19 10:12:30 +08:00
smellthemoon 3d28d99411
fix: to use the correct offset in span (#37780)
#37734

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-18 21:56:30 +08:00