Commit Graph

9805 Commits (2319018fcbde35e4aeb8e0679d7c993016da5a56)

Author SHA1 Message Date
cai.zhang 2319018fcb
fix: Check if the dynamic fields contain any static fields (#38025)
issue: #38024

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-12-01 18:06:37 +08:00
wei liu 108434969f
enhance: Add collection id to search request count metrics (#38069)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-29 17:28:38 +08:00
congqixia 61f73d552f
fix: Return thread watcher goroutine after closed (#38091)
Resolves #38090

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-29 15:10:36 +08:00
jaime a9080d8d96
enhance: the actual number of databases should equal the config value (#38006)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-29 14:44:36 +08:00
smellthemoon 48ed586167
enhance: pre allocate space to prevent slice grow (#38065)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-29 14:26:43 +08:00
congqixia 9d3035c8ab
enhance: Fix conflict complication error (#38087)
Related to #37957 #37989

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-29 12:00:36 +08:00
cai.zhang 5e152767a3
fix: Handle the error of the compaction queue being full (#37989)
issue: #37988

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-29 11:10:36 +08:00
Zhen Ye c6dcef7b84
enhance: move segcore codes of segment into one package (#37722)
issue: #33285

- move most cgo opeartions related to search/query into segcore package
for reusing for streamingnode.
- add go unittest for segcore operations.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-29 10:22:36 +08:00
sre-ci-robot 0894ed0016
[automated] Update Knowhere Commit (#38082)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-11-29 01:08:36 +08:00
Bingyi Sun e6af806a0d
enhance: optimize self defined rust error (#37975)
Prepare for issue: https://github.com/milvus-io/milvus/issues/37930

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-28 20:30:36 +08:00
SimFG 302650ae0e
fix: use the default partition for the limit quota when the request partition name is empty (#38005)
- issue: #37685

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-27 11:00:36 +08:00
SimFG 49ee46ec1d
enhance: support to config the default db properties (#38035)
- issue: #38034

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-27 10:04:34 +08:00
SimFG 971b4f17ae
fix: add the db information in the dml message (#37969)
- issue: #37966

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-27 10:02:35 +08:00
tinswzy 1dbb6cd7cb
enhance: refine the datacoord meta related interfaces (#37957)
issue: #35917 
This PR refines the meta-related APIs in datacoord to allow the ctx to
be passed down to the catalog operation interfaces

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-11-26 19:46:34 +08:00
SimFG 2208b7c2ef
fix: the too long default root password does not take effect (#37983)
- issue: #36987

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-26 17:24:35 +08:00
Zhen Ye 2b4f211d84
enhance: add switch for local rpc enabled (#37985)
issue: #33285

- Add switch for local rpc

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-26 17:00:54 +08:00
aoiasd db3c002788
fix:wrong warning log when idf oracle find L0 segment lacks (#37873)
relate: https://github.com/milvus-io/milvus/issues/35853

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-26 16:26:41 +08:00
congqixia cb6542339e
enhance: Mark cgo thread with tag name (#38000)
Related to #37999

This PR add `SetThreadName` API for marking cgo thread and utilize it
when initializing cgo worker.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-26 11:22:35 +08:00
tinswzy e76802f910
enhance: refine querycoord meta/catalog related interfaces to ensure that each method includes a ctx parameter (#37916)
issue: #35917 
This PR refine the querycoord meta related interfaces to ensure that
each method includes a ctx parameter.

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-11-25 11:14:34 +08:00
Zhen Ye fbb68ca370
enhance: make all index operation async scheduled by tokio (#37946)
issue: #37851
related pr: https://github.com/milvus-io/tantivy/pull/3

Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-25 10:12:34 +08:00
sre-ci-robot ed73dfca3f
[automated] Update Knowhere Commit (#37965)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-11-25 01:28:32 +08:00
Yinzuo Jiang 5a06faca39
feat: fp32 vector to fp16/bf16 vector conversion for RESTful API (#37556)
RESTful API. The influenced API are as follows:

- Handler. insert
- HandlerV1. insert/upsert
- HandlerV2. insert/upsert/search

We do not modify search API in Handler/HandlerV1 because they do not
support fp16/bf16 vectors.

module github.com/milvus-io/milvus/pkg:

Add `Float32ArrayToBFloat16Bytes()`, `Float32ArrayToFloat16Bytes()` and
`Float32ArrayToBytes()`. These method will be used in GoSDK in the
future.

issue: #37448

Signed-off-by: Yinzuo Jiang <yinzuo.jiang@zilliz.com>
Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
2024-11-24 17:46:33 +08:00
zhagnlu 62af24c1a1
fix: change search latency metric from us unit to ms unit (#37806)
#37805

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-11-24 17:26:33 +08:00
congqixia 1e76d2b9f1
fix: Use correct policy merging growing&l0 and add unit tests (#37950)
Related to #37574

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-24 17:10:33 +08:00
Zhen Ye b9a10a2f68
enhance: remove the rpc layer of coordinator when enabling standalone or mixcoord (#37815)
issue: #37764

- add a local client to call local server directly for
querycoord/rootcoord/datacoord.
- enable local client if milvus is running mixcoord or standalone mode.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-23 21:36:33 +08:00
sthuang bd0811434b
fix: fix grant v2 service (#37945)
issue: https://github.com/milvus-io/milvus/issues/37031

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-22 19:42:32 +08:00
smellthemoon df08d91c0f
enhance: error msg (#37901)
#37900

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-22 18:40:32 +08:00
jaime 7bbfe86bcd
enhance: add list index and segment index retrieval API for WebUI (#37861)
issue: #36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-22 16:58:34 +08:00
SimFG 7c5a8012cf
enhance: remove the collectionBasicInfo class in the proxy metacache (#37874)
/kind improvement
- issue: #37928

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-22 16:10:33 +08:00
Bingyi Sun 700a448a54
fix: Escape prefix before search in inverted index (#37925)
issue: https://github.com/milvus-io/milvus/issues/37912

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-22 14:10:33 +08:00
Bingyi Sun 06d73cf2e2
enhance: Remove raw tokenizer register. (#37886)
tantivy already register raw tokenizer by default

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-22 12:02:32 +08:00
congqixia b34bfb98a0
enhance: Refine Replica manager colle2Replicas secondary index (#37906)
Related to #37630

This PR add a new util coll2Replicas secondary index to reduce map
access & iteration while get replicas by collection

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-22 11:54:32 +08:00
congqixia 1ed686783f
enhance: Use `PrimaryKeys` to replace interface slice for segment delete (#37880)
Related to #35303

Reduce temporary memory usage for PK interface for segment delete.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-22 11:52:33 +08:00
SimFG 923a661dfe
enhance: filter the fields instead of create a new response obj (#37845)
/kind improvement

Here you only need to filter out the system fields, and you don’t need
to recreate a response, because recreating the response will cause this
part to be easily missed when adding fields later.

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-22 11:42:32 +08:00
congqixia cfa1f1f141
enhance: Add thread watcher to provide actual thread num (#37905)
Related to #37904

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-22 10:20:33 +08:00
congqixia 92e6ee6285
enhance: Use load pool for `CreateTextIndex` (#37898)
Related to #37895

Only resolves the starving issue which caused goroutine leakage

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-22 10:06:33 +08:00
SimFG 54aaeda63f
fix: add the request ctx for stream pipeline interface (#37835)
- issue: #37834

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-22 03:10:34 +08:00
sthuang 19572f5b06
enhance: RBAC new grant/revoke privilege (#37785)
issue: https://github.com/milvus-io/milvus/issues/37031
also fix issues: https://github.com/milvus-io/milvus/issues/37843,
https://github.com/milvus-io/milvus/issues/37842,
https://github.com/milvus-io/milvus/issues/37887

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-21 22:20:34 +08:00
wei liu 5f3601a6a5
fix: unstable integration test caused by paramtable.GetNodeID (#37909)
issue: #37908
cause paramtable is global single instance, which cause
paramtable.GetNodeID may return wrong server id in integration test.

This PR use node.GetNodeID to replace paramtable.GetNodeID

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-21 22:16:33 +08:00
Zhen Ye 1dc1a97e65
fix: use different thread pool for scheduler and merger (#37911)
issue: #37895
related pr: https://github.com/milvus-io/tantivy/pull/2

Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-21 21:34:33 +08:00
MimoN 3cb9511c57
fix: update webui filters (#37885)
issue: #36621
- fix the empty segment source
- remove http request timeout limit
- support more filters in seg and channel table

Signed-off-by: mimo.oyn <ning.ouyang@zilliz.com>
2024-11-21 20:04:33 +08:00
tinswzy e247ff9ee7
enhance: refine rootcoord/metatable interfaces to ensure that each method includes a ctx parameter (#37846)
issue: #35917
Before enhancing log trace information, it's necessary to pass the
context to the method entry point.
This PR first refine the rootcoord/metatable interfaces to ensure that
each method includes a ctx parameter.

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-11-21 19:30:32 +08:00
wei liu 965bda6e60
enhance: Add channel name to shard leader log in meta cache (#37856)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-21 19:24:31 +08:00
wei liu 97a44b62fd
fix: Data race in datacoord channel manager (#37866)
issue: #37865

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-21 19:00:32 +08:00
XuanYang-cn d7dcc752f1
enhance: Increase task capacity and clean illegal task (#37896)
1. taskQueueCapacity 256 is too small for production when we want to
re-write the entire collection

2. tasks should be cleaned when unable to recover, or the meta will
remain in etcd forever later.

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-21 18:48:32 +08:00
wei liu 0a440e0d38
fix: Prevent simultaneous balance of segments and channels (#37850)
issue: #33550
balance segment and balance segment execute at same time, which will
cause bounch of corner case.

This PR disable simultaneous balance of segments and channels

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-21 17:56:55 +08:00
wei liu 2159526e80
fix: Deadlock in compaction handler (#37868)
issue: #37803

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-21 17:26:46 +08:00
XuanYang-cn 70e6a00ba1
fix: Replace outer lock with concurrent map (#37817)
See also: #37493

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-21 16:08:33 +08:00
wei liu 7817ea6b8b
enhance: Remove unnecessary stack trace in error (#37816)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-21 15:10:32 +08:00
cai.zhang c07f056b17
fix: Use the ID to retrieve the real name when collectionName is empty (#37859)
issue: #36989

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-21 14:28:32 +08:00