Commit Graph

903 Commits (89d95901c50ed595ba613b30a3169f4e36257356)

Author SHA1 Message Date
wei liu 9d4ce6e581
enhance: Add restful api for devops to execute rolling upgrade (#29998) (#31846)
issue: #29261
pr: #29998
This PR Add restful api for devops to execute rolling upgrade, including
suspend/resume balance and manual transfer segments/channels.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-10 19:47:20 +08:00
cqy123456 47f767cf32
enhance: remove float16 in 2.3 branch (#31720)
issue: https://github.com/milvus-io/milvus/issues/31696

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-03-30 10:49:13 +08:00
PowderLi f2f0d44a5d
feat: [cherry-pick] restful phase two (#30430)
issue: #28348 #29732

Support to trace the grpc request, pr: #28349
Support to trace restful request and request error, pr: #28685

restful phase two, pr: #29728 #30343
include: collections, entities, partitions, users, roles, indexes,
aliases, import jobs

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: PowderLi <min.li@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
2024-03-25 10:39:09 +08:00
jaime 5ddb0b435f
fix: revoke session may be ignored due to server context cancellation in advance (#31213)
issue: #31219
pr: #31220

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-03-14 19:05:04 +08:00
congqixia 3c90475d55
enhance: [Cherry-pick] Add `ListIndexes` API from datacoord (#31104) (#31150)
Cherry-pick from master
pr: #31104
See also #31103

This PR add `listIndexes` API for datacoor server to list all indexes
for provided collection.
Comparing to the existing `DescribeIndex` API, the new one does NOT
check the segment index building progress to ease the burden when
invoking it

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-11 10:47:02 +08:00
yihao.dai 3eeeae8519
fix: Fix errors in the Index service APIs are ignored (#31077) (#31086)
In Index service APIs, return error if occurs instead of always
returning nil. Additionally, add more tests to cover this scenario.

issue: https://github.com/milvus-io/milvus/issues/31069,
https://github.com/milvus-io/milvus/issues/31027

pr: https://github.com/milvus-io/milvus/pull/31077

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-06 22:55:00 +08:00
wei liu af54c3ba85
fix: Make datacoord client retry on index api (#30656)
pr: #30654

This PR add retry on all interface which belong to indexcoord in milvus
2.2 and. move to data coord in milvus 2.3, to prevent meet unimplemented
error during rolling upgrade from milvus 2.2 to 2.3.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-04 11:37:09 +08:00
PowderLi c93f127c7d
fix: [cherry-pick] [restful v1] bug list (#30873)
master pr: #30871 issue: #30870
fix: vector field cannot be empty while insert
did a check whether the vector field is empty in advance

master pr: #30740
fix:
1. spelling mistake about metricsType #30643
2. int64 percious #20415
3. insert into collection which has multi vector fields #30674

enhance: support dataType: Float16Vector & BFloat16Vector #22837
#30980(master pr: #30969)
enhance: describe collection will show the field is partition key or not
#30789

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-03 17:56:59 +08:00
PowderLi a4219cbb0f
fix: [cherry-pick] set proxy.http.acceptTypeAllowInt64: true as default (#30738)
issue: #30680
pr: #30720

also let the parameter item to be refreshable

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-02-29 09:59:07 +08:00
chyezh a9625ec1ae
fix: nil ptr is used as nil interface in grpc client (#30755)
issue: #30715
pr: #30754

- Bug: Set nil struct pointer to describe nil interface.
Panic with segment violation when calling method on this nil struct
pointer.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-23 10:08:54 +08:00
yah01 f50799b7fd
fix: proxy may never setup if the port binded (#30035) (#30416)
the proxy miss-returned nil while failed to listen the port, then the
server continues to run but we can't connect to service resolve #30034
pr: #30035

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-02-02 16:21:06 +08:00
chyezh 77e123762f
enhance: add graceful stop timeout to avoid node stop hang under extreme cases (#30320)
1. add coordinator and proxy graceful stop timeout to 5s.
3. add other work node graceful stop timeout to 900s, and we should
potentially change this to 600s when graceful stop is smooth
4. change the order of datacoord component while stop.
5. `LivenessCheck` do not perform graceful shutdown now. 

issue: https://github.com/milvus-io/milvus/issues/30310
pr: #30317
also see: https://github.com/milvus-io/milvus/pull/30306

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-01-27 08:45:02 +08:00
SimFG 95cd6f20d0
fix: [2.3] wrong format expr for the delete rest api (#30218)
/kind improvement
issue: https://github.com/milvus-io/milvus/issues/30092
pr: #30217

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-01-24 11:27:05 +08:00
PowderLi 3dc2585d9b
enhance: support dataType: array & json (#30077)
issue: #30075 
master pr: #30076

deal with the array<?> field data correctly

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-21 14:00:56 +08:00
zhenshan.cao 9aceff5a6e
fix: duplicate dynamic field data by mistake (#30043)
issue: #30000 
pr: https://github.com/milvus-io/milvus/pull/30042

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-01-17 00:20:55 +08:00
PowderLi ff93e8b489
fix: [CHERRY-PICK] CollectionSchema.autoID is deprecated (#30011)
issue: [#30000](https://github.com/milvus-io/milvus/issues/30000)
related to: [milvus-proto
#202](https://github.com/milvus-io/milvus-proto/pull/202)
master pr: #30002

1. replace collSchema.AutoID with primaryField.AutoID
2. show `enableDynamic` & `enableDynamicField` at the same time
3. avoid data race about the access to metacache

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-16 14:32:53 +08:00
wayblink e1446da83c
feat: [Cherry-pick] Implement DescribeAlias and ListAliases interfaces (#29896)
#22882
pr: #29641

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-01-12 16:30:51 +08:00
congqixia 0c83440f99
enhance: [Cherry-pick] cache collection schema attributes to reduce proxy cpu (#29668) (#29692)
Cherry-pick from master
pr: #29668

See also #29113

The collection schema is crucial when performing search/query but some
of the information is calculated for every request.

This PR change schema field of cached collection info into a utility
`schemaInfo` type to store some stable result, say pk field,
partitionKeyEnabled, etc. And provided field name to id map for
search/query services.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-07 22:36:48 +08:00
PowderLi 9e7974a2c6
fix:[CHERRY_PICK] link with install path's libblob-chunk-manager (#29731)
issue: #29494
pr: #29496


1. link with install path's libblob-chunk-manager
2. performance of `ShouldBindWith` is better than `ShouldBindBodyWith`
3. the middleware shouldn't read the unrefreshed parameter repeatly
4. diff from old restful, named HandlersV1

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-07 19:52:49 +08:00
wei liu 5fa63da4e3
fix: Rename invalid parameterutil package path (#29334) (#29391)
pr: #29334
This PR renames the invalid parameterutil package path

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-22 10:06:41 +08:00
SimFG 74e72ce27e
enhance: [2.3] Support to get the param value in the runtime (#29298)
pr: #29297
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-12-21 20:36:43 +08:00
PowderLi 4ab02da3ac
enhance: RESTFUL search api support range search (#29097)
issue: #29004
master pr: #29055

add a new parameter: `params`, which is a map[string]float64;
but now only 2 valid item: radius + range_filter;

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-17 23:14:45 +08:00
congqixia d8d699401b
enhance: [cherry-pick] Add http method to control datacoord garbage collection (#29212)
Cherry-pick from master
pr: #29052
See also #29051

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi.Xia <congqi.xia@zilliz.com>
2023-12-15 02:16:38 +08:00
wei liu 7ef49adbd2
enhance: Use mockery to replace manual mock code (#29074) (#29182)
issue: #29043
pr: #29074
This PR remove mannul mock code for proxy and data coord

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-14 16:44:43 +08:00
PowderLi 92db41518d
enhance: upgrade proto to use partition related privileges (#29100)
issue: [milvus-proto
#212](https://github.com/milvus-io/milvus-proto/issues/212)
master pr: #28961

milvus can't use partition related privileges until upgrade
milvus-proto, even if them were added to milvus-proto

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-12 13:22:38 +08:00
shaoyue ac0f000c29
enhance: Add proxy.ginLogSkipPaths (#29008)
pr: #28945

/cc @czs007

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2023-12-08 17:55:30 +08:00
MrPresent-Han c21a1d59e4
fix: createCollection failed occasionally (#28592) (#28989)
related: #28592
pr:#https://github.com/milvus-io/milvus/pull/28712

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-08 11:34:39 +08:00
MrPresent-Han 5f4ac437b2
enhance: [Cherry-pick] Moving etcd client into session (#27069) (#28996)
relate: #26694
pr: https://github.com/milvus-io/milvus/pull/27069

Signed-off-by: Filip Haltmayer <filip.haltmayer@zilliz.com>
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Filip Haltmayer <81822489+filip-halt@users.noreply.github.com>
2023-12-07 16:22:34 +08:00
aoiasd cb7b1d1d51
enhance: [Cherry-pick] pack proxy connection code and support accesslog print SDK_Version (#28844)
relate: https://github.com/milvus-io/milvus/issues/28086
pr: https://github.com/milvus-io/milvus/pull/28835

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-07 12:06:34 +08:00
PowderLi cdd1305f26
enhance: find collection schema from cache (#28916)
issue: #28781 #28329
master pr: #28782

1. There is no need to call `DescribeCollection`, if the collection's
schema is found in the globalMetaCache
2. did `GetProperties` to check the access to Azure Blob Service

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-06 22:44:33 +08:00
aoiasd 224f2755c5
enhance: [Cherry-pick]Refine accesslog to support format by yaml and support more metric (#28740)
relate: https://github.com/milvus-io/milvus/issues/28086
pr: https://github.com/milvus-io/milvus/issues/28086

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-28 15:34:28 +08:00
jaime 2e78b4de96
fix: cmux graceful shutdown on proxy service #28383 (#28680)
issue: https://github.com/milvus-io/milvus/issues/28305
pr: #28383

Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-11-27 16:28:34 +08:00
yah01 5ca7851f4a
enhance: refine error messages (#28424) (#28614)
- Split the simple reason and full detail
- Refine existing error messages related: #28422
related: https://github.com/milvus-io/milvus/issues/28422
pr: #28424

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-24 10:04:24 +08:00
PowderLi 92e5f96064
fix: symbol 'GetStorageMetrics' and 'enableDynamicField' (#28581)
issue: #28579 #28504
pr: #28580

1. replace enableDynamic with enableDynamicField
2. cgo directly link to milvus_storage

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-21 15:30:23 +08:00
wei liu f2e972abb3
Fix external grpc server doesn't graceful stop (#28310)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-09 22:12:22 +08:00
wei liu d76c744646
Fix proxy not healthy before graceful stop (#28255)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-08 02:22:26 +08:00
PowderLi 43db873770
add a custom http header: Accept-Type-Allow-Int64 (#28125)
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-06 16:02:18 +08:00
wei liu 76b06abc29
refine stop order (#28016) (#28089)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-02 09:00:25 +08:00
jaime bf7f32b025
Refine Proxy stop (#27919)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-10-26 10:24:09 +08:00
jaime 4640928280
Fix initialization and backward compatibility issue for GRPC compression (#27894)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-10-24 22:14:14 +08:00
SimFG 9b0ecbdca7
Support to replicate the mq message (#27240)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-10-20 14:26:09 +08:00
PowderLi dcb7a89601
[restful] new interface: upsert (#27787)
interface: delete support expression

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-10-20 09:54:11 +08:00
SimFG 630636c4ec
Support the apikey authentication for the restful api (#27758)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-10-18 16:36:12 +08:00
SimFG 84d05b939b
Support the apikey authentication (#27723)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-10-17 21:00:11 +08:00
congqixia 2f201c25e2
Remove deprecated io/ioutil usage (#27747)
`io/ioutil` package is deprecated, use `io`,`os` package replacement
also added golangci-lint rule to block future reference

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: guoguangwu <guoguangwu@magic-shield.com>
2023-10-17 20:32:09 +08:00
PowderLi 09d8b76048
[restful] new context with grpc metadata (#27668)
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-10-17 20:00:14 +08:00
yah01 be980fbc38
Refine state check (#27541)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-11 21:01:35 +08:00
yah01 8394b3a1ec
Block creating new error from status reason (#27426)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-07 11:29:32 +08:00
Jiquan Long 0f14d18201
Optimize the codec code of session (#27360)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-10-01 10:33:30 +08:00
yah01 63ac43a3b8
Refine errors for import (#27379)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-30 10:31:28 +08:00