Commit Graph

277 Commits (21b41e96fcd776ee530aa3cdf51a1e2b8d049c4f)

Author SHA1 Message Date
chyezh 0c7474d7e8
enhance: add graceful stop timeout to avoid node stop hang under extreme cases (#30317)
1. add coordinator graceful stop timeout to 5s
2. change the order of datacoord component while stop
3. change querynode grace stop timeout to 900s, and we should
potentially change this to 600s when graceful stop is smooth

issue: #30310
also see pr: #30306

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-29 17:01:50 +08:00
chyezh e59bc1e526
enhance: use latest milvus proto, add unimplemented server (#30889)
issue: #30647

Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-29 10:13:00 +08:00
PowderLi 50a78b682e
fix: set proxy.http.acceptTypeAllowInt64: true as default (#30720)
issue: #30680

also let the parameter item to be refreshable

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-02-29 09:59:07 +08:00
PowderLi 6548182a4a
fix: [restful]vector field cannot be empty while insert (#30871)
issue: #30870 

did a check whether the vector field is empty in advance

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-02-28 10:36:57 +08:00
PowderLi 19911a411f
fix: [restful v2] bug list (#30740)
bug list
1. element data type is needed while create a collection with an array
field #30638
2. spelling mistake about metricsType #30643
3. new collection enable dynamic field / auto id as user defined #30665
4. convert rowCount from string to int #30661
5. try it's best to create a new collection #30652
6. int64 percious #20415
7. insert into collection which has multi vector fields #30674
8. cannot rename a collection to other database #30700
9. update the request body of "indexes/create" #30769
10. got [] while list indexes of a collection which has no index #30722
11. restful need encode password before call
CreateCredential/UpdateCredential #30730
12. some parameters missed the required label #30737
13.define the field to be or not to be a partition key while create a
collection #30797

enhance: support opentelemetry
enhance: support dataType: Float16Vector & BFloat16Vector #22837
enhance: describe collection will show the field is partition key or not
#30789

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-02-27 16:48:56 +08:00
chyezh 950624d8d3
fix: nil ptr is used as nil interface in grpc client (#30754)
issue: #30715

- Bug: Set nil struct pointer to describe nil interface.
Panic with segment violation when calling method on this nil struct
pointer.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-23 10:26:53 +08:00
PowderLi 1b87bcd60e
feat: restful support new features (#30485)
feat: restful support new features

1. search with groupingField #25324
2. hybrid search #25639

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-02-07 17:16:47 +08:00
PowderLi 5cf9bb236e
enhance: restful support import jobs (#30343)
issue: #28521 #29732

include
1. list collection's import jobs
2. create a new import job
3. get the progress of an import job

fix:
1. mix the order of dbName & collectionName #29728
2. trace log keep the same as v1
3. support traceID
4. azure precheck, blob name cannot end with / #29703

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-31 17:57:04 +08:00
yah01 2f778d9649
fix: proxy may never setup if the port binded (#30035)
the proxy miss-returned nil while failed to listen the port, then the
server continues to run but we can't connect to service
resolve #30034

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2024-01-31 17:37:04 +08:00
PowderLi 6abbab12fa
feat: restful phase two (#29728)
issue: #29732

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-28 16:03:01 +08:00
PowderLi 8fc4ebfa11
fix: empty MetricType (#30216)
issue: #30102 #30225

we should read MetricType from SearchResult,
because query node never
1. read metricType from LoadMeta
2. store to collection
3. set SearchRequest.MetricType

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-28 15:33:02 +08:00
PowderLi 08ca0a2ca5
feat: support etcd authentication (#30226)
issue: #28895
add 3 configuration for ETCD config

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-24 11:35:00 +08:00
SimFG b62748a1a6
fix: wrong format expr for the delete rest api (#30217)
/kind improvement
issue: #30092

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-01-24 11:27:06 +08:00
PowderLi 4f44942c80
enhance: support dataType: array & json (#30076)
issue: #30075

deal with the array<?> field data correctly

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-21 13:56:55 +08:00
PowderLi 631ebb1f07
fix: CollectionSchema.autoID is deprecated (#30002)
issue: #30000
related to: [milvus-proto
#202](https://github.com/milvus-io/milvus-proto/pull/202)

1. replace collSchema.AutoID with primaryField.AutoID
2. show `enableDynamic` & `enableDynamicField` at the same time
3. avoid data race about the access to metacache

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-18 16:16:52 +08:00
zhenshan.cao 31740445fa
fix: duplicate dynamic field data by mistake (#30042)
issue : #30000

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-01-17 19:02:56 +08:00
wayblink 1df3f90696
feat: Implement DescribeAlias and ListAliases interfaces (#29641)
#22882
/kind feature

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-01-11 19:12:51 +08:00
wayblink 05d735c322
enhance: Rename SearchV2 to HybridSearch (#29592)
related: https://github.com/milvus-io/milvus-proto/pull/233
issue: #29593
/kind enhancement

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-01-04 19:22:46 +08:00
congqixia 4f8c540c77
enhance: cache collection schema attributes to reduce proxy cpu (#29668)
See also #29113

The collection schema is crucial when performing search/query but some
of the information is calculated for every request.

This PR change schema field of cached collection info into a utility
`schemaInfo` type to store some stable result, say pk field,
partitionKeyEnabled, etc. And provided field name to id map for
search/query services.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-04 17:28:46 +08:00
PowderLi 5f00bad4b8
fix: link with install path's libblob-chunk-manager (#29496)
issue: #29494

1. link with install path's libblob-chunk-manager
2. performance of `ShouldBindWith` is better than `ShouldBindBodyWith`
3. the middleware shouldn't read the unrefreshed parameter repeatly

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-31 20:02:48 +08:00
yah01 a0e1a1eb31
feat: support enable/disable mmap for index (#29005)
support enable/disable mmap for index, the user could alter the index's
mode by `AlterIndex` method
related: https://github.com/milvus-io/milvus/issues/21866

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-21 18:07:24 +08:00
wei liu fcba1c0d9e
fix: Rename invalid parameterutil package path (#29334)
This PR renames the invalid parameterutil package path

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-21 11:40:42 +08:00
PowderLi bcd6865b29
enhance: add 3 builtin roles (#28961)
issue: #28960 [milvus-proto
#212](https://github.com/milvus-io/milvus-proto/issues/212)

add new configuration: builtinRoles
user can define roles in config file: `milvus.yaml`

there is an example:
1. db_ro, only have read privileges, include load
2. db_rw, read and write privileges, include create/drop/rename
collection
3. db_admin, not only read and write privileges, but also user
administration

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-18 14:28:41 +08:00
PowderLi 3f7d636ee8
enhance: RESTFUL search api support range search (#29055)
issue: #29004

add a new parameter: `params`, which is a map[string]float64;
but now only 2 valid item: radius + range_filter;

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-17 23:14:44 +08:00
wei liu fe1eeae2aa
enhance: Use mockery to replace manual mock code (#29074)
issue: #29043
This PR remove mannul mock code for proxy and data coord

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-13 10:46:44 +08:00
shaoyue 4a067a4c8c
enhance: Add proxy.ginLogSkipPaths (#28945)
Fix #28944

/cc @xiaofan-luan

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2023-12-06 10:26:35 +08:00
aoiasd 3cc4209d26
enhance: pack proxy connection code and support accesslog print SDK_Version (#28835)
relate: https://github.com/milvus-io/milvus/issues/28086
https://github.com/milvus-io/milvus/issues/28940

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-05 16:50:47 +08:00
PowderLi 20fc90c591
enhance: find collection schema from cache (#28782)
issue: #28781 #28329

1. There is no need to call `DescribeCollection`, if the collection's
schema is found in the globalMetaCache
2. did `GetProperties` to check the access to Azure Blob Service while
construct the ChunkManager

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-03 19:22:33 +08:00
aoiasd 89d8ce2f73
enhance: refine access log to support format access log by yaml and print name info. (#28319)
relate: https://github.com/milvus-io/milvus/issues/28086

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-28 15:32:31 +08:00
SimFG 9c46788d87
enhance: Support to trace restful request and request error (#28685)
issue: #28348

Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-11-27 20:14:26 +08:00
jaime c5f455dc6b
fix: cmux graceful shutdown on proxy service (#28383)
issue #28305

Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-11-27 16:28:34 +08:00
MrPresent-Han fc30d291be
fix createCollection failed occasionally (#28592) (#28712)
fix: create collection seldom failure #28592

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-11-27 11:10:25 +08:00
yah01 bfccfcd0ca
enhance: refine error messages (#28424)
- Split the simple reason and full detail
- Refine existing error messages
related: #28422

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 17:02:24 +08:00
PowderLi c238bff9fb
fix: symbol 'GetStorageMetrics' and 'enableDynamicField' (#28580)
/kind bug
to #28579 #28504

1. replace enableDynamic with enableDynamicField
2. cgo directly link to milvus_storage

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-21 10:20:22 +08:00
SimFG cfb6edea61
Support to trace the grpc request (#28349)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-11-13 20:16:18 +08:00
wei liu 8a370f115a
Fix external grpc server doesn't graceful stop (#28309)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-10 00:14:18 +08:00
wei liu 64f6dbabdc
Fix proxy not healthy before graceful stop (#28223)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-08 02:22:26 +08:00
PowderLi 0c0f012e03
add a custom http header: Accept-Type-Allow-Int64 (#27901)
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-01 11:42:16 +08:00
yah01 9658367a3c
Refine chunk manager errors (#27590)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-31 12:18:15 +08:00
Filip Haltmayer 6b1a106a31
Moving etcd client into session (#27069)
Signed-off-by: Filip Haltmayer <filip.haltmayer@zilliz.com>
2023-10-27 07:36:12 +08:00
jaime bf7f32b025
Refine Proxy stop (#27919)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-10-26 10:24:09 +08:00
jaime 4640928280
Fix initialization and backward compatibility issue for GRPC compression (#27894)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-10-24 22:14:14 +08:00
SimFG 9b0ecbdca7
Support to replicate the mq message (#27240)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-10-20 14:26:09 +08:00
PowderLi dcb7a89601
[restful] new interface: upsert (#27787)
interface: delete support expression

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-10-20 09:54:11 +08:00
SimFG 630636c4ec
Support the apikey authentication for the restful api (#27758)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-10-18 16:36:12 +08:00
SimFG 84d05b939b
Support the apikey authentication (#27723)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-10-17 21:00:11 +08:00
congqixia 2f201c25e2
Remove deprecated io/ioutil usage (#27747)
`io/ioutil` package is deprecated, use `io`,`os` package replacement
also added golangci-lint rule to block future reference

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: guoguangwu <guoguangwu@magic-shield.com>
2023-10-17 20:32:09 +08:00
PowderLi 09d8b76048
[restful] new context with grpc metadata (#27668)
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-10-17 20:00:14 +08:00
yah01 be980fbc38
Refine state check (#27541)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-11 21:01:35 +08:00
yah01 8394b3a1ec
Block creating new error from status reason (#27426)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-07 11:29:32 +08:00