Commit Graph

599 Commits (5b693c466d8716098ab1cdc3b3e16a72eeaeee1b)

Author SHA1 Message Date
jaime bd853be8c7
enhance: Add db label for some usual metrics (#30956)
issue: #31782

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-02 14:27:13 +08:00
wei liu 9111ecab82
enhance: Rootcoord's stop may block in quota_center's stop (#31447)
this PR fixed that rootcoord's stop may block in quota_center' stop

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-01 21:09:18 +08:00
cai.zhang be6b468a3b
fix: Check whether the DataType of the column is legal (#31697)
issue: #31696

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-29 09:33:16 +08:00
Bingyi Sun 3d66670619
fix: add lazy load field to mark segment load type (#31591)
issue: https://github.com/milvus-io/milvus/issues/31673

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-28 11:23:10 +08:00
SimFG 8f3e0b6b41
enhance: the return result of list db api (#31544)
issue: #31543

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-03-28 07:13:10 +08:00
SimFG b1a1cca10b
feat: add more operation detail info for better allocation (#30438)
issue: #30436

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-03-28 06:33:11 +08:00
yihao.dai 0fe5e90e8b
enhance: Remove import v1 (#31403)
Remove all code and logic related to import v1.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-22 15:29:09 +08:00
wei liu e6d50def4f
enhance: Enable properites in database meta (#31394)
issue: #30040

This PR add properties in database meta, so we can store some database
level info.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-19 19:21:06 +08:00
Jiquan Long dc2cdbe387
enhance: add more metrics (#31271)
/kind improvement
fix: #31272 

This pr add more metrics, which are:
- Slow query count, which the duration considered as slow can be
configurable;
- Number of deleted entities;
- Number of entities imported;
- Number of entities per collection;
- Number of loaded entities per collection;
- Number of indexed entities;
- Number of indexed entities, per collection, per index and whether it's
a vetor index;
- Quota states (LongTimeTickDelay, MemoryExhuasted, DiskQuotaExhuasted)
per database;

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-19 15:23:06 +08:00
Bingyi Sun 992cb6a532
enhace: optimize alter collection log print (#31275)
issue: https://github.com/milvus-io/milvus/issues/31142

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-18 15:19:06 +08:00
jaime db79be3ae0
fix: ctx cancel should be the last step while stopping server (#31220)
issue: #31219

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-03-15 10:33:05 +08:00
chyezh 6a9418d200
fix: lost dbname when only passing collection id to describeCollection (#31167)
issue: #30931

Signed-off-by: chyezh <chyezh@outlook.com>
2024-03-11 17:19:02 +08:00
chyezh 0c7474d7e8
enhance: add graceful stop timeout to avoid node stop hang under extreme cases (#30317)
1. add coordinator graceful stop timeout to 5s
2. change the order of datacoord component while stop
3. change querynode grace stop timeout to 900s, and we should
potentially change this to 600s when graceful stop is smooth

issue: #30310
also see pr: #30306

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-29 17:01:50 +08:00
congqixia af315539d4
fix: Remove time tick delay metrics when nodes go offline (#30833)
See also #30832

This PR removes time tick delay metrics when rootcoord GetMetrics
response does not have previously existed querynode/datanode

Also add unit tests for this case

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi.Xia <congqi.xia@zilliz.com>
2024-02-28 10:10:56 +08:00
zhenshan.cao bb93b22c84
fix: should return collectionName in response of ListAliases (#30532)
issue : https://github.com/milvus-io/milvus/issues/30369

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-02-12 08:30:55 +08:00
cai.zhang f619d792c0
enhance: Break down the granularity of collection info cache expired (#29977)
issue: #29772 

1. `DropPartition` only invalidates the cache related to the partition.
2. `CreateAlias` does not invalidate the cache.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-01-30 16:45:02 +08:00
MrPresent-Han 2a0eb1d2e6
feat: support general capacity restrict for cloud-side resoure contro… (#29845)
related: #29844

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-16 16:32:53 +08:00
wayblink 1df3f90696
feat: Implement DescribeAlias and ListAliases interfaces (#29641)
#22882
/kind feature

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-01-11 19:12:51 +08:00
yah01 e8496d4d49
enhance: filter out the not needed collections while listing (#29690)
this improves performance while many collections exist
resolve #29631

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-10 15:18:48 +08:00
smellthemoon 1c1f2a1371
enhance:change some logs (#29579)
related #29588

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-05 16:12:48 +08:00
yah01 99e0f1e65a
enhance: unable to compile C++ tests (#29616)
The tests need to call a private method, Milvus uses `#define` to
replace private with public, the hack trick works but would be broken if
the including order changed.

This uses friend to make all things work well

Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2024-01-04 13:20:46 +08:00
SimFG dd9c61831d
enhance: Support to get the param value in the runtime (#29297)
/kind improvement
issue: #29299

Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-12-22 18:36:44 +08:00
wei liu fcba1c0d9e
fix: Rename invalid parameterutil package path (#29334)
This PR renames the invalid parameterutil package path

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-21 11:40:42 +08:00
wei liu e41fd6fbde
enhance: Move proxy client manager to util package (#28955)
issue:  #28898

This PR move the `ProxyClientManager` to util package, in case of
reusing it's implementation in querycoord

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-20 19:22:42 +08:00
PowderLi bcd6865b29
enhance: add 3 builtin roles (#28961)
issue: #28960 [milvus-proto
#212](https://github.com/milvus-io/milvus-proto/issues/212)

add new configuration: builtinRoles
user can define roles in config file: `milvus.yaml`

there is an example:
1. db_ro, only have read privileges, include load
2. db_rw, read and write privileges, include create/drop/rename
collection
3. db_admin, not only read and write privileges, but also user
administration

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-18 14:28:41 +08:00
yah01 b8674811cf
fix: data race in ProxyClientManager (#29206)
this PR changed the ProxyClientManager to thread-safe
fix #29205

Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-14 18:22:39 +08:00
SimFG 5a7fb97e07
fix: missing the grant info when using the SelectGrant api with the default db (#29173)
issue: #29172

Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-12-14 16:20:43 +08:00
wei liu fe1eeae2aa
enhance: Use mockery to replace manual mock code (#29074)
issue: #29043
This PR remove mannul mock code for proxy and data coord

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-13 10:46:44 +08:00
cai.zhang 49b8657f95
enhance: Support implicit type conversion for parquet (#29046)
issue: #29019

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-12 16:14:44 +08:00
wayblink 994b239161
fix: fix datarace in root_coord proxy_client_manager (#28796)
bug: #28797

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-12-12 10:18:39 +08:00
xige-16 89b965e2b1
fix: clean snapshots meta (#28502)
issue: https://github.com/milvus-io/milvus/issues/28496
/kind bug

The input parameters collection.partitions and collection.Field are both
nil, so these two metas have not been cleared.

Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-12-05 16:04:36 +08:00
wei liu 6718986a5b
enhance: Remove too frequency read ddl log from rootcoord (#28891)
This PR removed too frequency log for such
`DescribeCollection/ShowPartition` operation from root coord

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-04 09:42:33 +08:00
congqixia 881a166b5a
fix: data race on rootcoord.proxyClientManager.proxyClient (#28787)
Fix #28786

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-28 15:52:26 +08:00
cai.zhang 1b7a503f89
enhance: Revert import support csv format (#28760)
Revert import support csv format.
issue: #28778

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-28 14:32:27 +08:00
jaime b1e0a27f31
enhance: Add logs for each step during service initialization (#28624)
/kind improvement

Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-11-27 16:30:26 +08:00
smellthemoon 79c0edb1d8
enhance:Remove msgbase unnecessary assignments (#28511)
remove some unnecessary assignments, for the reason that
commonpbutil.NewMsgBase has default value.

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-11-24 15:02:39 +08:00
wayblink 53646032d0
fix: rootcoord_collection_num metric is twice the real value (#28666)
#28665

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-11-23 20:42:23 +08:00
congqixia a2fe9dad49
enhance: Make etcd kv request timeout configurable (#28661)
See also #28660
This pr add request timeout config item for etcd kv request timeout
 Sync the default timeout value to same value for etcdKV & tikv config

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-23 19:34:23 +08:00
smellthemoon 73f2bab454
enhance:add some log when create client and get component states (#28160)
/kind improvement

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-11-22 09:12:22 +08:00
yah01 bfccfcd0ca
enhance: refine error messages (#28424)
- Split the simple reason and full detail
- Refine existing error messages
related: #28422

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 17:02:24 +08:00
yihao.dai c16fc854d1
Fix confusing quota log (#28262)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-11-13 15:38:18 +08:00
wei liu bce1054f92
Fix retry when proxy stopped (#28264)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-09 18:58:21 +08:00
KumaJie e88212ba4b
Add CSV file import function (#27149)
Signed-off-by: kuma <675613722@qq.com>
Co-authored-by: kuma <675613722@qq.com>
2023-10-31 22:47:23 +08:00
groot e6d4cde397
Fix duplicated segment id in bulkinsert task state (#27915)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-10-27 15:14:13 +08:00
Filip Haltmayer 6b1a106a31
Moving etcd client into session (#27069)
Signed-off-by: Filip Haltmayer <filip.haltmayer@zilliz.com>
2023-10-27 07:36:12 +08:00
SimFG 9b0ecbdca7
Support to replicate the mq message (#27240)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-10-20 14:26:09 +08:00
zhenshan.cao 020ad9a6bc
Rectify wrong exception messages associated with Array datatype (#27769)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-10-19 17:24:07 +08:00
wayblink e3f2122618
Expose metrics of stanby coordinators (#27698)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-10-16 15:04:09 +08:00
jaime ec1fe3549e
Add a stop hook to clean session (#27564)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-10-16 10:24:10 +08:00
yah01 15746b5b5f
Refine KV errors (#27588)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-13 17:15:34 +08:00