Commit Graph

481 Commits (0f8c6f49ff19624d6cd3c4aa788bb184baa051aa)

Author SHA1 Message Date
yah01 8f083e45cb
enhance: enable converting segcore error to merr (#29914)
this converts the segcore error to merr if possible

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-17 11:28:53 +08:00
congqixia 7cb6bebd96
enhance: replace magic number with ParamItem for dist handler (#30020)
See also #28817

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-16 17:33:03 +08:00
MrPresent-Han 2a0eb1d2e6
feat: support general capacity restrict for cloud-side resoure contro… (#29845)
related: #29844

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-16 16:32:53 +08:00
yah01 cda44c09f9
enhance: adjust the GPU pool size (#29937)
according to benchmark, the GPU pool size with 6 performs best

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-16 11:54:52 +08:00
jaime ee26561b5d
fix: remove checking if running inside container (#29940)
issue: https://github.com/milvus-io/milvus/issues/29846

Since checking if Milvus running inside the container is difficult,
which is decided by the system version and cgroup version, we try to get
the memory limit of the container each time instead of checking whether
inside the container first.

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-01-14 10:24:52 +08:00
Bingyi Sun e1258b8cad
feat: integrate storagev2 into loading segment (#29336)
issue: #29335

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-01-12 18:10:51 +08:00
zhenshan.cao a442194abc
feat: Add RBAC functionality to alias (#29885)
issue: https://github.com/milvus-io/milvus/issues/29781
issue: https://github.com/milvus-io/milvus-proto/issues/237

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-01-12 15:26:50 +08:00
jaime 90984a2cab
enhance: Support read hardware metrics for cgroupv2 (#29850)
issue: https://github.com/milvus-io/milvus/issues/29846

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-01-12 10:30:51 +08:00
wayblink 1df3f90696
feat: Implement DescribeAlias and ListAliases interfaces (#29641)
#22882
/kind feature

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-01-11 19:12:51 +08:00
Xu Tong e429965f32
Add float16 approve for multi-type part (#28427)
issue:https://github.com/milvus-io/milvus/issues/22837

Add bfloat16 vector, add the index part of float16 vector.

Signed-off-by: Writer-X <1256866856@qq.com>
2024-01-11 15:48:51 +08:00
wei liu 1f759837c4
fix: remove Unnecessary lock in config manager (#29836)
issue: #29709 #291712
to avoid concurrent recursive RLock and Lock cause deadlock, This PR
remove the unnecessary lock in config manager

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-11 13:48:49 +08:00
congqixia 8a6e1a4b27
enhance: pre-allocate result FieldData space to reduce copy & growslice (#29726)
See also: #29113

Add a new utitliy function in `pkg/util/typetuil` to pre-allocate field
data slice capacity acoording to search limit. This shall avoid copying
the data during `AppendFieldData` when previous slice is out of space.
And shall also save CPU time during high paylog.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-09 15:48:55 +08:00
xige-16 9702cef2b5
feat: Support multiple vector search (#29433)
issue #25639 

Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-08 15:34:48 +08:00
wayblink 635a7f777c
feat: add clustering key in create/describe collection (#29506)
#28410
/kind feature

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-01-07 19:56:48 +08:00
yihao.dai 23183ffb0f
feat: Add import reader for json (#29252)
This PR implements a new json reader for import.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-05 18:12:48 +08:00
smellthemoon 1c1f2a1371
enhance:change some logs (#29579)
related #29588

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-05 16:12:48 +08:00
wei liu 07057fcf7c
fix: Unexpected rpc msg size limit (#29682)
due to `clientMaxSendSize` and `serverMaxRecvSize` will limit the rpc
request size limit, they should use same config value, and
`serverMaxSendSize` and `clientMaxRecvSize` will limit the rpc response
size limit, they should use same config value too.

This PR fix unexpected rpc msg limit which caused by the wrong usage of
misunderstanding rpc config items

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-05 15:56:47 +08:00
MrPresent-Han 9e2e7157e9
feat: support search_group_by for milvus(#25324) (#28983)
related: #25324

Search GroupBy function, used to aggregate result entities based on a
specific scalar column.
several points to mention:

1. Temporarliy, the whole groupby is implemented separated from
iterative expr framework **for the first period**
2. In the long term, the groupBy operation will be incorporated into the
iterative expr framework:https://github.com/milvus-io/milvus/pull/28166
3. This pr includes some unrelated mocked interface regarding alterIndex
due to some unworth-to-mention reasons. All these un-associated content
will be removed before the final pr is merged. This version of pr is
only for review
4. All other related details were commented in the files comparison

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-05 15:50:47 +08:00
cqy123456 22bb84fa9d
feat:add new gpu index:GPU_BRUTE_FORCE and limit gpu index metric type (#29590)
issue: https://github.com/milvus-io/milvus/issues/29230
this pr do these things:
1. add gpu brute force;
2. limit gpu index only support l2 / ip;

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-01-05 15:24:48 +08:00
yihao.dai 3561586edf
feat: Add import reader for binlog (#28910)
This PR defines the new import reader interfaces and implement a binlog
reader for import.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-05 11:48:47 +08:00
yah01 9e0163e12f
enhance: use GPU pool for gpu tasks (#29678)
- this much improve the performance for GPU index

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-04 17:50:46 +08:00
congqixia da7c3cbd88
enhance: make delegator delete buffer holding all delete from cp (#29626)
See also #29625

This PR:
- Add a new implemention of `DeleteBuffer`: listDeleteBuffer
  - holds cacheBlock slice
  - `Put` method append new delete data into last block
  - when a block is full, append a new block into the list
- Add `TryDiscard` method for `DeleteBuffer` interface
  - For doubleCacheBuffer, do nothing
- For listDeleteBuffer, try to evict "old" blocks, which are blocks
before the first block whose start ts is behind provided ts
- Add checkpoint field for `UpdateVersion` sync action, which shall be
used to discard old cache delete block

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-04 17:02:46 +08:00
SimFG d23f87a393
enhance: Add concurrency for datacoord segment GC (#29561)
issue: https://github.com/milvus-io/milvus/issues/29553
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-01-03 13:16:57 +08:00
MrPresent-Han ed644983e2
enhance: add param for bloomfilter(#29388) (#29490)
related: #29388

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-28 18:10:46 +08:00
xige-16 0a70e8b601
enhance: Remove multiple vector field limit (#27827)
issue: https://github.com/milvus-io/milvus/issues/25639

/kind improvement
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-12-28 16:40:46 +08:00
Gao 8a4c0d4a3f
enhance: compile go assembly with build constraints for different arch (#29515)
issue #28657 
follow https://pkg.go.dev/cmd/go#hdr-Build_constraints to compile go
assembly with different cpu arch

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-12-28 15:40:47 +08:00
Gao 8a630f733a
enhance: add new optimize param for queryhook (#29495)
add a flag to indicate if we use search param optimizations, default is
on

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-12-28 10:04:46 +08:00
wei liu 839a72129e
fix: Auto balance param can't be updated by dynamic (#29501)
This PR fixed that auto balance param can't be updated by dynamic

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-27 14:30:53 +08:00
MrPresent-Han 7c7003bff6
enhance:refine the range of chunk size config value(#29388) (#29389)
related: #29388

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-26 17:36:46 +08:00
cqy123456 4c979538a4
enhance: update cagra index params in config and add params check (#29045)
issue:https://github.com/milvus-io/milvus/issues/29230
this pr do two things about cagra index:
 a.milvus yaml config support gpu memory settings

 b.add cagra-params check

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
Co-authored-by: yusheng.ma <yusheng.ma@zilliz.com>
2023-12-26 11:04:47 +08:00
aoiasd ac23e67f07
enhance: add default file path for access log (#29460)
relate:https://github.com/milvus-io/milvus/issues/29459

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-26 10:38:45 +08:00
MrPresent-Han bd3bde82f0
fix iterator lose data for duplicted result(#29406) (#29451)
related: #29406

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-25 23:28:47 +08:00
PowderLi 13988cbde7
fix: Partition related privileges should belong to Collection (#29428)
issue: #29427

it will try to find privilege_name from the object_type, while grant a
privilege to a role

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-24 20:32:45 +08:00
SimFG dd9c61831d
enhance: Support to get the param value in the runtime (#29297)
/kind improvement
issue: #29299

Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-12-22 18:36:44 +08:00
yah01 a0e1a1eb31
feat: support enable/disable mmap for index (#29005)
support enable/disable mmap for index, the user could alter the index's
mode by `AlterIndex` method
related: https://github.com/milvus-io/milvus/issues/21866

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-21 18:07:24 +08:00
wei liu fcba1c0d9e
fix: Rename invalid parameterutil package path (#29334)
This PR renames the invalid parameterutil package path

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-21 11:40:42 +08:00
yihao.dai f457b9f7c9
fix: Return time tick delay error and refine quota error messages (#29289)
This pr:
1. Handles the time tick delay error when converting old error codes to
milvus errors.
2. Enhances quota error messages by eliminating "force deny" and
substituting it with "quota exceeded."

issue: https://github.com/milvus-io/milvus/issues/29288

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-12-20 22:52:44 +08:00
aoiasd 46a937ac43
fix: msgstream memory leak caused by config event don't deregister (#29266)
relate: https://github.com/milvus-io/milvus/issues/28620

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-20 20:32:44 +08:00
XuanYang-cn 7d32861d53
enhance: Add channel check interval params (#29311)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-20 15:06:42 +08:00
PowderLi bcd6865b29
enhance: add 3 builtin roles (#28961)
issue: #28960 [milvus-proto
#212](https://github.com/milvus-io/milvus-proto/issues/212)

add new configuration: builtinRoles
user can define roles in config file: `milvus.yaml`

there is an example:
1. db_ro, only have read privileges, include load
2. db_rw, read and write privileges, include create/drop/rename
collection
3. db_admin, not only read and write privileges, but also user
administration

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-18 14:28:41 +08:00
zhagnlu a602171d06
enhance: Refactor runtime and expr framework (#28166)
#28165

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-18 12:04:42 +08:00
wei liu 820ee692fc
enhance: Add config for querycoord auto balance channel (#29231)
issue: #23726
This PR add control config to querycoord's background auto balance
channel operation

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-18 10:00:40 +08:00
congqixia 88b4b8b77c
enhance: make segments SQPool & LoadPool resizable (#29239)
See also #29223

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-15 20:22:39 +08:00
congqixia 4731c1b0d5
enhance: make SyncManager pool size refreshable (#29224)
See also #29223

This PR make `conc.Pool` resizable by adding `Resize` method for it. 
Also make newly added datanode `MaxParallelSyncMgrTasks` config
refreshable

---------

Signed-off-by: Congqi.Xia <congqi.xia@zilliz.com>
2023-12-15 09:58:43 +08:00
congqixia 25a4525297
enhance: Change sync manager parallel config item (#29216)
Since the sync manager is global in datanode now, the old
`maxParallelSyncTaskNum` does not fit into current implementation
anymore.

This PR add a new param item for sync mgr parallel control and enlarge
default value

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-14 20:46:41 +08:00
congqixia 9407be6eb2
enhance: Add a config item for partition name as regexp feature and disable it by default (#29154)
See also #29177
Add a config item for partition name as regexp feature and disable it by
default

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-14 10:04:38 +08:00
aoiasd b5ee563914
fix: accesslog can not print search expression (#28899)
relate: https://github.com/milvus-io/milvus/issues/28893

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-13 18:50:42 +08:00
Jiquan Long 520c07ec9d
fix: autoindex panic with flat index type (#29071)
issue: https://github.com/milvus-io/milvus/issues/29048

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-12-12 22:08:46 +08:00
Jiquan Long e5f155612a
fix: txn control not applied to snapshot.Remove (#29124)
issue: #29123

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-12-12 14:48:38 +08:00
wayblink 51f870da7e
feat: Introduce channelCheckpointUpdater to reduce goroutine use in ttNode (#28570)
/kind enhancement

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-12-12 13:48:42 +08:00
wei liu 69a71e11d1
enhance: Add config for auto upgrade segment index (#29112)
This PR add config for auto upgrade segment index to index engine's
version.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-12 11:56:37 +08:00
Enwei Jiao 0e65e90338
enhance: Support otlp with insecure (#29115)
issue: https://github.com/milvus-io/milvus/issues/28914

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-12-12 11:14:37 +08:00
yah01 0a87724f18
enhance: remove merger for load segments (#29062)
remove merger as now QueryNode could load segments concurrently
fix #29063

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-12 10:48:37 +08:00
cai.zhang 2b05460ef9
enhance: Make import-related error message clearer (#28978)
issue: #28976

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-08 10:12:38 +08:00
wayblink 6736f65345
feat: skip some empty ttMsg in Datanode flowgraph (#28756)
/kind feature

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-12-07 01:00:37 +08:00
shaoyue 4a067a4c8c
enhance: Add proxy.ginLogSkipPaths (#28945)
Fix #28944

/cc @xiaofan-luan

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2023-12-06 10:26:35 +08:00
groot ad1daebc8e
fix: Change ranges of HNSW parameters according to Knowhere (#28875)
fix: https://github.com/milvus-io/milvus/issues/28860

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-12-05 16:20:42 +08:00
yah01 bf633bb5d7
enhance: refine the retry error (#28573)
return the last error but not combining all errors, to improve
readability and erorr handling

resolve: #28572

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-30 18:34:32 +08:00
shaoyue 8b2b0d412c
enhance: storeageType default value change to remote (#28792)
/kind enhancement
/cc @PowderLi

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2023-11-30 11:34:27 +08:00
congqixia f9bb8e9648
enhance: Change const magic number in querycoord to param (#28819)
See also #28817

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-30 09:06:28 +08:00
cqy123456 3b1b14dd78
fix: update binlog index memory uasge before loading segments (#28528)
issue: #27678 
when interimIndex = true, memory predict should be update with the
memory usage of binlog index build process.

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-11-29 16:42:27 +08:00
Gao 5bd73d5503
enhance: add efficient distance computations in Go (#28657)
Related #28656
Add more efficient calc_distance at Go side.

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-11-28 18:20:26 +08:00
XuanYang-cn 606ec77b66
enhance: Unify levelzero segment config in DN (#28720)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-28 18:04:26 +08:00
aoiasd 89d8ce2f73
enhance: refine access log to support format access log by yaml and print name info. (#28319)
relate: https://github.com/milvus-io/milvus/issues/28086

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-28 15:32:31 +08:00
cai.zhang 1b7a503f89
enhance: Revert import support csv format (#28760)
Revert import support csv format.
issue: #28778

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-28 14:32:27 +08:00
SimFG 9c46788d87
enhance: Support to trace restful request and request error (#28685)
issue: #28348

Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-11-27 20:14:26 +08:00
cai.zhang c29b60e18e
enhance: Support Array DataType for bulk_insert (#28341)
issue: #28272 
Support array DataType for bulk_insert with json, binlog files.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-27 13:50:27 +08:00
Bingyi Sun 2843664a20
fix: channel name can not contains '-partition-' (#28689)
issue: https://github.com/milvus-io/milvus/issues/28675

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-24 10:14:24 +08:00
wei liu 4b8d3eaa41
enhance: Increase heartbeat timeout (#28648)
it's easy to trigger heartbeat timeout after 100ms when standalone cpu
usage reach 100%.
This PR increase the heartbeat timeout param to 2000ms

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-23 19:50:24 +08:00
congqixia a2fe9dad49
enhance: Make etcd kv request timeout configurable (#28661)
See also #28660
This pr add request timeout config item for etcd kv request timeout
 Sync the default timeout value to same value for etcdKV & tikv config

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-23 19:34:23 +08:00
zhagnlu bc43c5cf34
fix: Panic if minio port is empty (#28461)
#28460

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-11-23 19:24:25 +08:00
Bingyi Sun 4fedff6d47
feat: integrate storage v2 into the write path (#28440)
#28378

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-23 17:26:24 +08:00
yah01 c96d07682e
enhance: enhance the accuracy of memory usage (#28554)
before this, Milvus use container/system's memory info to get the memory
usage, which could be inaccurate.

we allocates the memory by private anon mmap,
then `rss - shared` would be the accurate memory usage

resolve #28553

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-23 15:12:23 +08:00
yah01 bfccfcd0ca
enhance: refine error messages (#28424)
- Split the simple reason and full detail
- Refine existing error messages
related: #28422

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 17:02:24 +08:00
XuanYang-cn e88bbaac24
feat: Add universal levelzero segment switch (#28483)
See also: #27349

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-20 19:04:22 +08:00
PowderLi 7029797452
Improve HTTP server include metrics and RESTful API (#28045)
/kind improvement
issue: #27653

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-19 17:14:20 +08:00
yah01 e9ff7ed13d
enhance: remove legacy error related code (#28385)
/kind improvement

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-14 15:02:18 +08:00
SimFG cfb6edea61
Support to trace the grpc request (#28349)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-11-13 20:16:18 +08:00
XuanYang-cn f8aa46419a
Add LevelZeroCompaction configs (#28190)
See also: #27606

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-13 11:18:19 +08:00
SimFG 7dda2e8814
Change some log level in the pkg package (#28181)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-11-08 23:34:22 +08:00
zhenshan.cao 2057fdce43
Change default storage type to minio (#28236)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-11-07 21:15:31 +08:00
wei liu 5b45a138b1
disable auto balance when old node exists (#28191)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-07 14:02:20 +08:00
Xiaofan da19e49daf
Support purge old session for standalone (#28184)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-11-06 21:21:42 +08:00
yah01 90e2c63d9e
Fix getting incorrect CPU num (#28146)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-11-06 06:02:16 +08:00
congqixia bf2f62c1e7
Add `WriteBuffer` to provide abstraction for delta policy (#27874)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-04 12:10:17 +08:00
SimFG ef68680639
Fix the unstable resource manager testcase (#28033)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-11-03 11:00:17 +08:00
aoiasd d2727cc0ff
Add write cache for access log (#27792)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-03 10:28:17 +08:00
wei liu 3b8fbbd91d
avoid sleep to ctx done during retry (#27992)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-01 18:52:24 +08:00
PowderLi 0c0f012e03
add a custom http header: Accept-Type-Allow-Int64 (#27901)
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-01 11:42:16 +08:00
Enwei Jiao 8ae9c947ae
Use OpenDAL to access object store (#25642)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-11-01 09:00:14 +08:00
cqy123456 4fbe3c9142
replace loaded binlog with binlog index for search performance (#27673)
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-11-01 02:20:15 +08:00
yah01 9658367a3c
Refine chunk manager errors (#27590)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-31 12:18:15 +08:00
yah01 2af46d7333
Increase the ChunkManager request timeout (#28015)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-31 09:06:13 +08:00
MrPresent-Han ccaec44930
enable parameter for dataCoord broker timeout(#26972) (#27825)
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-10-28 06:14:11 +08:00
yihao.dai f9c630247d
Construct plan directly when search with vector output (#27928)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-10-26 19:30:10 +08:00
congqixia 8ab01e5a14
Set ElementType for empty array FieldData (#27924)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-25 20:08:10 +08:00
Gao c8415453a1
Support prepare config for index (#27920)
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-10-25 19:54:12 +08:00
aoiasd 53246b1b38
Set accesslog default to close and use stdout (#27891)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-10-25 10:30:10 +08:00
wei liu 40723a292e
reduce compact parallel task num (#27899)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-10-25 09:40:12 +08:00
jaime 4640928280
Fix initialization and backward compatibility issue for GRPC compression (#27894)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-10-24 22:14:14 +08:00
congqixia 93a877f55e
Make qcv2 target&leader observer execute in parallel (#27844)
- Add `taskDispatcher` to submit and run task async safely
- Change `LeaderObeserver` and `TargetObserver` schedule and manual check action to submitting task into dispatcher
- Fix logic problem in collection observer when manual check return false

See also #27494

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-24 10:14:11 +08:00
Xiaofan 2ea7579dbb
Reduce rpc size for GetRecoveryInfoV2 (#27483)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-10-23 21:44:09 +08:00
zhagnlu 6060dd7ea8
Add chunk manager request timeout (#27692)
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-10-23 20:08:08 +08:00
congqixia 4a6790b63a
Make KeyLock generic (#27854)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-23 17:40:11 +08:00
SimFG 9b0ecbdca7
Support to replicate the mq message (#27240)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-10-20 14:26:09 +08:00
zhenshan.cao 020ad9a6bc
Rectify wrong exception messages associated with Array datatype (#27769)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-10-19 17:24:07 +08:00
smellthemoon 4b0ec156b3
Set channel work pool size in datanode (#27728)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-10-19 08:28:08 +08:00
yah01 9467de79c3
Use pool to block exceeded tasks (#27767)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-19 03:30:06 +08:00
yihao.dai 49b3a12804
Return newly defined merr instead of grpc unimplemented err (#27751)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-10-18 15:32:11 +08:00
congqixia 2f201c25e2
Remove deprecated io/ioutil usage (#27747)
`io/ioutil` package is deprecated, use `io`,`os` package replacement
also added golangci-lint rule to block future reference

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: guoguangwu <guoguangwu@magic-shield.com>
2023-10-17 20:32:09 +08:00
Futao Wei 599012a340
Implement SearchByPks path for Search (#25882)
Signed-off-by: unfode <forrest.futao.wei@gmail.com>
2023-10-16 15:34:08 +08:00
wayblink e3f2122618
Expose metrics of stanby coordinators (#27698)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-10-16 15:04:09 +08:00
wei liu 43f2452f4b
speed up shard leader cache update frequency (#27640)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-10-16 10:52:10 +08:00
yah01 be980fbc38
Refine state check (#27541)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-11 21:01:35 +08:00
foxspy 2b46bd1f08
fix diskann searchCache param compatibility (#27557)
Signed-off-by: xianliang <xianliang.li@zilliz.com>
2023-10-11 16:57:34 +08:00
yah01 2df9908c6a
Map old error code to new error (#27616)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-11 12:43:33 +08:00
yah01 3759857bc5
Refine Proxy errors (#27499)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-09 10:09:33 +08:00
XuanYang-cn 56c94cdfa7
Add channel manager in DataNode (#27308)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-10-08 21:37:33 +08:00
yah01 41495ed266
Improve the error message for getting all indexes of collection (#27389)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-08 21:23:32 +08:00
PowderLi 8d3069b1db
update openssl to 3.1.2 (#27399)
deal with root path's normalization

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-10-08 19:17:31 +08:00
congqixia 5d558623fe
Add revive sub-lints and fix existing problems (#27495)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-07 20:53:38 +08:00
congqixia 80eb5434d3
Fix var-naming caused by old PRs (#27501)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-07 14:43:32 +08:00
yah01 8394b3a1ec
Block creating new error from status reason (#27426)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-07 11:29:32 +08:00
yah01 63ac43a3b8
Refine errors for import (#27379)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-30 10:31:28 +08:00
yihao.dai 106c17f304
Make read ahead policy in ChunkCache configurable (#27291)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-28 15:47:27 +08:00
congqixia 258e1ccd66
Refine querynode scheduler lifetime (#26915)
This PR refines scheduler lifetime control:
- Move private tri-state into lifetime package
- Make scheduler block incoming "Add" task
- Make scheduler Stop wait until all previously accepted task done

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-28 10:21:26 +08:00
MrPresent-Han 505e8509b7
refine the default values for segcore threadpool size (#27406)
/kind improvement

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-09-27 20:43:28 +08:00
wei liu 4071132f6a
reload loading collection when qc recover (#27300)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-09-27 11:43:28 +08:00
SimFG 9fb4c27a90
Fix rootcoord failure to start when reaching the limit of role num (#27361)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-27 11:03:26 +08:00
congqixia a3dd2756cf
Add predicates for TxnKV operations (#27365)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-27 10:21:26 +08:00
yah01 6539a5ae2c
Refine DataCoord status (#27262)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-26 17:15:27 +08:00
cai.zhang 30a6c6072e
Fix bug for loading array data panic (#27343)
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-09-26 12:23:26 +08:00
wayblink 7dd0be1b2c
Enable bulkinsert binlog data with partitionkey (#27241)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-09-26 10:25:25 +08:00
wei liu 27046e242b
add timeout for graceful stop (#27326)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-09-26 10:15:25 +08:00
foxspy 5db4a0489e
dynamic index version control (#27335)
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-25 21:39:27 +08:00
foxspy 370b6fde58
milvus support multi index engine (#27178)
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-22 09:59:26 +08:00
congqixia 67a471e09c
Fix gofumpt issue in scann_checker_test.go (#27305)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-22 00:39:26 +08:00
Gao ab2d8dd0a7
Add ScaNN index param checker (#27268)
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-09-21 14:51:25 +08:00
SimFG 26f06dd732
Format the code (#27275)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
yihao.dai fe01d54eca
Set kafka read timeout to 10s and make it configurable (#27238)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-20 22:01:27 +08:00
smellthemoon d1f6825b86
Specify componet ip (#27161)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-09-20 11:49:25 +08:00
yah01 9baff1b81c
Rename error to match the nameing rules (#27214)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-20 10:59:24 +08:00
Xiaofan 6635398a6d
Fix Bin log concurrency by adding a pool (#27189)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-09-19 18:05:22 +08:00
cai.zhang a362bb1457
Support array datatype (#26369)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-09-19 14:23:23 +08:00
XuanYang-cn 10116b85ac
Release before remove when releasing flowgraphs (#27191)
GetAndRemove removes the fg from manager immediately,
while the flowgraph is still releasing. This PR will remove
the fg from flowgraphManager AFTER flowgraphs released.

- Add Remove for ConcurrentMap
- Move collections() into flowgraph manager

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-19 11:53:22 +08:00
yah01 b9ab18d692
Replace error code with merr for RateLimiter (#27176)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-19 10:37:23 +08:00
congqixia cc9974979f
Add staticcheck linter and fix existing problems (#27174)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-19 10:05:22 +08:00
PowderLi 4feb3fa7c6
support azure (#26398)
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-09-19 10:01:23 +08:00
yihao.dai 4b2802033d
Fix datanode panic due to concurrent compaction and delete processing (#27167)
Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-09-18 19:35:29 +08:00
Enwei Jiao 8e17bc3013
Use merr to handle segcore errors (#27137)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-18 14:07:22 +08:00
MrPresent-Han 7939f0e7d5
enable ctx traceId for assignsegment on dc(#26972) (#27108) (#27030)
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-09-18 11:39:20 +08:00
xige-16 488b423e1b
Fix high cpu usage caused by proto.size (#27054)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-09-17 20:55:21 +08:00
yah01 0f2883ef9d
Optimize the generic zero value (#27102)
allocate by declare,
this make it possible to allocate the value in stack,
faster and less presure for GC

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-14 19:05:20 +08:00
congqixia f0d0651989
Do not reset connection immediately if grpc code is `Canceled` or `DeadlineExceeded` (#27014)
We found lots of connection reset & canceled due to recent retry change
Current implementation resets connection no matter what the error code is
To sync behavior to previous retry, skip reset connection only if cancel error happens too much.

Also adds a config item for minResetInterval for grpc reset connection

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-13 15:01:18 +08:00
chyezh 791e6ef6c6
[Improvement] add pulsar metrics and fix timeout (#26907)
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2023-09-13 12:03:19 +08:00
yah01 00c65fa0d7
Refine QueryNode errors (#27013)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-12 16:07:18 +08:00
yah01 45d9fb5929
Fix wrong error codes & names (#26904)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-12 14:41:18 +08:00
congqixia 9e0977622d
Remove config content output before credential mask implemented (#26993)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-11 19:33:17 +08:00
congqixia 44f308fff5
Handle nil status in `merr.OK` (#26940)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-08 18:43:16 +08:00
Xu Tong 9166011c4a
Add float16 vector (#25852)
Signed-off-by: Writer-X <1256866856@qq.com>
2023-09-08 10:03:16 +08:00
XuanYang-cn 7f1ae35e72
Add timeout in dispatcher, AsConsumer and Seek (#26686)
See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-08 09:51:17 +08:00
congqixia af5c01082b
Refine delegator lifetime control (#26881)
- Add SafeChan interface in lifetime package
- Embed SafeChan into  interface
- Replace private lifetime struct in delegator package with
- Refine delegator on-going task lifetime control and wait all accepted task done
- Fix potential goroutine leakage from  if delegator closed concurrently

/kind improvement

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-07 10:11:15 +08:00
yiwangdr 337edc321b
tikv integration (#26246)
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2023-09-07 07:25:14 +08:00
SimFG 28681276e2
Improve the retry of the rpc client (#26795)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-06 17:43:14 +08:00
congqixia 7e2fa55440
Fix misuse of waitgroup in metrics mutex ut (#26865)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-06 12:39:14 +08:00
Enwei Jiao fb0705df1b
Decouple basetable and componentparam (#26725)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-05 10:31:48 +08:00
wei liu 1097776477
stop heartbeat if reach heartbeat limit (#26728)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-09-04 17:51:48 +08:00
XuanYang-cn b2e7cbdf4b
Remove TimeTravel in compactor (#26785)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-04 17:41:48 +08:00
zhagnlu 7056e9c0f7
Increase minio log level to avoid unnecessary log (#26776)
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-09-04 17:25:48 +08:00
yihao.dai 4340cbfba2
Merge syncCPLagTooBehind policy into syncPeriodically policy (#26713)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-04 14:37:47 +08:00
yah01 3349db4aa7
Refine errors to remove changes breaking design (#26521)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-04 09:57:09 +08:00
congqixia e8f1b1736e
Remove log.Error(err.error())-style log (#26783)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-01 13:09:01 +08:00
smellthemoon e56b0018e4
Remove merr mask logic to make it ok to compare resp.status with origin error (#26773)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-09-01 11:09:00 +08:00
Enwei Jiao d206ebf220
Change default config files (#26780)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-01 09:19:00 +08:00
Cai Yudong 8dc16b599b
Add binary metric types SUBSTRUCTURE/SUPERSTRUCTURE back (#26766)
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-08-31 20:07:00 +08:00
XuanYang-cn 86e9f1818f
Change default compaction timeout to 15mins (#26757)
See also: #26566

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-08-31 18:07:00 +08:00
Gao 7e36f819b4
Make autoindex config clean (#26732)
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-08-31 10:25:01 +08:00
congqixia d343888f3c
Fix pkg codecov & static-check script (#26720)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-31 09:59:01 +08:00
jaime c603f1c244
Remove mysql metastore (#26633)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-08-29 14:36:26 +08:00
congqixia ee7aef9272
Make pulsar request timeout configurable (#26525)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-23 20:46:23 +08:00
wei liu 7af0f7d90c
avoid concurrent sub/unsub on same channel (#26454)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-23 10:10:22 +08:00
jaime b82fc8d6e7
[skip e2e] Add connection log for etcd and grpc (#26536)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-08-22 21:20:24 +08:00
SimFG d13ca54414
Change the default gracefulStopTimeout and only warn when fail to refresh policy cache (#26450)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-08-22 09:04:21 +08:00
cai.zhang 94846995bf
Refine error message for field not exist (#26330)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-08-21 11:52:20 +08:00
yihao.dai c3128aaef3
Accelerate server id interceptor validation (#26468)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-08-20 21:24:19 +08:00
wayblink c5a1b41f95
Update session ttl to 60s (#26346)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-08-18 22:52:20 +08:00
wei liu 79ccf06cf6
refine proxy to querynode heartbeat interval (#26426)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-18 14:26:19 +08:00
cqy123456 7e3323a518
update dimension range of diskann. (#26405)
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-08-17 20:26:20 +08:00
jaime 4a388cf450
[skip e2e] Align default value with milvus.yaml file (#25619)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-08-17 20:24:19 +08:00
yihao.dai 63b86b32a6
Add server id validation interceptor (#26395)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-08-17 20:20:20 +08:00
wayblink ab8214fef9
Add ut to ensure hash result (#26417)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-08-17 18:26:21 +08:00
XuanYang-cn be30838cf9
Refine some comments (#26401)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-08-17 10:54:18 +08:00
wei liu 4aac7a6642
refine error reason of query/search failed (#26342)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-16 19:38:18 +08:00
wei liu 74133a3996
refine retry on grpc (#26360)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-16 19:36:18 +08:00
yah01 9723787141
Calculate memory usage without page cache (#26389)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-16 17:04:17 +08:00
SimFG 19f210974f
Support to update the hook config in the runtime (#26350)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-08-16 16:03:33 +08:00
SimFG 8b54914287
Add go.mod file for the milvus/pkg package (#26359)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-08-16 14:49:32 +08:00
congqixia 4742049ecf
Fix unstable TestSingleflight unit test (#26363)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-16 10:07:33 +08:00
congqixia f8bcf60e55
Adjust some confusing Warning log to INFO (#26356)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-15 17:49:33 +08:00
cqy123456 376642b933
reset diskann min dim limit (#26344)
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-08-15 16:45:34 +08:00