milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
19,505 Commits
176 Branches
157 Tags
571 MiB
Commit Graph

602 Commits (2.4-hotfix)

Author SHA1 Message Date
xige-16 0a70e8b601
enhance: Remove multiple vector field limit (#27827) 
issue: https://github.com/milvus-io/milvus/issues/25639

/kind improvement
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-12-28 16:40:46 +08:00
XuanYang-cn 4b406e5973
enhance: Add CompactionTaskNum metrics (#29518) 
See also: #27606

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-12-28 15:46:55 +08:00
Gao 8a4c0d4a3f
enhance: compile go assembly with build constraints for different arch (#29515) 
issue #28657 
follow https://pkg.go.dev/cmd/go#hdr-Build_constraints to compile go
assembly with different cpu arch

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-12-28 15:40:47 +08:00
Gao 8a630f733a
enhance: add new optimize param for queryhook (#29495) 
add a flag to indicate if we use search param optimizations, default is
on

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-12-28 10:04:46 +08:00
XuanYang-cn fe04598900
enhance: Add compaction type label to metrics (#29485) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-12-27 15:56:48 +08:00
wei liu 839a72129e
fix: Auto balance param can't be updated by dynamic (#29501) 
This PR fixed that auto balance param can't be updated by dynamic

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-27 14:30:53 +08:00
aoiasd a76e3b2813
Refine delete by expression for forbid proxy dml task scheduler hang (#29340) 
relate: https://github.com/milvus-io/milvus/issues/29146

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-12-26 19:52:48 +08:00
MrPresent-Han 7c7003bff6
enhance:refine the range of chunk size config value(#29388) (#29389) 
related: #29388

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-12-26 17:36:46 +08:00
aoiasd 9e6da45497
fix: Use uber atomic instead sync/atomic which only supported after go v1.20 (#29377) 
relate: https://github.com/milvus-io/milvus/issues/29376

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-12-26 16:06:49 +08:00
cqy123456 4c979538a4
enhance: update cagra index params in config and add params check (#29045) 
issue:https://github.com/milvus-io/milvus/issues/29230
this pr do two things about cagra index:
 a.milvus yaml config support gpu memory settings

 b.add cagra-params check

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
Co-authored-by: yusheng.ma <yusheng.ma@zilliz.com>
 2023-12-26 11:04:47 +08:00
aoiasd ac23e67f07
enhance: add default file path for access log (#29460) 
relate:https://github.com/milvus-io/milvus/issues/29459

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-12-26 10:38:45 +08:00
MrPresent-Han bd3bde82f0
fix iterator lose data for duplicted result(#29406) (#29451) 
related: #29406

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-12-25 23:28:47 +08:00
PowderLi 13988cbde7
fix: Partition related privileges should belong to Collection (#29428) 
issue: #29427

it will try to find privilege_name from the object_type, while grant a
privilege to a role

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-12-24 20:32:45 +08:00
SimFG dd9c61831d
enhance: Support to get the param value in the runtime (#29297) 
/kind improvement
issue: #29299

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-12-22 18:36:44 +08:00
XuanYang-cn 7a6aa8552a
fix: add back existing datanode metrics (#29360) 
See also: #29204

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-12-22 14:20:43 +08:00
yah01 a0e1a1eb31
feat: support enable/disable mmap for index (#29005) 
support enable/disable mmap for index, the user could alter the index's
mode by `AlterIndex` method
related: https://github.com/milvus-io/milvus/issues/21866

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-21 18:07:24 +08:00
XuanYang-cn 48f506b077
fix: Seperate msgstream initctx with lifecycle ctx (#29308) 
The close of msgstream should be controlled by Close() not ctx from
outside when init clients

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-12-21 11:56:45 +08:00
wei liu fcba1c0d9e
fix: Rename invalid parameterutil package path (#29334) 
This PR renames the invalid parameterutil package path

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-21 11:40:42 +08:00
wei liu cb0676153f
enhance: add metrics for stopping querynode balance progress (#29201) 
This PR add three metrics to track the stopping balance progress.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-21 10:14:52 +08:00
yihao.dai f457b9f7c9
fix: Return time tick delay error and refine quota error messages (#29289) 
This pr:
1. Handles the time tick delay error when converting old error codes to
milvus errors.
2. Enhances quota error messages by eliminating "force deny" and
substituting it with "quota exceeded."

issue: https://github.com/milvus-io/milvus/issues/29288

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-12-20 22:52:44 +08:00
aoiasd 46a937ac43
fix: msgstream memory leak caused by config event don't deregister (#29266) 
relate: https://github.com/milvus-io/milvus/issues/28620

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-12-20 20:32:44 +08:00
XuanYang-cn 7d32861d53
enhance: Add channel check interval params (#29311) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-12-20 15:06:42 +08:00
aoiasd 010d8362ad
fix: datanode L0 segment num wrong (#29050) 
relate:  https://github.com/milvus-io/milvus/issues/27675

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-12-20 11:20:44 +08:00
congqixia cbf0f9c527
enhance: change cp metric to absolute unix ts (#29328) 
See also #29327

Change channel checkpoint metrics to unix seconds instead of checkpoint
timestamp lag value

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-20 11:04:45 +08:00
PowderLi bcd6865b29
enhance: add 3 builtin roles (#28961) 
issue: #28960 [milvus-proto
#212](https://github.com/milvus-io/milvus-proto/issues/212)

add new configuration: builtinRoles
user can define roles in config file: `milvus.yaml`

there is an example:
1. db_ro, only have read privileges, include load
2. db_rw, read and write privileges, include create/drop/rename
collection
3. db_admin, not only read and write privileges, but also user
administration

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-12-18 14:28:41 +08:00
zhagnlu a602171d06
enhance: Refactor runtime and expr framework (#28166) 
#28165

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-12-18 12:04:42 +08:00
wei liu 820ee692fc
enhance: Add config for querycoord auto balance channel (#29231) 
issue: #23726
This PR add control config to querycoord's background auto balance
channel operation

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-18 10:00:40 +08:00
congqixia 88b4b8b77c
enhance: make segments SQPool & LoadPool resizable (#29239) 
See also #29223

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-15 20:22:39 +08:00
congqixia 4731c1b0d5
enhance: make SyncManager pool size refreshable (#29224) 
See also #29223

This PR make `conc.Pool` resizable by adding `Resize` method for it. 
Also make newly added datanode `MaxParallelSyncMgrTasks` config
refreshable

---------

Signed-off-by: Congqi.Xia <congqi.xia@zilliz.com>
 2023-12-15 09:58:43 +08:00
congqixia 25a4525297
enhance: Change sync manager parallel config item (#29216) 
Since the sync manager is global in datanode now, the old
`maxParallelSyncTaskNum` does not fit into current implementation
anymore.

This PR add a new param item for sync mgr parallel control and enlarge
default value

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-14 20:46:41 +08:00
congqixia 9407be6eb2
enhance: Add a config item for partition name as regexp feature and disable it by default (#29154) 
See also #29177
Add a config item for partition name as regexp feature and disable it by
default

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-14 10:04:38 +08:00
aoiasd b5ee563914
fix: accesslog can not print search expression (#28899) 
relate: https://github.com/milvus-io/milvus/issues/28893

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-12-13 18:50:42 +08:00
Jiquan Long 520c07ec9d
fix: autoindex panic with flat index type (#29071) 
issue: https://github.com/milvus-io/milvus/issues/29048

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-12-12 22:08:46 +08:00
Jiquan Long e5f155612a
fix: txn control not applied to snapshot.Remove (#29124) 
issue: #29123

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-12-12 14:48:38 +08:00
wayblink 51f870da7e
feat: Introduce channelCheckpointUpdater to reduce goroutine use in ttNode (#28570) 
/kind enhancement

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-12-12 13:48:42 +08:00
wei liu 69a71e11d1
enhance: Add config for auto upgrade segment index (#29112) 
This PR add config for auto upgrade segment index to index engine's
version.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-12 11:56:37 +08:00
Enwei Jiao 0e65e90338
enhance: Support otlp with insecure (#29115) 
issue: https://github.com/milvus-io/milvus/issues/28914

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-12-12 11:14:37 +08:00
yah01 0a87724f18
enhance: remove merger for load segments (#29062) 
remove merger as now QueryNode could load segments concurrently
fix #29063

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-12 10:48:37 +08:00
cai.zhang 2b05460ef9
enhance: Make import-related error message clearer (#28978) 
issue: #28976

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-12-08 10:12:38 +08:00
yihao.dai 43abe9ce31
enhance: Delete datanode metric after flowgraph released (#29003) 
`DataNodeFlowGraphBufferDataSize` monitoring metric is leaked. This PR
resolve it.

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-12-07 17:00:35 +08:00
yah01 9b3e06ae86
enhance: add more metrics for level zero segments (#29029) 
- Add SegmentNum metric for level zero segments
- Add level zero segments size metirc

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-07 14:34:35 +08:00
wayblink 6736f65345
feat: skip some empty ttMsg in Datanode flowgraph (#28756) 
/kind feature

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-12-07 01:00:37 +08:00
shaoyue 4a067a4c8c
enhance: Add proxy.ginLogSkipPaths (#28945) 
Fix #28944

/cc @xiaofan-luan

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2023-12-06 10:26:35 +08:00
groot ad1daebc8e
fix: Change ranges of HNSW parameters according to Knowhere (#28875) 
fix: https://github.com/milvus-io/milvus/issues/28860

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-12-05 16:20:42 +08:00
yah01 bf633bb5d7
enhance: refine the retry error (#28573) 
return the last error but not combining all errors, to improve
readability and erorr handling

resolve: #28572

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-30 18:34:32 +08:00
shaoyue 8b2b0d412c
enhance: storeageType default value change to remote (#28792) 
/kind enhancement
/cc @PowderLi

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2023-11-30 11:34:27 +08:00
congqixia f9bb8e9648
enhance: Change const magic number in querycoord to param (#28819) 
See also #28817

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-30 09:06:28 +08:00
cqy123456 3b1b14dd78
fix: update binlog index memory uasge before loading segments (#28528) 
issue: #27678 
when interimIndex = true, memory predict should be update with the
memory usage of binlog index build process.

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-11-29 16:42:27 +08:00
Gao 5bd73d5503
enhance: add efficient distance computations in Go (#28657) 
Related #28656
Add more efficient calc_distance at Go side.

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-11-28 18:20:26 +08:00
XuanYang-cn 606ec77b66
enhance: Unify levelzero segment config in DN (#28720) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-28 18:04:26 +08:00
aoiasd 89d8ce2f73
enhance: refine access log to support format access log by yaml and print name info. (#28319) 
relate: https://github.com/milvus-io/milvus/issues/28086

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-11-28 15:32:31 +08:00
cai.zhang 1b7a503f89
enhance: Revert import support csv format (#28760) 
Revert import support csv format.
issue: #28778

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-28 14:32:27 +08:00
aoiasd ce2436127c
fix: log can not reduce level (#28750) 
relate: https://github.com/milvus-io/milvus/issues/28749

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-11-28 10:02:27 +08:00
congqixia dd682658da
enhance: Add channel checkpoint ts lag metric (#28767) 
See also #28765

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-28 07:00:28 +08:00
SimFG 9c46788d87
enhance: Support to trace restful request and request error (#28685) 
issue: #28348

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-11-27 20:14:26 +08:00
cai.zhang c29b60e18e
enhance: Support Array DataType for bulk_insert (#28341) 
issue: #28272 
Support array DataType for bulk_insert with json, binlog files.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-27 13:50:27 +08:00
aoiasd 8a4cfb7d6a
enhance: add l0 metric and fix datacoord no need drop l0 segment when flush (#28373) 
relate: https://github.com/milvus-io/milvus/issues/27675

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-11-24 15:58:24 +08:00
Bingyi Sun 2843664a20
fix: channel name can not contains '-partition-' (#28689) 
issue: https://github.com/milvus-io/milvus/issues/28675

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-11-24 10:14:24 +08:00
wei liu 4b8d3eaa41
enhance: Increase heartbeat timeout (#28648) 
it's easy to trigger heartbeat timeout after 100ms when standalone cpu
usage reach 100%.
This PR increase the heartbeat timeout param to 2000ms

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-23 19:50:24 +08:00
congqixia a2fe9dad49
enhance: Make etcd kv request timeout configurable (#28661) 
See also #28660
This pr add request timeout config item for etcd kv request timeout
 Sync the default timeout value to same value for etcdKV & tikv config

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-23 19:34:23 +08:00
zhagnlu bc43c5cf34
fix: Panic if minio port is empty (#28461) 
#28460

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-11-23 19:24:25 +08:00
Bingyi Sun 4fedff6d47
feat: integrate storage v2 into the write path (#28440) 
#28378

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-11-23 17:26:24 +08:00
SimFG de13865769
enhance: Add load/release partitions to replicate msg stream (#28399) 
/kind improvement
issue: #25655

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-11-23 15:38:24 +08:00
yah01 c96d07682e
enhance: enhance the accuracy of memory usage (#28554) 
before this, Milvus use container/system's memory info to get the memory
usage, which could be inaccurate.

we allocates the memory by private anon mmap,
then `rss - shared` would be the accurate memory usage

resolve #28553

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-23 15:12:23 +08:00
Enwei Jiao 3c3a57cb92
fix: Fix kafka config type error (#28642) 
issue https://github.com/milvus-io/milvus/issues/28588

---------

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-11-23 14:36:22 +08:00
yah01 bfccfcd0ca
enhance: refine error messages (#28424) 
- Split the simple reason and full detail
- Refine existing error messages
related: #28422

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-21 17:02:24 +08:00
XuanYang-cn e88bbaac24
feat: Add universal levelzero segment switch (#28483) 
See also: #27349

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-20 19:04:22 +08:00
PowderLi 7029797452
Improve HTTP server include metrics and RESTful API (#28045) 
/kind improvement
issue: #27653

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-19 17:14:20 +08:00
congqixia f1fc19e8a9
enhance: Add unittest for `config.EventDispatcher` (#28552) 
These is no enough unittest case for EventDispatcher, see also #28540. 
This PR add unit test case for all methods for EventDispatcher
Related to #28538

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-18 19:12:20 +08:00
congqixia c948a437a9
fix: protect `EventDispatcher` map with mutex (#28540) 
Add mutex protection for `EventDispatcher.registry` map 
Fix #28538

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-17 20:32:20 +08:00
yah01 e9ff7ed13d
enhance: remove legacy error related code (#28385) 
/kind improvement

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-14 15:02:18 +08:00
congqixia 371875f650
Fix MQ Factory timeout unit test (#28395) 
This PR change timeout case to a longer duration in case of slow runner
Also change timeout case behavior of using timeout deadline instead of waiting
See also #28394

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-13 21:44:18 +08:00
SimFG cfb6edea61
Support to trace the grpc request (#28349) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-11-13 20:16:18 +08:00
XuanYang-cn f8aa46419a
Add LevelZeroCompaction configs (#28190) 
See also: #27606

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-13 11:18:19 +08:00
yah01 6f59510723
Upgrade go to 1.20 (#24390) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-11-10 07:46:18 +08:00
SimFG 7dda2e8814
Change some log level in the pkg package (#28181) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-11-08 23:34:22 +08:00
zhenshan.cao 2057fdce43
Change default storage type to minio (#28236) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-11-07 21:15:31 +08:00
wei liu 5b45a138b1
disable auto balance when old node exists (#28191) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-07 14:02:20 +08:00
Xiaofan da19e49daf
Support purge old session for standalone (#28184) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-11-06 21:21:42 +08:00
yah01 90e2c63d9e
Fix getting incorrect CPU num (#28146) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-11-06 06:02:16 +08:00
congqixia bf2f62c1e7
Add `WriteBuffer` to provide abstraction for delta policy (#27874) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-04 12:10:17 +08:00
SimFG ef68680639
Fix the unstable resource manager testcase (#28033) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-11-03 11:00:17 +08:00
aoiasd d2727cc0ff
Add write cache for access log (#27792) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-11-03 10:28:17 +08:00
yah01 dc89730a50
Support collection-level mmap control (#26901) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-02 23:52:16 +08:00
wei liu 3b8fbbd91d
avoid sleep to ctx done during retry (#27992) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-01 18:52:24 +08:00
PowderLi 0c0f012e03
add a custom http header: Accept-Type-Allow-Int64 (#27901) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-01 11:42:16 +08:00
Enwei Jiao 8ae9c947ae
Use OpenDAL to access object store (#25642) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-11-01 09:00:14 +08:00
cqy123456 4fbe3c9142
replace loaded binlog with binlog index for search performance (#27673) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-11-01 02:20:15 +08:00
yah01 9658367a3c
Refine chunk manager errors (#27590) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-31 12:18:15 +08:00
yah01 2af46d7333
Increase the ChunkManager request timeout (#28015) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-31 09:06:13 +08:00
MrPresent-Han ccaec44930
enable parameter for dataCoord broker timeout(#26972) (#27825) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-10-28 06:14:11 +08:00
yihao.dai f9c630247d
Construct plan directly when search with vector output (#27928) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-10-26 19:30:10 +08:00
congqixia 8ab01e5a14
Set ElementType for empty array FieldData (#27924) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-25 20:08:10 +08:00
Gao c8415453a1
Support prepare config for index (#27920) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-10-25 19:54:12 +08:00
aoiasd 53246b1b38
Set accesslog default to close and use stdout (#27891) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-10-25 10:30:10 +08:00
wei liu 40723a292e
reduce compact parallel task num (#27899) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-10-25 09:40:12 +08:00
jaime 4640928280
Fix initialization and backward compatibility issue for GRPC compression (#27894) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-10-24 22:14:14 +08:00
congqixia ee184a709e
Bump Milvus Version & milvus-proto/go-api to v2.3.2 (#27893) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-24 21:02:11 +08:00
jaime 6749957e71
Refine RPC call in unwatch drop channel (#27864) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-10-24 17:46:15 +08:00
congqixia 93a877f55e
Make qcv2 target&leader observer execute in parallel (#27844) 
- Add `taskDispatcher` to submit and run task async safely
- Change `LeaderObeserver` and `TargetObserver` schedule and manual check action to submitting task into dispatcher
- Fix logic problem in collection observer when manual check return false

See also #27494

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-24 10:14:11 +08:00
Xiaofan 2ea7579dbb
Reduce rpc size for GetRecoveryInfoV2 (#27483) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-10-23 21:44:09 +08:00
zhagnlu 6060dd7ea8
Add chunk manager request timeout (#27692) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-10-23 20:08:08 +08:00
congqixia 4a6790b63a
Make KeyLock generic (#27854) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-23 17:40:11 +08:00
SimFG 086220dd15
Add the mock factory of msg stream (#27834) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-10-20 20:14:08 +08:00
SimFG 9b0ecbdca7
Support to replicate the mq message (#27240) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-10-20 14:26:09 +08:00
zhenshan.cao 020ad9a6bc
Rectify wrong exception messages associated with Array datatype (#27769) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-10-19 17:24:07 +08:00
smellthemoon 4b0ec156b3
Set channel work pool size in datanode (#27728) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-10-19 08:28:08 +08:00
yah01 9467de79c3
Use pool to block exceeded tasks (#27767) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-19 03:30:06 +08:00
zhenshan.cao 2b13078b14
Refine insert when send message (#27491) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-10-18 21:12:08 +08:00
yihao.dai 49b3a12804
Return newly defined merr instead of grpc unimplemented err (#27751) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-10-18 15:32:11 +08:00
congqixia 2f201c25e2
Remove deprecated io/ioutil usage (#27747) 
`io/ioutil` package is deprecated, use `io`,`os` package replacement
also added golangci-lint rule to block future reference

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: guoguangwu <guoguangwu@magic-shield.com>
 2023-10-17 20:32:09 +08:00
Futao Wei 599012a340
Implement SearchByPks path for Search (#25882) 
Signed-off-by: unfode <forrest.futao.wei@gmail.com>
 2023-10-16 15:34:08 +08:00
wayblink e3f2122618
Expose metrics of stanby coordinators (#27698) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-10-16 15:04:09 +08:00
wei liu 43f2452f4b
speed up shard leader cache update frequency (#27640) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-10-16 10:52:10 +08:00
dependabot[bot] 98e5a7391c
Bump golang.org/x/net from 0.10.0 to 0.17.0 in /pkg (#27657) 
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.10.0 to 0.17.0.
- [Commits](https://github.com/golang/net/compare/v0.10.0...v0.17.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-10-14 02:53:35 +08:00
yah01 15746b5b5f
Refine KV errors (#27588) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-13 17:15:34 +08:00
yah01 be980fbc38
Refine state check (#27541) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-11 21:01:35 +08:00
foxspy 2b46bd1f08
fix diskann searchCache param compatibility (#27557) 
Signed-off-by: xianliang <xianliang.li@zilliz.com>
 2023-10-11 16:57:34 +08:00
yah01 2df9908c6a
Map old error code to new error (#27616) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-11 12:43:33 +08:00
smellthemoon a9982aa7c6
Remove topic empty restriction when preCreated topic (#27351) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-10-10 18:53:32 +08:00
yah01 3759857bc5
Refine Proxy errors (#27499) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-09 10:09:33 +08:00
XuanYang-cn 56c94cdfa7
Add channel manager in DataNode (#27308) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-10-08 21:37:33 +08:00
yah01 41495ed266
Improve the error message for getting all indexes of collection (#27389) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-08 21:23:32 +08:00
PowderLi 8d3069b1db
update openssl to 3.1.2 (#27399) 
deal with root path's normalization

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-10-08 19:17:31 +08:00
congqixia 5d558623fe
Add revive sub-lints and fix existing problems (#27495) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-07 20:53:38 +08:00
congqixia 80eb5434d3
Fix var-naming caused by old PRs (#27501) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-07 14:43:32 +08:00
yah01 8394b3a1ec
Block creating new error from status reason (#27426) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-07 11:29:32 +08:00
yah01 63ac43a3b8
Refine errors for import (#27379) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-30 10:31:28 +08:00
yihao.dai f962fc186f
Fix panic due to return nil pulsar client (#27429) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-28 16:19:26 +08:00
yihao.dai 106c17f304
Make read ahead policy in ChunkCache configurable (#27291) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-28 15:47:27 +08:00
congqixia 258e1ccd66
Refine querynode scheduler lifetime (#26915) 
This PR refines scheduler lifetime control:
- Move private tri-state into lifetime package
- Make scheduler block incoming "Add" task
- Make scheduler Stop wait until all previously accepted task done

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-28 10:21:26 +08:00
SimFG c9653b1683
Add some log and improve TestSessionProcessActiveStandBy test case (#27403) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-28 09:35:27 +08:00
MrPresent-Han 505e8509b7
refine the default values for segcore threadpool size (#27406) 
/kind improvement

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-27 20:43:28 +08:00
wei liu 4071132f6a
reload loading collection when qc recover (#27300) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-27 11:43:28 +08:00
SimFG 9fb4c27a90
Fix rootcoord failure to start when reaching the limit of role num (#27361) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-27 11:03:26 +08:00
congqixia a3dd2756cf
Add predicates for TxnKV operations (#27365) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-27 10:21:26 +08:00
yah01 6539a5ae2c
Refine DataCoord status (#27262) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-26 17:15:27 +08:00
cai.zhang 30a6c6072e
Fix bug for loading array data panic (#27343) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-09-26 12:23:26 +08:00
jaime 7119cb29ca
Fix kafka consumer connection leak (#27224) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-09-26 10:31:27 +08:00
wayblink 7dd0be1b2c
Enable bulkinsert binlog data with partitionkey (#27241) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-09-26 10:25:25 +08:00
wei liu 27046e242b
add timeout for graceful stop (#27326) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-26 10:15:25 +08:00
foxspy 5db4a0489e
dynamic index version control (#27335) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-25 21:39:27 +08:00
congqixia 1d76565894
Add metrics for garbage collection (#27303) 
Also fix second metrics usage in compaction

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-22 18:47:25 +08:00
congqixia 670cb386e7
Add back `gocritic` linter and fix related issues (#27289) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-22 10:05:26 +08:00
foxspy 370b6fde58
milvus support multi index engine (#27178) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-22 09:59:26 +08:00
congqixia 67a471e09c
Fix gofumpt issue in scann_checker_test.go (#27305) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-22 00:39:26 +08:00
Gao ab2d8dd0a7
Add ScaNN index param checker (#27268) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-09-21 14:51:25 +08:00
SimFG 26f06dd732
Format the code (#27275) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-21 09:45:27 +08:00
yihao.dai fe01d54eca
Set kafka read timeout to 10s and make it configurable (#27238) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-20 22:01:27 +08:00
smellthemoon d1f6825b86
Specify componet ip (#27161) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-09-20 11:49:25 +08:00
yah01 9baff1b81c
Rename error to match the nameing rules (#27214) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-20 10:59:24 +08:00
Xiaofan 6635398a6d
Fix Bin log concurrency by adding a pool (#27189) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-09-19 18:05:22 +08:00
cai.zhang a362bb1457
Support array datatype (#26369) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-09-19 14:23:23 +08:00
XuanYang-cn 10116b85ac
Release before remove when releasing flowgraphs (#27191) 
GetAndRemove removes the fg from manager immediately,
while the flowgraph is still releasing. This PR will remove
the fg from flowgraphManager AFTER flowgraphs released.

- Add Remove for ConcurrentMap
- Move collections() into flowgraph manager

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-19 11:53:22 +08:00
yah01 b9ab18d692
Replace error code with merr for RateLimiter (#27176) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-19 10:37:23 +08:00
congqixia cc9974979f
Add staticcheck linter and fix existing problems (#27174) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-19 10:05:22 +08:00
PowderLi 4feb3fa7c6
support azure (#26398) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-09-19 10:01:23 +08:00
yihao.dai 4b2802033d
Fix datanode panic due to concurrent compaction and delete processing (#27167) 
Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-09-18 19:35:29 +08:00
Enwei Jiao 8e17bc3013
Use merr to handle segcore errors (#27137) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-18 14:07:22 +08:00
MrPresent-Han 7939f0e7d5
enable ctx traceId for assignsegment on dc(#26972) (#27108) (#27030) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-18 11:39:20 +08:00
xige-16 488b423e1b
Fix high cpu usage caused by proto.size (#27054) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-09-17 20:55:21 +08:00
yah01 0f2883ef9d
Optimize the generic zero value (#27102) 
allocate by declare,
this make it possible to allocate the value in stack,
faster and less presure for GC

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-14 19:05:20 +08:00
XuanYang-cn ea2b909c6d
Fix historm metrics buckets range for index (#26976) 
See also: #26743

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-13 17:21:18 +08:00
congqixia f0d0651989
Do not reset connection immediately if grpc code is `Canceled` or `DeadlineExceeded` (#27014) 
We found lots of connection reset & canceled due to recent retry change
Current implementation resets connection no matter what the error code is
To sync behavior to previous retry, skip reset connection only if cancel error happens too much.

Also adds a config item for minResetInterval for grpc reset connection

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-13 15:01:18 +08:00
chyezh 791e6ef6c6
[Improvement] add pulsar metrics and fix timeout (#26907) 
Signed-off-by: chyezh <ye.zhen@zilliz.com>
 2023-09-13 12:03:19 +08:00
yah01 00c65fa0d7
Refine QueryNode errors (#27013) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-12 16:07:18 +08:00
yah01 45d9fb5929
Fix wrong error codes & names (#26904) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-12 14:41:18 +08:00
congqixia 1aa3a1e067
Refine `NewNmqClient` logic with nats.ConsumeDialer option (#26987) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-11 20:17:17 +08:00
congqixia 9e0977622d
Remove config content output before credential mask implemented (#26993) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-11 19:33:17 +08:00
congqixia 44f308fff5
Handle nil status in `merr.OK` (#26940) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-08 18:43:16 +08:00
Xu Tong 9166011c4a
Add float16 vector (#25852) 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-09-08 10:03:16 +08:00
XuanYang-cn 7f1ae35e72
Add timeout in dispatcher, AsConsumer and Seek (#26686) 
See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-08 09:51:17 +08:00
congqixia af5c01082b
Refine delegator lifetime control (#26881) 
- Add SafeChan interface in lifetime package
- Embed SafeChan into  interface
- Replace private lifetime struct in delegator package with
- Refine delegator on-going task lifetime control and wait all accepted task done
- Fix potential goroutine leakage from  if delegator closed concurrently

/kind improvement

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-07 10:11:15 +08:00
yiwangdr 337edc321b
tikv integration (#26246) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-09-07 07:25:14 +08:00
SimFG 28681276e2
Improve the retry of the rpc client (#26795) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-06 17:43:14 +08:00
congqixia 7e2fa55440
Fix misuse of waitgroup in metrics mutex ut (#26865) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-06 12:39:14 +08:00
Enwei Jiao fb0705df1b
Decouple basetable and componentparam (#26725) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-05 10:31:48 +08:00
congqixia 1a8cf5c415
Organize all mockery generation commands in Makefile (#26826) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-04 21:19:48 +08:00
wei liu 1097776477
stop heartbeat if reach heartbeat limit (#26728) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-04 17:51:48 +08:00
XuanYang-cn b2e7cbdf4b
Remove TimeTravel in compactor (#26785) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-04 17:41:48 +08:00
zhagnlu 7056e9c0f7
Increase minio log level to avoid unnecessary log (#26776) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-09-04 17:25:48 +08:00
yihao.dai 4340cbfba2
Merge syncCPLagTooBehind policy into syncPeriodically policy (#26713) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-04 14:37:47 +08:00
yah01 3349db4aa7
Refine errors to remove changes breaking design (#26521) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-04 09:57:09 +08:00
yihao.dai c6024a32f5
Add rate limit ratio monitoring (#26709) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-03 20:47:02 +08:00
congqixia e8f1b1736e
Remove log.Error(err.error())-style log (#26783) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-01 13:09:01 +08:00
smellthemoon e56b0018e4
Remove merr mask logic to make it ok to compare resp.status with origin error (#26773) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-09-01 11:09:00 +08:00
XuanYang-cn ef75784715
Fix LoadSegmentLatency metric p99 (#26761) 
See also: #26743

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-01 11:07:07 +08:00
Enwei Jiao d206ebf220
Change default config files (#26780) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-01 09:19:00 +08:00
Cai Yudong 8dc16b599b
Add binary metric types SUBSTRUCTURE/SUPERSTRUCTURE back (#26766) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-08-31 20:07:00 +08:00
XuanYang-cn 86e9f1818f
Change default compaction timeout to 15mins (#26757) 
See also: #26566

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-31 18:07:00 +08:00
XuanYang-cn 8d54509e54
Fix CompactionLatency metrics (#26747) 
- Refine compactor logs

See also: #26743

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-31 17:35:03 +08:00
Gao 7e36f819b4
Make autoindex config clean (#26732) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-08-31 10:25:01 +08:00
congqixia d343888f3c
Fix pkg codecov & static-check script (#26720) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-31 09:59:01 +08:00
jaime c603f1c244
Remove mysql metastore (#26633) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-08-29 14:36:26 +08:00
yah01 c3f5856fbc
Fix data race for config with FileSource (#26518) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-24 09:18:24 +08:00
congqixia ee7aef9272
Make pulsar request timeout configurable (#26525) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-23 20:46:23 +08:00
congqixia 145387fdcb
Bump proto go-api to v2.3.0 (#26561) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-23 20:18:23 +08:00
wei liu 7af0f7d90c
avoid concurrent sub/unsub on same channel (#26454) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-23 10:10:22 +08:00
jaime b82fc8d6e7
[skip e2e] Add connection log for etcd and grpc (#26536) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-08-22 21:20:24 +08:00
SimFG d13ca54414
Change the default gracefulStopTimeout and only warn when fail to refresh policy cache (#26450) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-08-22 09:04:21 +08:00