Commit Graph

19996 Commits (0a3d456688e9df297353433a8080b1546ea595a1)

Author SHA1 Message Date
SimFG 1d48d0aeb2
enhance: use different value to get related data size according to segment type (#33017)
issue: #30436

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-14 14:59:33 +08:00
Cai Yudong 4fc7915c70
enhance: unify data generation test APIs (#32955)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-14 14:33:33 +08:00
chyezh 96489b814d
fix: remove busy log (#33042)
issue: #32963

Signed-off-by: chyezh <chyezh@outlook.com>
2024-05-14 14:20:32 +08:00
congqixia 861977ab60
fix: Start `LeaderCacheObserver` before `SyncAll` (#33035)
Related to #33033

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-14 13:25:32 +08:00
XuanYang-cn 40e5c0be70
fix: Enable to balance released standby channel (#32986)
See also: #32879

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-14 11:51:32 +08:00
aoiasd dc058eaf61
fix: should init access logger before init server (#32976)
relate: https://github.com/milvus-io/milvus/issues/32968

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-14 10:47:32 +08:00
zhuwenxing 3871009cf5
test: add timeout for GetMetrics grpc interface (#33024)
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-05-14 10:29:32 +08:00
congqixia 4ae7cabb04
fix: Remove channel when create flowgraph timeout (#33014)
See also #33013

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-14 10:07:33 +08:00
wei liu cba2c7a3be
enhance: clean channel node info in meta store (#32988)
issue: #32910
see also: #32911
when channel exclusive mode is enabled, replica will record channel node
info in meta store, and if the balance policy changes, which means
channel exclusive mode is disabled, we should clean up the channel node
info in meta store, and stop to balance node between channels.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-14 10:05:40 +08:00
yihao.dai a984e46a29
enhance: Remove rootcoord from datanode broker (#32818)
issue: https://github.com/milvus-io/milvus/issues/32827

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-14 10:03:32 +08:00
sre-ci-robot 09cca1fb51
[automated] Update Knowhere Commit (#33030)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-05-14 01:47:32 +08:00
zhuwenxing 9e3f3d99f7
test: add timeout for import testcases (#33006)
add timeout for import testcases

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-05-13 19:35:32 +08:00
XuanYang-cn efdbd8e7c1
enhance: Enable to upload by batch (#32788)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-13 19:07:32 +08:00
XuanYang-cn 29b621f759
fix: Make compactor able to clear empty segments (#32821)
See also: #32553

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-13 18:21:32 +08:00
jaime ba625835bc
enhance: Add metrics for segment index files size (#32979)
issue:#32980

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-05-13 17:59:32 +08:00
chyezh 293f14a8b9
fix: remove redundant replica recover (#32985)
issue: #22288 

- replica recover should be only triggered by replica recover

Signed-off-by: chyezh <chyezh@outlook.com>
2024-05-13 15:25:32 +08:00
foxspy f6777267e3
enhance: add score compute consistency config for knowhere (#32997)
issue: https://github.com/milvus-io/milvus/issues/32583
related: #32584

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-05-13 14:21:31 +08:00
SimFG 4031abd2fa
enhance: change default partition num to 16 when using partition key (#32950)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-13 14:19:31 +08:00
congqixia 12ec3d61d9
fix: Fill deltalog entry num & time range in L0 compactions (#33004)
Resolves #33003

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-13 14:13:31 +08:00
Bingyi Sun 4724779b3b
enhance: remove fallback keys for config generator (#32946)
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-05-13 13:33:31 +08:00
Xiaofan 230c88b1de
enhance: improve reduce performance on very large topk (#32871)
fix #32870
reduce cpu usage for reduce on large k

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-05-13 11:55:31 +08:00
congqixia ef41f80bd0
enhance: Sync go.mod version & dep to main pkg (#32973)
Related to #31293
See also #32951 #32952 #32953 #32954

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-13 10:25:32 +08:00
congqixia 0e5765b116
enhance: Utilize `TestLocations` ability to accelerate write & compaction (#32948)
See also #32642

This PR reuses hash locations for bloom filter prediction utilizing
`storage.Location`, like enhancement #32642.

Also adds a utility struct in storage: `LocationCache` to storage
locations for variable K (numbers of hash functions)

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-13 10:15:32 +08:00
wei liu 8a9a42198d
fix: Proxy crash due to shard leader cache data race (#32971)
issue: #32970
cause InvalidateShardLeaderCache use wrong lock, which may cause data
race in meta cache, then proxy may crash

This PR fixed that use leaderMut when try to access shard leader cache.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-11 14:32:12 +08:00
Xiaofan 36f1ea93a5
enhance: optimize plan parser pool to avoid unnessary recycle (#32869)
fix #32868
plan parser takes too much cpu on high qps,this pr try to avoid create
lexer and parser too freequent

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-05-11 10:51:31 +08:00
Xiaofan b044e5503e
enhance:Improve load speed (#32898)
fix #32897
add memory check when load collection

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-05-11 10:29:31 +08:00
congqixia c0e62e6629
enhance: Use collection default consistency level for restv2 (#32956)
Set `UseDefaultConsistency` to true so that restv2 read API shall use
collection consistency level setting correctly.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-11 00:47:31 +08:00
sre-ci-robot 4577394f4f Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-05-10 12:01:00 +00:00
chyezh 1c84a1c9b6
fix: lru related issue fixup patch (#32916)
issue: #32206, #32801

- search failure with some assertion, segment not loaded and resource
insufficient.

- segment leak when query segments

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-05-10 19:17:30 +08:00
wei liu 25689859a1
fix: Load index metric use wrong time unit (#32935)
issue:#32899

This PR fix the wrong metric value of load index, which introduced by
pr#32567, use wrong time unit for load index metrics

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-10 18:07:30 +08:00
congqixia 244d2c04f6
feat: Add `milvusclient` package and migrate GoSDK (#32907)
Related to #31293

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-10 18:01:30 +08:00
yiwangdr 855192eb3d
fix: sync milvus.yaml (#32920)
issue: https://github.com/milvus-io/milvus/issues/25309

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-05-10 17:29:31 +08:00
wei liu e2332bdc17
enhance: Enable channel exclusive balance policy (#32911)
issue: #32910  
* split replica's node list to channels when create replicas
 * balance nodes among channels when node change happens
 * implement channel level balance, let balance happens in channel level

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-10 17:27:31 +08:00
Gao f7d29118a5
enhance: update knowhere version (#32931)
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-05-10 16:13:31 +08:00
Cai Yudong dc89c6f810
enhance: remove duplicated data generation APIs for bulk insert test (#32889)
Issue: #22837

including following changes:
1. Add API CreateInsertData() and BuildArrayData() in
internal/util/testutil
2. Remove duplicated test APIs from importutilv2 unittest and bulk
insert integration test

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-10 15:27:31 +08:00
yanliang567 82a1e1ebe1
test: Update insert data tests and refactor some data gen functions (#32924)
related issue: #32653

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-05-10 14:57:32 +08:00
wei liu 04a8ec69f6
fix: Segment on stopping query node can't be release successfully (#32929)
issue: #32901
Cause release segment request need be send to delegator, but it need
replica to info find segment's delegator. but the stopping query node
will be marked as read only in replica, then `replica.Contains()` just
return true for rwNode in replica. then it can't get replica info by
stopping query node and release segment will be blocked.

This PR make `replica.Contains()` return true for both roNode and
rwNode.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-10 14:33:30 +08:00
aoiasd cbbfb5b6d6
enhance: Add update milvus yaml command to makefile (#32857)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-10 14:31:37 +08:00
Jiquan Long 0783582e2e
fix: temporarily disable two-phase retrieval when lru is enabled (#32927)
issue: #31822

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-10 14:19:45 +08:00
zhuwenxing e776c807c4
test: add more json datatype in restful test cases (#32930)
add more json datatype in restful test cases

related to https://github.com/milvus-io/milvus/issues/31051

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-05-10 14:11:30 +08:00
Bingyi Sun 5cbf081111
fix: fix index resource estimation (#32842)
issue: #32820

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-05-10 11:53:30 +08:00
Bingyi Sun b7ef8da360
fix: set channel checkpoint to delta position (#32878)
issue: https://github.com/milvus-io/milvus/issues/32853

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-05-10 11:51:30 +08:00
zhuwenxing 9a269f1489
test: add import checker to chaos test (#32908)
add import checker to chaos test

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-05-10 11:43:30 +08:00
Buqian Zheng bb7765cbd6
fix: fix Indexing.Iterator ut: build index with all data at once (#32844)
issue: #32843

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-05-10 11:31:30 +08:00
Bingyi Sun 1e1fba0588
enhance: add lazy load retry configurations (#32848)
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-05-10 10:55:30 +08:00
wei liu c35797c399
enhance: expose DescribeDatabase api in proxy (#32732)
issue: #32707

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-09 22:51:30 +08:00
congqixia efa58ae423
enhance: Utilize coll2replica mapping when getting rg by collection (#32892)
See also #32165

In old `GetResourceGroupByCollection` implementation, it iterates all
replicas to match collection id, which is slow and CPU time consuming.
This PR make it utilize the coll2Replicas mapping by calling
`GetByCollection` and mapping replicas into resource group.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-09 19:37:30 +08:00
congqixia 77fa615772
fix: Make SyncManager callback func ignore nil error (#32891)
introduced by #32865

sync manager callback handler panicked when error is nil

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-09 18:03:31 +08:00
aoiasd 54a51b1236
enhance: Support dynamic config for opentelemetry trace (#32169)
relate: https://github.com/milvus-io/milvus/issues/31940

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-09 17:43:30 +08:00
shaoting-huang ca0cf9b3b1
doc: fix typos in design docs (#32885)
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-05-09 15:43:30 +08:00