e3d50a192d
test: wait all tasks finished in import test ( #33674 )
...
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-06-07 10:35:12 +08:00
19936d9994
fix:[skip e2e] make more retries on codecov uploader ( #33683 )
...
fix #32235
inspired by
https://github.com/Kong/kubernetes-testing-framework/blob/main/.github/workflows/tests.yaml#L53-L69
### What is changing:
Introducing the
[wretry.action](https://github.com/Wandalen/wretry.action ) to
automatically retry the
[codecov-action](https://github.com/codecov/codecov-action ) in our CI/CD
pipeline. This change is intended to enhance the robustness of our
workflow by mitigating transient failures in the codecov-action.
### Follow-up changes needed:
This retry mechanism will be removed once the issue
[codecov/codecov-action#926 ](https://github.com/codecov/codecov-action/issues/926 )
is resolved or if GitHub Actions provides a built-in retry feature for
actions.
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
2024-06-07 10:33:52 +08:00
c61fb1eff5
enhance: do check when add not empty logpath ( #33640 )
...
meta only store logid
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-06-07 10:19:51 +08:00
ecee7d90d4
enhance: try to speed up the loading of small collections ( #33570 )
...
- issue: #33569
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-06-07 08:25:53 +08:00
9c2e3259d1
test: add coo format sparse vector in restful test ( #33677 )
...
* add coo format sparse vector
* search data and insert data in the same sparse format or a different
format
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-06-06 18:05:51 +08:00
27cc9f2630
enhance: Support analyze data ( #33651 )
...
issue: #30633
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
2024-06-06 17:37:51 +08:00
cfea3f43cf
fix: Don't sync L0 segments to channel watcher ( #33664 )
...
issue: #33540
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-06 15:59:50 +08:00
86274f70bd
test: improve concurrency and reduce import test execution time ( #33356 )
...
improve concurrency and reduce import test execution time
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-06-06 15:57:58 +08:00
4dd0c54ca0
fix: Fix l0 compactor may cause DN from OOM ( #33554 )
...
See also: #33547
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-06 14:33:52 +08:00
f6e251514f
fix: Write back dbid modification for nonDB id collection ( #33641 )
...
See also #33608
Make `fixDefaultDBIDConsistency` also write back collection dbid
modification when nonDB id collection is found.
This fix shall prevent dropped collections of this kind show up again
after dropping and restart.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-06 14:29:53 +08:00
b69740c8f3
enhance: Remove unnecessary log info during load segment ( #33663 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-06 14:13:50 +08:00
68c9e7db8c
fix: Sync dropped segment for dropped partition ( #33331 )
...
See also: #33330
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-06 10:25:52 +08:00
fd191dd7db
[automated] Update Knowhere Commit ( #33655 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-06 01:53:50 +08:00
feeb869ff9
enhance: Remove compaction plans on the datanode ( #33548 )
...
issue: #33546
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-05 20:27:51 +08:00
703fc73f71
enhance: disk index support binary vector ( #33631 )
...
issue:https://github.com/milvus-io/milvus/issues/22837
related https://github.com/milvus-io/milvus/pull/33575
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-05 19:37:57 +08:00
412ccfbb20
enhance: Refine IndexNode code and ensure compatibility ( #33458 )
...
issue: #33432 , #33183
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-05 19:35:52 +08:00
8ad26093ba
fix: fix load failure ( #33599 )
...
issue: #33533
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-05 19:19:51 +08:00
b9bafc76b4
enhance: access log support print with consistency level ( #33503 )
...
relate: https://github.com/milvus-io/milvus/issues/33563
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-06-05 17:13:50 +08:00
01ce32caa1
enhance: Print more disk quota info ( #33596 )
...
/kind enhancement
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-05 16:51:56 +08:00
bbdf99a45e
fix: Fix import segment size is uneven ( #33605 )
...
The data coordinator computed the appropriate number of import segments,
thus when importing in the data node, one can randomly select a segment.
issue: https://github.com/milvus-io/milvus/issues/33604
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-05 15:41:51 +08:00
545d4725fb
fix: correct get vector data size for bf16/fp16/binary vector ( #33377 )
...
related #22837
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-06-05 14:31:57 +08:00
597f4c5e03
enhance: Make hasMoreResult accurate when hit number larger than limit ( #33609 )
...
See also milvus-io/milvus-sdk-go#756
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-05 11:51:51 +08:00
4ec7cbcae7
test: remove some local config ( #33567 )
...
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-06-05 11:27:50 +08:00
35532a3e7d
fix: Fill stats log id and check validity ( #33477 )
...
1. Fill log ID of stats log from import
2. Add a check to validate the log ID before writing to meta
issue: https://github.com/milvus-io/milvus/issues/33476
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-05 11:17:56 +08:00
05a80f4def
test: refine file dir in import test ( #33600 )
...
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-06-05 10:29:51 +08:00
25080bb455
enhance: Print `UseDefaultConsistency` param in read requests ( #33617 )
...
`UseDefaultConsistency` param is crucial for debugging slow query
problems. It could be confusing when guarantee timestamp is 1 while this
param is not logged
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-05 10:27:50 +08:00
516f39b616
fix: remove date in the end of ubuntu base image version ( #33073 )
...
CVE issues https://github.com/milvus-io/milvus/issues/31997
This PR will fix **32** LOW and **42** MEDIUM CVEs
**ubuntu:focal-20220426 (ubuntu 20.04)**
Total: 99 (UNKNOWN: 0, LOW: 55, MEDIUM: 44, HIGH: 0, CRITICAL: 0)
**ubuntu:focal (ubuntu 20.04)**
Total: 25 (UNKNOWN: 0, LOW: 23, MEDIUM: 2, HIGH: 0, CRITICAL: 0)
---------
Signed-off-by: weizhenkun <zhenkun.wei@qq.com>
Co-authored-by: weizhenkun <zhenkun.wei@qq.com>
2024-06-05 10:25:50 +08:00
ac4f3997ce
enhance: Reconstructing Compaction to possess persistence capability ( #33265 )
...
issue #33586
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-06-05 10:17:50 +08:00
387b7cd7f4
enhance:avoid maintain checkpoint info in sync manager ( #33413 )
...
relate: https://github.com/milvus-io/milvus/issues/32915
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-06-05 10:05:50 +08:00
8858fcb40a
fix: fix loaded entity num is inaccurate ( #33521 )
...
issue: #33520
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-04 20:09:54 +08:00
d610fdf033
enhance: change Allocator to generic ( #33581 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-06-04 19:05:49 +08:00
c6f8a73bb2
enhance: optimize some cache to reduce memory usage ( #33534 )
...
#33533
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-04 14:09:47 +08:00
2422084a29
fix: paramtable cache cause dynamic config non-dynamic ( #33473 )
...
relate: https://github.com/milvus-io/milvus/issues/33461
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-06-04 11:39:46 +08:00
ac5e098e13
feat: Add e2e test cases for GoSDK ( #33378 )
...
- Add e2e test cases for Go `milvusclient`
- Fix client SparseEmbedding to vector
issue: #33419
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2024-06-04 10:57:47 +08:00
7f4698f4a7
enhance: Use map PK to timestamp in buffer insert ( #33566 )
...
Related to #27675
Store pk to minimal timestamp in `inData` instead of bloom filter to
check whether some delete entry hit current insert batch
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-04 10:07:48 +08:00
d25c755480
[automated] Update Knowhere Commit ( #33573 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-04 01:55:46 +08:00
22a059d4de
enhance: update dependency for blobloom ( #33565 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-03 21:59:46 +08:00
44d7e03e56
fix: reset the RootCoordQuotaStates metric before recording this metric ( #33553 )
...
- issue: #33539
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-06-03 21:09:47 +08:00
d0a0eac0a4
enhance: adding virtual resource allocator ( #33508 )
...
See #33559
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-06-03 19:45:46 +08:00
2b285e5573
fix: Wrap init segcore tracing with golang timeout ( #33494 )
...
See also #33483
Wrap `C.InitTrace` & `C.SetTrace` with timeout preventing otlp
initializtion hangs forever when endpoint is not set correctly
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-03 19:25:51 +08:00
34c6a989ab
enhance: Avoid load bf in delegator when qn worker has no more memory ( #33557 )
...
query coord send load request to delegator, delegator load bf first,
then forward load request to qn worker. but when qn worker has no more
memory, it will return load failed immediatelly. then delegator roll
back the loaded bf. query coord wil retry the load request, and
delegator will load and roll back bf again and again.
this PR delay the loading bf step until load segment succeed in worker.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-03 19:23:45 +08:00
180d754158
fix: speed up segment lookup via channel name in datacoord ( #33530 )
...
issue: #33342
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-06-03 14:47:47 +08:00
7a2127b09f
enhance: Avoid redundant meta operations of import ( #33518 )
...
issue: https://github.com/milvus-io/milvus/issues/33513
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-03 14:15:53 +08:00
0382628668
enhance: Add more tracing for l0 compactor ( #33435 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-03 10:19:49 +08:00
2c7bb0b8ac
fix: replace removeWithPrefix with remove to avoid delete redundantly ( #33328 )
...
#33288
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-05-31 18:05:45 +08:00
c6a1c49e02
enhance: Use Blocked Bloom Filter instead of basic bloom fitler impl. ( #33405 )
...
issue: #32995
To speed up the construction and querying of Bloom filters, we chose a
blocked Bloom filter instead of a basic Bloom filter implementation.
WARN: This PR is compatible with old version bf impl, but if fall back
to old milvus version, it may causes bloom filter deserialize failed.
In single Bloom filter test cases with a capacity of 1,000,000 and a
false positive rate (FPR) of 0.001, the blocked Bloom filter is 5 times
faster than the basic Bloom filter in both querying and construction, at
the cost of a 30% increase in memory usage.
- Block BF construct time {"time": "54.128131ms"}
- Block BF size {"size": 3021578}
- Block BF Test cost {"time": "55.407352ms"}
- Basic BF construct time {"time": "210.262183ms"}
- Basic BF size {"size": 2396308}
- Basic BF Test cost {"time": "192.596229ms"}
In multi Bloom filter test cases with a capacity of 100,000, an FPR of
0.001, and 100 Bloom filters, we reuse the primary key locations for all
Bloom filters to avoid repeated hash computations. As a result, the
blocked Bloom filter is also 5 times faster than the basic Bloom filter
in querying.
- Block BF TestLocation cost {"time": "529.97183ms"}
- Basic BF TestLocation cost {"time": "3.197430181s"}
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-31 17:49:45 +08:00
322a4c5b8c
enhance: Remove StringPrimaryKey to reduce unnecessary copy and function call cost ( #33486 )
...
issue: #33497
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-31 15:41:45 +08:00
6b51bae5be
test: add different nq for restful v2 test ( #33456 )
...
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-05-31 13:57:45 +08:00
3336b91ce6
test: add channel exclusive balance test and resource group test ( #33093 )
...
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-05-31 13:55:52 +08:00
4159a4d5d7
test: update nightly ci ( #33430 )
...
update nightly ci
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-05-31 13:54:07 +08:00
zhuwenxing
sammy.huang
smellthemoon
SimFG
cai.zhang
XuanYang-cn
congqixia
wei liu
sre-ci-robot
cqy123456
zhagnlu
aoiasd
yihao.dai
Gao
GenkenWei
zhenshan.cao
jaime
Bingyi Sun
ThreadDao
Ted Xu
yiwangdr