1e3e34d149
enhance: [ci] organize values file in the form of milvus deployment options ( #35832 )
...
https://github.com/milvus-io/milvus/issues/35842
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-30 13:07:15 +08:00
a3f2f044d6
fix: not set nullable when stream writer write headers ( #35799 )
...
#35802
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-29 20:59:00 +08:00
99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode ( #35406 )
...
issue: #33285
- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-29 10:03:08 +08:00
b75b25794b
enhance: [skip e2e] Fix test script always skip pip install ( #35766 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-28 17:01:00 +08:00
bb9bed64a3
enhance: new nightly ci to support testing distributed-streaming ( #35672 )
...
issue: https://github.com/milvus-io/milvus/issues/35704
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-27 14:24:59 +08:00
0e7877d413
fix: [skip e2e]unstable l0 it ( #35612 )
...
See also: #35617
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-26 18:53:04 +08:00
a90133cdf5
test: Optimize sparse vector case ( #35661 )
...
/kind improvement
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2024-08-26 14:54:57 +08:00
f756f01445
test: update test cases ( #35640 )
...
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-08-26 12:00:58 +08:00
5c245d51c4
enhance: Refresh proxy cache after restore rbac meta ( #35635 )
...
issue: #35443
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-22 19:09:01 +08:00
570a8879e7
test: update gosdk case for valid empty sparse vector ( #35621 )
...
/kind improvement
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2024-08-22 10:54:56 +08:00
ed4eaffc9d
enhance: add csv support for bulkinsert ( #34938 )
...
See this issue for details: #34937
---------
Signed-off-by: OxalisCu <2127298698@qq.com>
2024-08-21 17:47:01 +08:00
ba6db117e3
enhance: add some integration test in null ( #35599 )
...
#31728
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-21 17:44:56 +08:00
7ac339ac64
test: Update init collection method ( #35596 )
...
Related issue: #32653
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-08-21 09:22:56 +08:00
80a7c78f28
enhance: import supports null in parquet and json formats ( #35558 )
...
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-20 16:50:55 +08:00
031ee6f155
enhance: support httpv1/v2 throttle and add it for httpV2( #35350 ) ( #35470 )
...
related: #35350
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-08-20 16:16:55 +08:00
249dc4d9eb
test: Add tests for upsert with auto id ( #35556 )
...
Related issue: #34668
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-08-20 14:20:56 +08:00
e09dc3be58
enhance: Mark query node as read only after suspend ( #35492 )
...
issue: #34985 #35493
after querynode has been suspended, it's not allow to load
segment/channel on it, which means the node is read only. to be
compatible with resource group design, after query node has been
suspend, we remove it from it's original resource group, make it a read
only query node in replica. then two things will happens:
1. it's original resource group will be lacking of query nodes, query
coord will assign new node to it.
2. querycoord will try to move out all segments/channels after querynode
has been suspended
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-20 14:02:54 +08:00
22ced010cd
enhance: make configure load param feature be compatible with old sdk ( #35520 )
...
issue: #31570 #35521
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-20 10:30:55 +08:00
6e29d713b3
enhance: [skip e2e]dockerfile for milvus-sdk-go test ( #35426 )
...
https://github.com/milvus-io/milvus/issues/35567
---------
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-20 09:48:55 +08:00
b4743b4ca8
enhance: [GoSDK] Sync latest milvus proto and mockery ( #35511 )
...
Related to #35443
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-19 16:22:55 +08:00
967f38672a
enhance: Add integration tests for l0 ( #35429 )
...
See also: #34796
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-19 10:56:54 +08:00
635e9ddb0e
test: fix go sdk cases parallel context deadline ( #35519 )
...
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2024-08-16 20:32:53 +08:00
f4a91e135b
enhance: Allow empty sparse row ( #34700 )
...
issue: #29419
* If a sparse vector with 0 non-zero value is inserted, no ANN search on
this sparse vector field will return it as a result. User may retrieve
this row via scalar query or ANN search on another vector field though.
* If the user uses an empty sparse vector as the query vector for a ANN
search, no neighbor will be returned.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-08-16 14:14:54 +08:00
1d49358f82
enhance: Add BackupRBAC/RestoreRBAC API to enable rbac backup ( #35444 )
...
issue: #35443
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-16 10:10:53 +08:00
344dc6a9f8
enhance: enable to set load config in cluster level ( #35169 )
...
issue: #35170
This PR enable to set load configs in cluster level, such as replicas
and resource groups. then when load collections will use the load
config.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-07 12:38:21 +08:00
c19fe95154
fix: support string match for hybrid and bitmap index ( #35294 )
...
#34841
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-07 09:54:22 +08:00
a62118ca33
test: update cases for index describe and upsert with autoID ( #35191 )
...
issue: #33419
---------
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2024-08-05 18:00:16 +08:00
d8668fe405
enhance: upgrade go version to 1.21.11 ( #35257 )
...
issue: #35142
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-08-05 16:12:15 +08:00
4b553b0333
enhance: revert remove duplicated pk function ( #35103 )
...
issue: #34778
Revert "fix: fix query count(*) concurrently"
Revert "enhance: mark duplicated pk as deleted "
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-05 10:48:17 +08:00
eb5461905b
[automated] Update Pytest image changes ( #35244 )
...
Update Pytest image changes
See changes:
241c71fddesre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-05 09:24:33 +08:00
241c71fdde
enhance: use docker compose instead of docker-compose ( #35208 )
...
https://github.com/milvus-io/milvus/issues/35209
---------
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-02 19:32:32 +08:00
db06b86594
enhance: Sync otlp dependency version to fix security issue ( #35192 )
...
Related to #34434
otelgrpc DoS vulnerability due to unbound cardinality metrics
https://github.com/milvus-io/milvus/security/dependabot/91
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-02 16:22:20 +08:00
196a7986b3
enhance: Change the fixed value to a ratio for clustering segment size ( #35076 )
...
issue: #34495
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-01 22:04:14 +08:00
a642a26ed4
enhance: Resolve ChunkFileWriter lint issue ( #35166 )
...
See also #34483
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-01 16:52:13 +08:00
5bbb1c201c
enhance:support l2 single compaction ( #34935 )
...
#34928
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-01 14:36:13 +08:00
dfda9f0478
enhance: Add depguard rules to ban deprecated proto lib ( #35140 )
...
See also #34394 #34252
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-01 10:01:49 +08:00
6106a48acb
fix: upsert result use the previous pk ( #34672 )
...
#34668
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-31 15:25:51 +08:00
de8a266d8a
enhance: Enable linux code checker ( #35084 )
...
See also #34483
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 15:53:51 +08:00
a8a4779749
fix: fix parse plan proto failed for search type ( #34944 )
...
#25848
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-29 21:19:49 +08:00
86322e0468
fix: fix query count(*) concurrently ( #35007 )
...
#34778
#34849
fix two problems:
1. count(*) incorrect, if growing insert duplicated (pk, timestamp)
pairs that pk and timestamp all same, need to keep just one pair.
2. count(*) may core dump, if get_real_count interface get snapshot and
do mvcc at not consistency status, mainly happens under concurrency.
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-29 19:53:50 +08:00
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 ( #34394 )
...
issue: #34252
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
2372452fac
enhance: Optimized the GC logic to ensure that memory is released in time ( #34949 )
...
issue: #34703
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-28 23:53:47 +08:00
783f9d9c33
fix: Unify hook singleton implementation in proxy ( #34887 )
...
Related to #34885
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 18:07:53 +08:00
4ee6c69217
enhance: Add Segment Level in milvus segment info APIs ( #34763 )
...
See also #34746
This PR add segment level field in response of
`GetPersistentSegmentInfo` and `GetQuerySegmentInfo`
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 10:01:46 +08:00
482373b3d3
enhance: Sync go e2e package proto dependency ( #34913 )
...
Go e2e package point to a lower verion of milvus-proto and may cause
compile error locally.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-23 16:09:43 +08:00
4c45bc412f
enhance: Add integration test for clustering compaction ( #34881 )
...
issue: #34792
---------
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:13:43 +08:00
c339df26fc
enhance: refine clustering compaction basic it ( #34793 )
...
#34792
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-22 11:27:51 +08:00
b22e549844
enhance: Rename config of sealing by growing segmetns size ( #34787 )
...
/kind improvement
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-19 20:27:41 +08:00
4939f82d4f
enhance: Seal by total growing segments size ( #34692 )
...
Seals the largest growing segment if the total size of growing segments
of each shard exceeds the size threshold(default 4GB). Introducing this
policy can help keep the size of growing segments within a suitable
level, alleviating the pressure on the delegator.
issue: https://github.com/milvus-io/milvus/issues/34554
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-17 21:45:41 +08:00
acb33bba4d
enhance: Preserve fixed-size memory in delegator node for growing segment. ( #34596 )
...
issue: #34595
When consuming insert data on the delegator node, QueryCoord will move
out some sealed segments to manage its memory usage. After the growing
segment gets flushed, some sealed segments from other workers will be
moved back to the delegator node. To avoid the frequent movement of
segments, we estimate the maximum growing row count and preserve a
fixed-size memory in the delegator node.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-15 20:51:46 +08:00
yellow-shine
smellthemoon
Zhen Ye
congqixia
XuanYang-cn
ThreadDao
nico
wei liu
OxalisCu
yanliang567
Chun Han
Buqian Zheng
zhagnlu
sthuang
sre-ci-robot
cai.zhang
wayblink
yihao.dai