40d34f711e
test: update test cases ( #35867 )
...
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-09-04 10:37:03 +08:00
cc02dc0a55
fix: Handle Input/Output Errors in vsnprintf and snprintf ( #35898 )
...
Fix for Issue: #35897
2024-09-04 08:15:04 +08:00
69b1eea6f2
enhance: [GoSDK] Add load option for field partial load ( #35920 )
...
Related to #35415
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-03 19:27:04 +08:00
9da86529a7
enhance: Add disk filemananger parallel load control to reduce the memory consumption ( #35281 )
...
issue: #35280
add parallel control to limit the memory consumption during index file
loading
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-09-03 18:01:03 +08:00
f68df9a11e
fix: SkipIndex cause segment fault ( #35907 )
...
issue: #35882
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-03 17:15:03 +08:00
f068729a26
test: fix repeated random partition name ( #35805 )
...
/kind improvement
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2024-09-03 17:13:12 +08:00
cb49b32358
test: add bitmap index cases ( #35909 )
...
Signed-off-by: wangting0128 <ting.wang@zilliz.com>
2024-09-03 16:49:03 +08:00
74048ce34f
fix:rename mmap file path to avoid directory conflict ( #35810 )
...
#35784
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-03 16:05:03 +08:00
c84ea5465c
fix: Fix some replicas don't participate in the query after the failure recovery ( #35850 )
...
issue: #35846
querycoord will notify proxy to update shard leader cache after
delegator location changes, but during querynode's failure recovery,
some delegator may become unserviceable due to lacking of segments, and
back to serviceable after segment loaded, so we also need to notify
proxy to invalidate shard leader cache when delegator serviceable state
changes.
This PR will maintain querynode's serviceable state during heartbeat,
and notify proxy to invalidate shard leader cache if serviceable state
changes.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-03 15:39:03 +08:00
6130a85444
enhance: Remove bf from streaming node ( #35902 )
...
Remove bf from streaming node:
1. When watching vchannels, skip loading bloom filters for segments.
2. Bypass bloom filter checks for delete messages, directly writing to
L0 segments.
3. Remove flushed segments proactively after flush.
issue: https://github.com/milvus-io/milvus/issues/33285 ,
https://github.com/milvus-io/milvus/issues/34585
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-03 14:17:02 +08:00
325f1987d9
enhance: rewrite index params for compatibility ( #35788 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-02 19:41:03 +08:00
3698c53a72
enhance: Check load fields for previous loaded collection ( #35905 )
...
Related to #35415
This PR make querycoord report error when load request tries to update
load fields list, which is currently not supported.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-02 18:39:03 +08:00
4641fd9195
enhance: make search groupby stop when reaching topk groups ( #35814 )
...
related: #33544
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-02 18:25:03 +08:00
57422cb2ed
test: add array inverted index function test ( #35874 )
...
/kind improvement
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-09-02 17:49:02 +08:00
b2eb9fe2a7
fix: memory leak in unittest and open the USE_ASAN option when build unittest ( #35855 )
...
issue: #35854
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-02 15:59:04 +08:00
6fd33285e1
fix: Fix compile error ( #35901 )
...
/kind improvement
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-02 14:50:35 +08:00
2c9bb4dfa3
feat: Support stats task to sort segment by PK ( #35054 )
...
issue: #33744
This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-02 14:19:03 +08:00
9d80137698
fix: Check clustering key skip load behavior ( #35865 )
...
feature issue: #35415
See also #35861
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-02 11:17:02 +08:00
576ac2bbed
fix: Fix the reference to a variable after it has been moved ( #35875 )
...
#35607
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-02 10:05:02 +08:00
5ea2454fdf
feat: tantivy tokenizer binding ( #35801 )
...
fix : #35800
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-09-01 17:13:03 +08:00
1413ffe9b1
enhance: Rename preAllocatedSegments ( #35871 )
...
Rename `preAllocatedSegments` to `preAllocatedSegmentIDs` to avoid
confusion.
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-01 17:09:01 +08:00
2e090b2426
enhance: Refine flusher channel management ( #35870 )
...
Change the ChannelTask to ChannelLifetime, only removing it upon
unregistering.
issue: https://github.com/milvus-io/milvus/issues/33285
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-01 17:07:10 +08:00
ef451f5e1f
enhance: Add build-go target ( #35844 )
...
Add a `build-go` target to the Makefile that only compiles Go.
issue: https://github.com/milvus-io/milvus/issues/35611
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-01 17:05:09 +08:00
407fc933e7
fix: bump bytedance/sonic to v1.12.2 to fix compilation error with go 1.23.0 ( #35879 )
...
fixes : #35878
Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
2024-09-01 13:43:01 +08:00
311f860676
enhance: support to drop the role which is related the privilege list ( #35727 )
...
- issue: #35545
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-30 15:17:00 +08:00
1e3e34d149
enhance: [ci] organize values file in the form of milvus deployment options ( #35832 )
...
https://github.com/milvus-io/milvus/issues/35842
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-30 13:07:15 +08:00
ac736ad472
enhance: friendly for docker cache ( #35789 )
...
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-30 13:05:34 +08:00
323400c190
enhance: Enable to write multiple segments in mix compactor ( #35705 )
...
Prevent segments to be written larger than maxSize * expansionRate
See also: #35584
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-30 11:29:01 +08:00
1c3abd1aea
fix: Fix unstable TestFlusher ut ( #35831 )
...
issue: https://github.com/milvus-io/milvus/issues/35830
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-30 10:25:01 +08:00
f9be01ce89
fix: mockery config ( #35834 )
...
issue: #33285
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-30 09:49:00 +08:00
07bb071aaf
enhance: add the unmarshal method for the rbac msg ( #35692 )
...
/kind improvement
- issue: #35391
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-30 07:23:00 +08:00
671112d17b
enhance: add more info to hybrid index log ( #35808 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-29 21:07:04 +08:00
a3f2f044d6
fix: not set nullable when stream writer write headers ( #35799 )
...
#35802
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-29 20:59:00 +08:00
d10aa4626f
enhance: [skip e2e] add make run-test-cpp with support for filter gtest ( #35829 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-08-29 20:03:02 +08:00
4373c6994f
fix: [GoSDK] reset cache after `UsingDatabase` ( #35638 )
...
Related to milvus-io/milvus-sdk-go#809
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 19:41:02 +08:00
69265978bf
fix: Fix arrow go client ( #35819 )
...
issue: https://github.com/milvus-io/milvus/issues/35662
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-29 17:53:01 +08:00
3c8941a88e
enhance: Print log only when rate limit updates ( #35806 )
...
The debug log for "RateLimiter register for rateType" is too frequent
and in e2e cases, the may print 18M times in one run.
This PR make the log be printed only when rate limit is updated.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 15:07:00 +08:00
09ef3f1b4f
fix: Make sure querycoord observers started once ( #35811 )
...
Related to #35809
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 14:45:00 +08:00
70bc0e4538
enhance: add skip 3rdparty env var ( #35736 )
...
issue: #35611
- call `conan install` is too slow for developer, add option to skip it.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-29 14:21:01 +08:00
b3089b5bdc
feat: support range search pagination retains order ( #35738 )
...
issue: #35464
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-08-29 14:09:00 +08:00
73be0ba941
enhance: avoid the coexistence of the old coordinator and the new node/proxy ( #35720 )
...
issue: #35719
In standalone mode, block the start process until the new coordinator is
active to avoid the coexistence of the old coordinator and the new
node/proxy
1. In the start/restart process, the new coordinator will become active
immediately and will not be blocked
2. In the rolling upgrade process, the new coordinator will not be
active until the old coordinator is down, and it will be blocked
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-29 13:55:01 +08:00
e34b0c43a1
enhance: add GitHub Actions to delete spam issues comments ( #35807 )
...
/kind improvement
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-08-29 13:19:00 +08:00
b0ac04d104
enhance: set database properties to restrict read access ( #35745 )
...
issue: #35744
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-29 13:17:01 +08:00
b51b4a2838
fix: try get not exist file after upgrade ( #35740 )
...
https://github.com/milvus-io/milvus/issues/35741
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-29 11:09:01 +08:00
86691656f3
enhance: Change frequent balancer debug log to rated one ( #35749 )
...
"skip balance" log is too frequent in debug level. This PR changes it
into rated on.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 10:07:00 +08:00
985d84d3ee
enhance: Add skip load validation for create collection task ( #35737 )
...
Related to #35415
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 10:05:08 +08:00
99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode ( #35406 )
...
issue: #33285
- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-29 10:03:08 +08:00
ae53ce381b
delete pvc when finished ( #35783 )
...
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-29 09:57:06 +08:00
c5c9ef9876
fix: Check response size before add to counter ( #35778 )
...
Related to #35767
prometheus counter cannot add negative value
when response is not written(say timeout/network broken) panicking may
happen if not check
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 00:05:00 +08:00
1cd9123a6c
Update all contributors
...
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-08-28 12:00:44 +00:00
nico
Abdullah Ahmed
congqixia
foxspy
Zhen Ye
ThreadDao
wt
zhagnlu
wei liu
yihao.dai
Chun Han
zhuwenxing
cai.zhang
Jiquan Long
Yinzuo Jiang
SimFG
yellow-shine
XuanYang-cn
smellthemoon
zhenshan.cao
Patrick Weizhi Xu
jaime
sre-ci-robot