50905e0b45
fix: Check string array max length after type matching ( #36449 )
...
issue: #36029
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-27 09:45:14 +08:00
d29e01e284
fix: port listen racing in mix or standalone mode ( #36442 )
...
issue: #36441
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-26 21:23:16 +08:00
7ff41697f9
enhance: make slotUsage a field of CompactionTask ( #36510 )
...
#36509
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-09-26 20:19:14 +08:00
c056620899
fix: Skip unnecessary query node health check in proxy ( #36491 )
...
issue: #36490
After the query node changes from a delegator to a worker, proxy should
skip this querynode's health check.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-26 18:13:15 +08:00
55be814a58
enhance: make TransferChannel/TransferSegment idempotent ( #36489 )
...
issue: #36488
when call TransferChannel/TransferSegment, querycoord will generate and
submit balance task to scheduler, if segment/channel's task already
exist in scheduler, submit task will failed.
to make TransferChannel/TransferSegment idempotent, we should skip to
submit if task already exist in scheduler.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-26 18:11:23 +08:00
5dfa1c3397
fix: Segment unbalance after many times load/release ( #36537 )
...
issue: #36536
query coord use `segmentTaskDeleta/channelTaskDelta` to measure the
executing workload for querynode in scheduler, and we maintains the
`segmentTaskDeleta/channelTaskDelta` by `scheulder.Add(task)` and
`scheduler.remove(task)`, but `scheduler.remove(task)` has been called
in unexpected way, which cause a wrong
`segmentTaskDeleta/channelTaskDelta` value and affect the segment assign
logic, causes segment unbalance.
This PR moves to compute the `segmentTaskDeleta/channelTaskDelta` when
access, to avoid the wrong value affect.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-26 15:13:15 +08:00
0799d927c6
fix:fix term expr overflow bug ( #36525 )
...
#36520
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-26 15:01:14 +08:00
b60164b882
enhance: support null in bulk insert of binlog to help backup null ( #36526 )
...
https://github.com/milvus-io/milvus/issues/36341
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-26 14:35:14 +08:00
c94b69c2f6
enhance: update the expr version and format the expr http response ( #36406 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-26 14:27:20 +08:00
4493aa2142
fix: querycoord collection num metric ( #36471 )
...
related to: #36456
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-09-26 14:23:13 +08:00
4fd9b0a8e3
enhance: Return segment id hint in QueryStream response ( #36487 )
...
Related to #36482
This PR reuses `SealedSegmentIDsRetrieved` field in `RetrieveResults`
struct to store segment id hint.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-26 10:13:14 +08:00
9e8cafcbe2
enhance: Skip loading bf in datanode ( #36367 )
...
Skip loading bf in datanode:
1. When watching vchannels, skip loading bloom filters for segments.
2. Bypass bloom filter checks for delete messages, directly writing to
L0 segments.
3. Remove flushed segments proactively after flush.
issue: https://github.com/milvus-io/milvus/issues/34585
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-26 10:11:15 +08:00
447e326629
[automated] Update Knowhere Commit ( #36527 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-09-26 01:15:13 +08:00
8495bc6bbc
fix: fix broken Sparse Float Vector raw data mmap ( #36183 )
...
issue: https://github.com/milvus-io/milvus/issues/36182
* improved `Column.h` to make the code much more readable and
maintainable, and added detailed comments.
* fixed an issue where `ArrayColumn::NumRows()` always returns 0 when
the mmap backing storage is a file.
* removed unused `ColumnBase` constructors and unnecessary members so we
don't get confused.
* Updated `test_chunk_cache.cpp` to make the tests parameterized: to
test both mmap enabled and disabled. Added sparse field in the test to
add coverage.
* re-enabled test `Sealed::GetSparseVectorFromChunkCache`.
* But 2 other disabled tests `Sealed::WarmupChunkCache` and
`Sealed::GetVectorFromChunkCache` remain disabled, there seems to be
errors. @bigsheeper PTAL.
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-09-25 18:59:13 +08:00
3cd0b26285
enhance: Enable dynamic update loaded collection's replica ( #35822 )
...
issue: #35821
After collection loaded, if we need to increase/decrease collection's
replica, we need to release and load it again.
milvus offers 4 solution to update loaded collection's replica, this PR
aims to dynamic change the replica number without release, and after
replica number changed, milvus will execute load replica or release
replica in async, and the replica loaded status can be checked by
getReplicas API.
Notice that if set too much replicas than querynode can afford，the new
replica won't be loaded successfully until enough querynode joins.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-25 10:13:18 +08:00
ed95568a05
enhance: Fix PR conflict in reduce unit test ( #36470 )
...
Related to #36433 #36180
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-24 18:01:13 +08:00
3bd7ec8751
fix: Fix cornor case that segment can't be move out from stopping node ( #36431 )
...
issue: #36426
the old constriant requires only segment on current target can be
balanced, which is wrong, and caused that segment can't be move out from
stopping node, if it's only exist in next target.
by design, stopping balance need to move out all segment on it by
balance task, thus the unfair old constriant should be removed.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-24 17:01:14 +08:00
52cce4de58
fix: iaccurate size estimation for encoded array data ( #36373 )
...
issue: #36029
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-24 14:51:14 +08:00
350dde666d
fix: streaming node dead lock ( #36403 )
...
issue: #36388
- fix dead lock.
- fix barrier timetick failure.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-24 14:33:13 +08:00
d55d9d6e1d
fix: change pymilvus version for hybridsearch-groupby( #36407 ) ( #36451 )
...
related: #36407
---------
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-24 14:29:13 +08:00
98a917c5d4
enhance: [skip e2e] Add unittest for reducing duplicated pk from multi segments ( #36433 )
...
Related to #35505 #36362
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-24 14:11:13 +08:00
8cda48a96a
enhance: Use mmap.scalarIndex config for text index ( #36400 )
...
issue: https://github.com/milvus-io/milvus/issues/35273
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-24 12:21:13 +08:00
df7ae08851
fix: iterator cursor progress too fast( #36179 ) ( #36180 )
...
related: #36179
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-24 11:45:13 +08:00
1833913f44
enhance: Add streaming forward policy switch for delegator ( #36330 )
...
Related to #35303
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-23 18:01:12 +08:00
6e880d19a8
enhance: add search params in search request in restful ( #36304 )
...
https://github.com/milvus-io/milvus/issues/36321
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-23 16:51:12 +08:00
c50fe71163
fix: long buffering causes mq to be unable to receive messages. ( #36420 )
...
- issue: #36397
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-23 16:33:18 +08:00
eb23e23cd2
enhance: refine parameter relationship for hybridsearch_group_by( #35096 ) ( #36289 )
...
related: #35096
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-20 14:55:11 +08:00
4b077e1bd2
fix: Fix the compatibility bug between stats task and segment ( #36359 )
...
issue: #33744
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-20 14:33:11 +08:00
d2c774fb6d
fix: Return all compactTo segments after support split ( #36361 )
...
Related to #36360
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-20 14:11:11 +08:00
3b10085f61
enhance: Optimize workload based replica selection policy ( #36181 )
...
issue: #35859
This PR introduce two new param: toleranceFactor and checkRequestNum,
after every checkRequestNum request has been assigned, try to compute
querynode's workload score.
if the diff is less than the toleranceFactor, replica selection policy
will fallback to round_robin, which reduce the average cost to about
500ns.
if the diff is larger than the toleranceFactor, replica selection policy
will compute querynode's score to select the target node with smallest
score in every assigment.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-20 12:33:11 +08:00
363004fd44
enhance: simplify reduction on single search result ( #36334 )
...
See: #36122
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-09-20 11:59:10 +08:00
89397d1e66
enhance: adjust parquet reader type check with null type ( #36266 )
...
#36252
remove no need type check. if users use null type writer to write
parquet, hope it successfully.
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-19 18:43:10 +08:00
e84c4f2a92
fix: to fill default value when nullable and default value are both enable ( #36030 )
...
#36003
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-19 17:35:11 +08:00
f65261215b
fix: streaming node health check panic ( #36336 )
...
issue: #36335
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-19 17:11:12 +08:00
139787371e
feat: support embedding bm25 sparse vector and flush bm25 stats log ( #36036 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-19 10:57:12 +08:00
167e4fb10d
[automated] Update Knowhere Commit ( #36352 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-09-19 01:01:10 +08:00
47da9023a6
fix: add future stateful lock ( #36332 )
...
issue: #36323
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-18 20:15:11 +08:00
23b95aeba3
fix: remove element type check ( #35828 )
...
https://github.com/milvus-io/milvus/issues/36275
Array's element type is not same with schema's. It is INT32 for INT16
and INT8
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-09-18 11:37:10 +08:00
dcd904d2fa
fix: keep inner topK to avoid exceeding efSearch ( #36284 )
...
issue: #https://github.com/milvus-io/milvus/issues/36243
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-09-16 11:01:25 +08:00
763fd0dfc5
enhance: Use a separate mmap config for chunk cache ( #36276 )
...
issue: https://github.com/milvus-io/milvus/issues/35273
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-15 16:23:09 +08:00
517f8b3755
enhance: Refine the code for returning error ( #36103 )
...
issue: #36023
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-15 15:23:14 +08:00
e486442ca6
enhance: Disallow the keywords as a field name or dynamic field name ( #36101 )
...
issue: #35873
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-15 15:19:14 +08:00
a61668c77e
feat: Introduce stats task for import ( #35868 )
...
This PR introduce stats task for import:
1. Define new `Stats` and `IndexBuilding` states for importJob
2. Add new stats step to the import process: trigger the stats task and
wait for its completion
3. Abort stats task if import job failed
issue: https://github.com/milvus-io/milvus/issues/33744
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-15 15:17:08 +08:00
2ff3765058
enhance: catch std::stoi exception and improve error msg ( #36267 )
...
issue: #36255
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-14 16:17:08 +08:00
3352030a84
enhance: Graceful stop flowgraph manager when stopping datanode ( #36229 )
...
Flowgraph manager is not stopped durong datanode stopping procedure
which may lead to unexpect flowgraph behavior during/after datanode stop
progress.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-14 15:55:08 +08:00
b8b4aea4f5
enhance: restrict max group size( #33544 ) ( #36223 )
...
related: #33544
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-14 15:45:08 +08:00
fc1bdd4c84
fix: to forbid bulk insert with nullable field in numpy files ( #36246 )
...
#36241
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-14 15:35:07 +08:00
329fb421cd
fix: fix search/query/count may access same growing and sealed segment ( #36258 )
...
issue: #36257
during syncTargetVersion, sealed segment should be excluded, to avoid
it's growing segment be conusmed from stream again.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-14 14:21:07 +08:00
d07338b4b9
enhance: Ignore index check for non-loaded vector field ( #36170 )
...
Related to #35996
For `Field Partial Load` feature, Milvus shall ignore index check for
non-loaded vector field.
Also, this PR unifies the logic of index check for load collection and
load partitions tasks.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-14 11:01:08 +08:00
2d05b7f219
fix: to add check in partition key field when set nullable==true ( #36218 )
...
#36213
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-14 10:53:08 +08:00
3bc7d63be9
fix: overwrite correct selection when pk duplicated ( #35826 )
...
Related to #35505
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-14 10:27:08 +08:00
9d8d332c88
fix: Fix improper use of offset in HybridSearch ( #36244 )
...
issue :https://github.com/milvus-io/milvus/issues/36243
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-09-13 22:05:15 +08:00
489087d18b
enhance: refactor executor framework V2 ( #35251 )
...
#32636
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-13 20:57:09 +08:00
22cce44afc
fix: metrics stored_index_files_size is never cleared ( #36160 )
...
issue: #36159
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-13 20:09:15 +08:00
58d3200986
enhance: Filter out non-hit delete records during load delta ( #36207 )
...
Related to #35303
This PR utilizes pk index in segment to exclude non-hit delete record
during load delete records. This ability is crucial when l0/delete
forward policy only replies on segment itself(without BF filtering).
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-13 19:05:08 +08:00
f7d950d465
fix: [skip e2e] Fix unstable ut TestCollectionObserver ( #36231 )
...
issue: #36237
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-13 19:01:09 +08:00
fb2a41a94c
fix: Clean dirty segment/channel on querynode ( #36202 )
...
issue: #36201
after querynode has been remove from replica, all dirty segment/channel
on it should be released.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-13 18:15:08 +08:00
b4c1603a57
fix: skip recover flusher on streaming node when channel is dropped ( #36149 )
...
issue: #36016,#36230
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-13 17:19:08 +08:00
bd658a6510
enhance: Enable dynamic update replica selection policy ( #35860 )
...
issue: #35859
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-13 17:05:15 +08:00
c03eb6f664
fix: streaming node consume blocks if recv message is too large ( #36151 )
...
issue: #36081
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-13 16:41:08 +08:00
c22a2cebb6
fix: split stream query result to avoid grpc response too large error ( #36090 )
...
relate: https://github.com/milvus-io/milvus/issues/36089
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-13 15:07:09 +08:00
f0f2fb4cf0
enhance: span tracing of c++ part ( #36205 )
...
fix: https://github.com/milvus-io/milvus/issues/36204
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-09-13 11:19:09 +08:00
06a706e5f0
fix: fix ReadWrite privilege group deny all global API ( #36144 )
...
issue: #35471
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-13 10:33:08 +08:00
c6eb6c7cb2
enhance: Add error handler for write buffer ( #36216 )
...
Related to #36215
This PR add error handler setting option providing the possibility to
change error handling behavior other than panicking.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-13 10:11:09 +08:00
b9c037f558
feat: adding cache to expression parse ( #36185 )
...
See #36122
This PR improves the proxy node performance by adding cache to
expression parse.
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-09-13 10:03:09 +08:00
8395c8a8db
enhance: Update stats task to optional ( #35947 )
...
issue: #33744
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-12 20:37:08 +08:00
e8840a1b41
enhance: Add metrics for Delete entries num of L0seg ( #36175 )
...
- Add metrics *DataCoordL0DeleteEntriesNum*
- Remove metrics *DataCoordRateStoredL0Segment*
See also: #36147
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-09-12 18:07:08 +08:00
5e5e87cc2f
enhance: rename some params and reduce default bitmapCardinalityLimit… ( #36138 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-12 12:09:08 +08:00
11dbe1e755
enhance: Add L0 forward policy to support remote load ( #36189 )
...
Related to #35303
This PR add a param item to support change l0 forward behavior from bf
filtering and forward to remote load.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-12 12:01:08 +08:00
da227ff9a1
feat: Support create collection with functions ( #35973 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
Support create collection with functions. Prepare for support bm25
function.
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-12 10:43:06 +08:00
700a443e85
enhance: Check ANN field loaded in proxy ( #36172 )
...
Related to #35415
Currently ANN field is loaded in Parital load is not performed in proxy,
this cause error message not clear and no error returned when collection
is empty
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-11 17:43:06 +08:00
e7ea1d7a04
enhance: improve log encoding performance on proxy nodes ( #36123 )
...
See #36122
This PR is designed to enhance log performance through two improvements:
1. Optimize JSON encoding by switching JSON serializer to
`json-iterator`.
2. Adding support of lazy initialization `WithLazy`.
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-09-11 14:51:07 +08:00
3f75bf1f20
fix: clustering compact not support null ( #36152 )
...
#36055
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-11 14:49:06 +08:00
29ddb10e8c
enhance: timer instead of ticker ( #36133 )
...
https://github.com/milvus-io/milvus/issues/36132
Signed-off-by: fengjun2016 <jornfeng@gmail.com>
2024-09-10 15:23:06 +08:00
89bf226f0b
feat: support keyword text match ( #35923 )
...
fix : #35922
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-09-10 15:11:08 +08:00
9d0aa5c202
fix: empty result when having only one subReq( #36098 ) ( #36128 )
...
related: #36098
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-10 14:25:07 +08:00
53a8a24554
fix: fix empty indices of sparse float ( #35403 )
...
https://github.com/milvus-io/milvus/issues/35401
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-09-10 14:23:07 +08:00
4850641943
fix: BinlogDeserializeReader leak ( #36087 )
...
https://github.com/milvus-io/milvus/issues/36086
Signed-off-by: fengjun2016 <jornfeng@gmail.com>
2024-09-10 12:43:07 +08:00
851f3b9883
fix: Make legacy non-lexicographic branch break swtich ( #36125 )
...
Related to #35941
Previous PR: #36034
This patch makes the switch branching logic correct and make the unit
test work for cases which does not select the whole dataset.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-10 10:15:07 +08:00
c0c12c6c5b
fix: Use SliceSetEqual to compare load field list ( #36051 )
...
Related to #36037
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-10 10:13:14 +08:00
3a381bc247
enhance: Bulkinsert supports null in csv formats ( #35912 )
...
see details in this issue
https://github.com/milvus-io/milvus/issues/35911
---------
Signed-off-by: OxalisCu <2127298698@qq.com>
2024-09-09 19:17:07 +08:00
aaa8487590
enhance: Remove duplicated log in proxy service ( #36110 )
...
Remove duplicate identical log in proxy/service.go
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-09 17:19:05 +08:00
51870aef08
enhance: Use stats Handler to record request/response size metrics ( #36107 )
...
Related to #36102
This PR use newly added `grpcSizeStatsHandler` to reduce calling
`proto.Size` since the request & response size info is recorded by grpc
framework.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-09 17:13:05 +08:00
3123093dd7
enhance: Use `MARISA_LABEL_ORDER` when building trie index ( #36034 )
...
Related to #35941
Previous PR: #35943
This PR make `Trie` index using `MARISA_LABEL_ORDER`, which make
predictive search iterating in lexicographic order.
When trie index is build in label order, lexicographc could be utilized
accelerating `Range` operations.
However according to the official document, using `MARISA_LABEL_ORDER`
will make "exact match lookup, common prefix search, and predictive
search" slower.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-09 14:29:05 +08:00
2687747278
fix: Set an empty segment if compaction deleted all inserts ( #36044 )
...
See also: #36038
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-09-09 14:23:05 +08:00
99817953eb
fix: delay to start the metric server port ( #36080 )
...
- issue: #36083
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-09 14:01:05 +08:00
8787e65b1f
fix: lifeTime not released in time ( #36093 )
...
https://github.com/milvus-io/milvus/issues/36092
Signed-off-by: fengjun2016 <jornfeng@gmail.com>
2024-09-09 11:25:05 +08:00
30a99b66c1
fix: Fix logic dead lock when delegator has high memory usage ( #36065 )
...
issue: #36064
when delegator has high memory usage, load l0 segment will failed. and
balance segment task will blocked by load segment task, then delegator
cann't free memory by moving out some segment, causes a logic dead lock.
this PR remove the limit for balance, we permit segment and balance
execute in parallel. which won't cause side effect due to:
1. one segment can only has one task in qc's scheduler, and load/release
task will replace balance task if necessary
2. balance speed has been limited, and it won't block load segment task.
3. if collection has load task and balance task at same time, load task
will be scheduled first due to high proirity.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-09 10:21:06 +08:00
208c8a2328
fix:support config index offsetcache and fix create same index again ( #35985 )
...
#35971
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-08 18:23:05 +08:00
91d23ecbe1
fix: memory leak in proxy meta cache ( #36075 )
...
issue: #36074
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-08 17:33:05 +08:00
6b4ae0c65e
enhance: Log warn on delayed compaction tasks ( #36049 )
...
/kind enhancement
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-08 17:23:05 +08:00
e480b103bd
feat: supporing hybrid search group_by ( #35982 )
...
related: #35096
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-08 17:09:04 +08:00
62f4a6a112
fix: binlog reader not released in time ( #36078 )
...
https://github.com/milvus-io/milvus/issues/36077
Signed-off-by: fengjun2016 <jornfeng@gmail.com>
2024-09-07 08:15:06 +08:00
7859faf8ea
fix: Change deltalog memory estimation factor to one ( #36033 )
...
See also: #36031
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-09-06 18:21:05 +08:00
a103dd5eb3
enhance: Fix SearchOnSealed clang-format lint ( #36056 )
...
Related to #36008
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-06 16:47:04 +08:00
21b135c7c2
fix: not append valid data when transfer to insert record ( #36027 )
...
fix not append valid data when transfer to insert record and add a tiny
check when in groupBy field.
#35924
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-06 14:53:04 +08:00
5247631289
fix: fill the metric type field in the LoadMetaInfo object ( #35962 )
...
- issue: #35960
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-05 20:50:23 -07:00
9871841fa0
fix: datacoord broker use querycoord broker config and get segment info timeout ( #36026 )
...
relate: https://github.com/milvus-io/milvus/issues/36025
Will case channel with large segment num watch failed.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-06 08:15:05 +08:00
11325d9ed5
fix: binary arith expression on inverted index ( #35945 )
...
issue: https://github.com/milvus-io/milvus/issues/35946
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-09-05 20:01:05 +08:00
75676fbd11
fix: Fix dynamic release partition may fail search/query request ( #35919 )
...
issue: #33550
cause concurrent issue may occur between remove parition in target
manager and sync segment list to delegator. when it happens, some
segment may be released in delegator, and those segment may also be
synced to delegator, which cause delegator become unserviceable due to
lack of necessary segments, then search/query fails.
this PR make sure that all write access to target_manager will be
executed in serial to avoid the concurrent issues.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-05 18:47:03 +08:00
560e8e70b0
enhance: reduce mmap_rss after chunkcache warmup ( #35974 )
...
related pr: https://github.com/milvus-io/milvus/pull/35965
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-09-05 18:07:05 +08:00
8b043f58dc
fix: Fix DB limiter nodes are mistakenly cleaned up ( #35991 )
...
This issue only occurs for a short time right after a table is created.
To avoid this, we simply reduce the frequency of cleaning up invalid
limiter nodes.
issue: https://github.com/milvus-io/milvus/issues/35933
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-05 17:35:04 +08:00
f985173da0
fix: Fill load field list from old version load info ( #35993 )
...
See also #35959
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 16:57:05 +08:00
c61eea737b
enhance: Fix trace.cpp lint format issue ( #36004 )
...
Introduced by #35928
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 16:33:04 +08:00
32e55a02ea
fix: Fix privilege group hasn't been register for validate ( #35937 )
...
issue: #35471
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-05 15:35:04 +08:00
7b21032d19
fix: Check all values for `trie.predictive_search` ( #35943 )
...
Related to #35941
For marisa trie `predictive_search` default behavior, it value iterated
is not in lexicographic order.
This PR is a brute force fix to make range operator returns correct
values.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 15:01:04 +08:00
9e96ed4873
fix: Fix tracing config update logic ( #35928 )
...
Related to #35927
There are serveral issue this PR addresses:
- Use `ResetTraceConfig` method instead init one in update event handler
- Implement dynamic stats.Handler to receive tracing config update event
- Update `enable_trace` flag when `ResetTraceConfig` is invoked
- Change `enable_trace` to `std::atomic<bool>` in case of data race
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 14:27:04 +08:00
5e3f700e5d
enhance: Remove too frequent logs in Delete ( #35980 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-09-05 14:25:03 +08:00
8593c4580a
enhance: Add delete buffer related quota logic ( #35918 )
...
See also #35303
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 11:39:03 +08:00
80a9efd96d
fix: not check nullable and default value in pk field ( #35987 )
...
#35926
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-05 11:27:04 +08:00
66ed289a85
enhance: Fix typo of clustering key not loaded msg ( #35948 )
...
Related to #35415
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 10:49:03 +08:00
90bdb171ab
fix: Fix data race for clustering compaction writer ( #35957 )
...
issue: #35950
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-05 04:07:10 +08:00
6eb8b3f745
fix: err degenerated to a new variable ( #35891 )
...
https://github.com/milvus-io/milvus/issues/35890
Signed-off-by: fengjun2016 <jornfeng@gmail.com>
2024-09-04 14:57:04 +08:00
24fb10114b
enhance: remove cooling off in rate limiter for read requests ( #35935 )
...
issue: #35934
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-04 14:39:10 +08:00
ea36d13ab0
feat: add static view for the expr ( #35887 )
...
- issue: #35886
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-04 11:09:04 +08:00
cc02dc0a55
fix: Handle Input/Output Errors in vsnprintf and snprintf ( #35898 )
...
Fix for Issue: #35897
2024-09-04 08:15:04 +08:00
9da86529a7
enhance: Add disk filemananger parallel load control to reduce the memory consumption ( #35281 )
...
issue: #35280
add parallel control to limit the memory consumption during index file
loading
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-09-03 18:01:03 +08:00
f68df9a11e
fix: SkipIndex cause segment fault ( #35907 )
...
issue: #35882
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-03 17:15:03 +08:00
74048ce34f
fix:rename mmap file path to avoid directory conflict ( #35810 )
...
#35784
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-03 16:05:03 +08:00
c84ea5465c
fix: Fix some replicas don't participate in the query after the failure recovery ( #35850 )
...
issue: #35846
querycoord will notify proxy to update shard leader cache after
delegator location changes, but during querynode's failure recovery,
some delegator may become unserviceable due to lacking of segments, and
back to serviceable after segment loaded, so we also need to notify
proxy to invalidate shard leader cache when delegator serviceable state
changes.
This PR will maintain querynode's serviceable state during heartbeat,
and notify proxy to invalidate shard leader cache if serviceable state
changes.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-03 15:39:03 +08:00
6130a85444
enhance: Remove bf from streaming node ( #35902 )
...
Remove bf from streaming node:
1. When watching vchannels, skip loading bloom filters for segments.
2. Bypass bloom filter checks for delete messages, directly writing to
L0 segments.
3. Remove flushed segments proactively after flush.
issue: https://github.com/milvus-io/milvus/issues/33285 ,
https://github.com/milvus-io/milvus/issues/34585
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-03 14:17:02 +08:00
325f1987d9
enhance: rewrite index params for compatibility ( #35788 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-02 19:41:03 +08:00
3698c53a72
enhance: Check load fields for previous loaded collection ( #35905 )
...
Related to #35415
This PR make querycoord report error when load request tries to update
load fields list, which is currently not supported.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-02 18:39:03 +08:00
4641fd9195
enhance: make search groupby stop when reaching topk groups ( #35814 )
...
related: #33544
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-02 18:25:03 +08:00
b2eb9fe2a7
fix: memory leak in unittest and open the USE_ASAN option when build unittest ( #35855 )
...
issue: #35854
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-02 15:59:04 +08:00
6fd33285e1
fix: Fix compile error ( #35901 )
...
/kind improvement
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-02 14:50:35 +08:00
2c9bb4dfa3
feat: Support stats task to sort segment by PK ( #35054 )
...
issue: #33744
This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-02 14:19:03 +08:00
9d80137698
fix: Check clustering key skip load behavior ( #35865 )
...
feature issue: #35415
See also #35861
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-02 11:17:02 +08:00
576ac2bbed
fix: Fix the reference to a variable after it has been moved ( #35875 )
...
#35607
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-02 10:05:02 +08:00
5ea2454fdf
feat: tantivy tokenizer binding ( #35801 )
...
fix : #35800
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-09-01 17:13:03 +08:00
1413ffe9b1
enhance: Rename preAllocatedSegments ( #35871 )
...
Rename `preAllocatedSegments` to `preAllocatedSegmentIDs` to avoid
confusion.
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-01 17:09:01 +08:00
2e090b2426
enhance: Refine flusher channel management ( #35870 )
...
Change the ChannelTask to ChannelLifetime, only removing it upon
unregistering.
issue: https://github.com/milvus-io/milvus/issues/33285
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-01 17:07:10 +08:00
407fc933e7
fix: bump bytedance/sonic to v1.12.2 to fix compilation error with go 1.23.0 ( #35879 )
...
fixes : #35878
Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
2024-09-01 13:43:01 +08:00
311f860676
enhance: support to drop the role which is related the privilege list ( #35727 )
...
- issue: #35545
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-30 15:17:00 +08:00
323400c190
enhance: Enable to write multiple segments in mix compactor ( #35705 )
...
Prevent segments to be written larger than maxSize * expansionRate
See also: #35584
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-30 11:29:01 +08:00
1c3abd1aea
fix: Fix unstable TestFlusher ut ( #35831 )
...
issue: https://github.com/milvus-io/milvus/issues/35830
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-30 10:25:01 +08:00
f9be01ce89
fix: mockery config ( #35834 )
...
issue: #33285
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-30 09:49:00 +08:00
671112d17b
enhance: add more info to hybrid index log ( #35808 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-29 21:07:04 +08:00
a3f2f044d6
fix: not set nullable when stream writer write headers ( #35799 )
...
#35802
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-29 20:59:00 +08:00
3c8941a88e
enhance: Print log only when rate limit updates ( #35806 )
...
The debug log for "RateLimiter register for rateType" is too frequent
and in e2e cases, the may print 18M times in one run.
This PR make the log be printed only when rate limit is updated.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 15:07:00 +08:00
09ef3f1b4f
fix: Make sure querycoord observers started once ( #35811 )
...
Related to #35809
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 14:45:00 +08:00
b3089b5bdc
feat: support range search pagination retains order ( #35738 )
...
issue: #35464
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-08-29 14:09:00 +08:00
b0ac04d104
enhance: set database properties to restrict read access ( #35745 )
...
issue: #35744
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-29 13:17:01 +08:00
b51b4a2838
fix: try get not exist file after upgrade ( #35740 )
...
https://github.com/milvus-io/milvus/issues/35741
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-29 11:09:01 +08:00
86691656f3
enhance: Change frequent balancer debug log to rated one ( #35749 )
...
"skip balance" log is too frequent in debug level. This PR changes it
into rated on.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 10:07:00 +08:00
985d84d3ee
enhance: Add skip load validation for create collection task ( #35737 )
...
Related to #35415
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 10:05:08 +08:00
99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode ( #35406 )
...
issue: #33285
- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-29 10:03:08 +08:00
c5c9ef9876
fix: Check response size before add to counter ( #35778 )
...
Related to #35767
prometheus counter cannot add negative value
when response is not written(say timeout/network broken) panicking may
happen if not check
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 00:05:00 +08:00
9b96841ae9
fix: wrong construction in evalctx ( #35772 )
...
issue: #35771
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-28 19:21:00 +08:00
91223deb91
fix: Use `SafeChan` as TxnManager.closed ( #35763 )
...
Resolves : #35762
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-28 19:06:59 +08:00
56b3490a98
fix: Fix rate wasn't limited to the expected value ( #35699 )
...
Each time the rate is reset, the token bucket is fully refilled, causing
the rate wasn't limited to the expected value. This PR addresses the
issue by preventing the token reset.
issue: https://github.com/milvus-io/milvus/issues/35675 ,
https://github.com/milvus-io/milvus/issues/35702
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-28 14:13:00 +08:00
a52ba3d09d
enhance: allow many segments for inverted index ( #35616 )
...
fix: https://github.com/milvus-io/milvus/issues/35615
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-28 11:30:59 +08:00
d8aa01bc1a
fix: inconsistent meta view causes rate limit invalid ( #35665 )
...
issue: #35663
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-28 11:21:04 +08:00
b7ea1defd3
fix: mistaken deletions may occur during GC channel checkpoints ( #35707 )
...
issue: #35706
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-28 10:11:05 +08:00
98866205fa
fix: munmap deallocate too much memory ( #35725 )
...
issue: #35693
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-27 17:18:59 +08:00
9868fe4e6c
fix: Fix panic due to empty candidate import segments ( #35673 )
...
issue: https://github.com/milvus-io/milvus/issues/35662
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-27 17:08:59 +08:00
4d2f96c760
enhance: support bitmap mmap ( #35399 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-27 16:34:59 +08:00
bfd9d86fe9
feat: support groupby size on go-layer( #33544 ) ( #33845 )
...
related: #33544
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-08-27 14:21:00 +08:00
fe83805d56
fix: loss data bug for deprecated querynode DoubleBuffer ( #35128 )
...
relate: https://github.com/milvus-io/milvus/issues/31548
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-08-27 14:10:59 +08:00
f12e368a76
fix: Fill nil schema so that Milvus can watch channel for those upgraded from 2.2 to 2.4 #35695 ( #35694 )
...
See also: [#35701 ](https://github.com/milvus-io/milvus/issues/35701 )
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-27 10:36:59 +08:00
3e1052f889
enhance: use the msg position obj when getting replicate channel position ( #35606 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-27 10:28:59 +08:00
7d3c0d748c
fix: the incorrect number of partitions in rootcoord meta ( #35600 )
...
issue: #35698
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-26 15:20:58 +08:00
82743c5c50
fix: Clear channelcp meta and metrics ASAP ( #35658 )
...
See also: #35588
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-26 12:22:57 +08:00
6ddfd02f01
[automated] Update Knowhere Commit ( #35688 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-26 01:04:57 +08:00
4236c3bafb
enhance: enable MV on binary vectors ( #35668 )
...
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-08-25 16:27:09 +08:00
098c10922b
enhance: add some check when create collection ( #35629 )
...
not support nullable==true in vector type. check it when create
collection.
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-25 16:23:05 +08:00
0dc5e89007
enhance: reduce the log level of frequent log ( #35652 )
...
fix #35651
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-08-25 16:20:57 +08:00
615a653988
fix: Fix offset out of range for creating Trie index ( #35553 )
...
issue: #35550
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-25 15:50:57 +08:00
f2b83d316b
enhance: Support memory mode chunk cache ( #35347 )
...
Chunk cache supports loading raw vectors into memory.
issue: https://github.com/milvus-io/milvus/issues/35273
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-25 15:42:58 +08:00
50fcfe8ef1
enhance: add nan and inf check ( #35683 )
...
fix #35594
add float check on files
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-08-25 15:22:57 +08:00
42f7800b5b
enhance: add bitmap offset cache to speed up retrieve raw data ( #35498 )
...
#35458
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-24 01:40:58 +08:00
75da36d1aa
enhance: enable asan for milvus ( #35627 )
...
issue: #35626
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-23 21:06:58 +08:00
59387f0b3a
fix: Use correct release func when datanode unwatch a channel ( #35655 )
...
See also #35654
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-23 14:09:03 +08:00
e8e3544a11
enhance: add hit segment num metrics for queryHook ( #35577 )
...
issue: #35576
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-08-23 12:49:02 +08:00
a773836b89
enhance: optimize milvus core building ( #35610 )
...
issue: #35549,#35611,#35633
- remove milvus_segcore milvus_indexbuilder..., add libmilvus_core
- core building only link once
- move opendal compilation into cmake
- fix odr
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-23 12:35:02 +08:00
ab532ae199
enhance: Add back BF lazy load logic for datanode watch channel ( #35646 )
...
Add back lazy loading statslog when watch dml channel on datanode.
Related to #22994 #27675
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-22 19:42:57 +08:00
5c245d51c4
enhance: Refresh proxy cache after restore rbac meta ( #35635 )
...
issue: #35443
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-22 19:09:01 +08:00
9fff07936a
fix: Re-read value after `once` initialization ( #35642 )
...
Related to #35641
See also #35271
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-22 17:24:58 +08:00
582d2eec79
enhance: Move datanode/indexnode manager to session pkg ( #35634 )
...
Related to #28861
Move session manager, worker manager to session package. Also renaming
each manager to corresponding node name(datanode, indexnode).
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-22 16:02:56 +08:00
3107701fe8
enhance: optimize retrieve on dynamic field ( #35580 )
...
#35514
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
Co-authored-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-08-22 14:24:56 +08:00
c992a61a23
enhance: Separate allocator pkg in datacoord ( #35622 )
...
Related to #28861
Move allocator interface and implementation into separate package. Also
update some unittest logic.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-22 10:06:56 +08:00
024eccbde0
enhance: add pkg-config for knowhere ( #35433 )
...
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-08-22 09:56:56 +08:00
ed4eaffc9d
enhance: add csv support for bulkinsert ( #34938 )
...
See this issue for details: #34937
---------
Signed-off-by: OxalisCu <2127298698@qq.com>
2024-08-21 17:47:01 +08:00
731d45abbe
enhance: provide more general configuration to control mmap behavior ( #35359 )
...
- issue: #35273
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-21 00:22:54 +08:00
41646c8439
feat: integrate new deltalog format ( #35522 )
...
See #34123
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-08-20 19:06:56 +08:00
bc2bbdc82f
enhance: Forbid access log print error message with line break ( #34491 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-08-20 17:36:55 +08:00
80a7c78f28
enhance: import supports null in parquet and json formats ( #35558 )
...
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-20 16:50:55 +08:00
2fbc628994
feat: Support field partial load collection ( #35416 )
...
Related to #35415
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-20 16:49:02 +08:00
031ee6f155
enhance: support httpv1/v2 throttle and add it for httpV2( #35350 ) ( #35470 )
...
related: #35350
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-08-20 16:16:55 +08:00
3491608256
fix: Match int8_t and int16_t in Array::get_data ( #35579 )
...
Related to #35578
Previously int16/int8 bitmap index may read int32 array as int16, which
may cause build index with half of the data(if array is full) and half
zeros. This causes BITMAP index lost information.
This PR matches int8_t & int16_t while `get_data` when building index.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-20 16:10:56 +08:00
e09dc3be58
enhance: Mark query node as read only after suspend ( #35492 )
...
issue: #34985 #35493
after querynode has been suspended, it's not allow to load
segment/channel on it, which means the node is read only. to be
compatible with resource group design, after query node has been
suspend, we remove it from it's original resource group, make it a read
only query node in replica. then two things will happens:
1. it's original resource group will be lacking of query nodes, query
coord will assign new node to it.
2. querycoord will try to move out all segments/channels after querynode
has been suspended
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-20 14:02:54 +08:00
3948bd4e79
fix: Added check for validating varchar,array max length ( #35499 )
...
issue : https://github.com/milvus-io/milvus/issues/34150
This is for numpy,parquet,json readers.
---------
Signed-off-by: Nischay Yadav <nischay.yadav@ibm.com>
2024-08-20 11:42:55 +08:00
22ced010cd
enhance: make configure load param feature be compatible with old sdk ( #35520 )
...
issue: #31570 #35521
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-20 10:30:55 +08:00
4d69898cb2
enhance: support single pchannel level transaction ( #35289 )
...
issue: #33285
- support transaction on single wal.
- last confirmed message id can still be used when enable transaction.
- add fence operation for segment allocation interceptor.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-19 21:22:56 +08:00
227ecd3901
enhance: Remove the check for channel cp lag when generating compaction plan ( #35383 )
...
issue: https://github.com/milvus-io/milvus/issues/35382
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-19 19:40:55 +08:00
337e065902
fix: querynode hang when failing to allocate disk space for mmap( #35184 ) ( #35187 )
...
related: #35184
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-08-19 15:30:55 +08:00
14ef88f423
fix: Fix proxy panic after Import related api failed ( #35540 )
...
issue: #35539
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-19 15:22:55 +08:00
967f38672a
enhance: Add integration tests for l0 ( #35429 )
...
See also: #34796
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-19 10:56:54 +08:00
80dbe87759
enhance: support null value in index ( #35238 )
...
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-16 15:30:54 +08:00
f87af9bc54
enhance: Exclude L0 segment from readable snapshot ( #35507 )
...
L0 segments now do not contain insert data and may cause confusion for
query hook optimizer if counted as sealed segment number.
This PR add segment level flag in segment entry and exclude L0 segments
while get readable segment snaphsot
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-16 15:28:53 +08:00
5e9097c2dd
fix: err degenerated to a new variable ( #35413 )
...
https://github.com/milvus-io/milvus/issues/35412
Signed-off-by: fengjun2016 <jornfeng@gmail.com>
2024-08-16 14:24:54 +08:00
a570567644
enhance: Enable ReadOnly/ReadWrite/Admin Privilege Group to simplify RBAC grant progress ( #35472 )
...
issue: #35471
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-16 14:18:54 +08:00
f4a91e135b
enhance: Allow empty sparse row ( #34700 )
...
issue: #29419
* If a sparse vector with 0 non-zero value is inserted, no ANN search on
this sparse vector field will return it as a result. User may retrieve
this row via scalar query or ANN search on another vector field though.
* If the user uses an empty sparse vector as the query vector for a ANN
search, no neighbor will be returned.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-08-16 14:14:54 +08:00
1d49358f82
enhance: Add BackupRBAC/RestoreRBAC API to enable rbac backup ( #35444 )
...
issue: #35443
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-16 10:10:53 +08:00
b896143965
enhance: Improve bitset performance for AVX512 ( #35479 )
...
see #35478
optimized functions take 20%+ less time to run
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-08-16 07:44:53 +08:00
1bbf7a3c0e
enhance: Optimize the use of locks and avoid double flush clustering buffer writer ( #35486 )
...
issue: #35436
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-16 02:24:58 +08:00
fe20366b5c
enhance: Remove duplicated schema helper creation in proxy ( #35489 )
...
Related to PRs of #35415
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-15 19:18:53 +08:00
c0200eec39
enhance: limit getSegmentInfo batch size to avoid excced grpc message limit ( #35394 )
...
issue: #35395
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-15 19:17:00 +08:00
f6aaf3fef2
fix: force update next target if target can't be loaded ( #35365 )
...
issue: #35361
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-15 19:15:00 +08:00
b4b1ed6910
fix: before exiting, make sure the goroutine has exited ( #35411 )
...
https://github.com/milvus-io/milvus/issues/35409
Signed-off-by: fengjun2016 <jornfeng@gmail.com>
2024-08-15 18:16:53 +08:00
d20d6ea551
fix: Functional-notation casting vulnerability fix ( #35252 )
...
Fix for issue: https://github.com/milvus-io/milvus/issues/35200
2024-08-15 16:20:53 +08:00
b2cc4b0776
feat: add the rbac msg and send them to the replicate channel ( #35392 )
...
- issue: #35391
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-15 12:06:52 +08:00
b1ea645390
[automated] Update Knowhere Commit ( #35474 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-15 01:14:53 +08:00
f99a761d87
enhance: Support dynamic field in SchemaHelper ( #35461 )
...
Related to #35415
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-14 18:04:59 +08:00
0400eb91c1
enhance: remove the invalid log ( #35465 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-14 18:00:53 +08:00
0868a978d1
fix: streaming service stat manager crash when pv stat is empty ( #35439 )
...
issue: #33285
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-14 12:04:52 +08:00
36d59867f3
fix: err cannot be captured in defer in data_sync_service ( #35448 )
...
https://github.com/milvus-io/milvus/issues/35447
Signed-off-by: fengjun2016 <jornfeng@gmail.com>
2024-08-14 10:26:52 +08:00
c42976ee6f
enhance: Init ChannelCP when creating a channel ( #35387 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-14 10:16:58 +08:00
a690e6e004
fix: panic in timetick_sender ( #35451 )
...
https://github.com/milvus-io/milvus/issues/35450
Signed-off-by: fengjun2016 <jornfeng@gmail.com>
2024-08-14 10:10:53 +08:00
f5b4a972a0
fix: checkResultTicker leak ( #35377 )
...
https://github.com/milvus-io/milvus/issues/35376
Signed-off-by: fengjun2016 <jornfeng@gmail.com>
2024-08-13 21:36:52 +08:00
1b0ea49d25
fix: incorrect memory size and delta count in SegmentInfo ( #35369 )
...
issue: #35368
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-13 17:20:19 +08:00
196b343a94
fix: Fix data race for clustering compaction ( #35435 )
...
issue: #35436
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-13 17:10:20 +08:00
e58d656e65
fix: Fix compile error ( #35442 )
...
issue: https://github.com/milvus-io/milvus/issues/33285
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-13 15:46:19 +08:00
efadf22802
enhance: Append create partition msg to wal ( #35398 )
...
issue: https://github.com/milvus-io/milvus/issues/33285
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-13 14:28:20 +08:00
16b0aee97f
enhance: timetick interceptor optimization ( #35287 )
...
issue: #33285
- remove redundant goroutine by using insepctor.
- remove the coutinous non-message timetick persistence
- periodically push the time tick forward without persistent timetick
message.
- add 'message type filter' deliver filter.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-12 18:58:25 +08:00
06f9ba294e
enhance: Add more ut for l0CompactionTask ( #35100 )
...
See also: #34796
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-09 14:24:18 +08:00
e1d725b938
fix: rollback grpc to 1.50.1 ( #35385 )
...
issue: #35298
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-08-09 14:08:17 +08:00
61a426bc6f
enhance: Init channel cp metric when DataCoord recover ( #35353 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-09 10:08:18 +08:00
c0ee25afd8
fix: Use k locations only for basic BF test location ( #35380 )
...
Related to #35379
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-09 07:52:22 +08:00
143502e708
enhance: Fix progress always zero due to integer divide ( #35374 )
...
See also #35183
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-08 22:26:16 +08:00
aaab827a16
fix: Fix the issue of missing stats log after clustering compaction ( #35266 )
...
issue: #35265
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-08 14:24:17 +08:00
626b1b2f5e
fix:redefine hybrid internal index type ( #35314 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-08 10:32:16 +08:00
7420115b5e
enhance: pinpoint revision of 3thirdparty package ( #35325 )
...
issue: https://github.com/milvus-io/milvus/issues/35340
---------
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-08 09:46:22 +08:00
ae1b81ac1a
fix: fix panic when generating plans ( #35309 )
...
issue: https://github.com/milvus-io/milvus/issues/35335
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-08-07 18:14:16 +08:00
72a175478f
enhance: Append drop partition msg to wal ( #35326 )
...
issue: https://github.com/milvus-io/milvus/issues/33285
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-07 17:28:16 +08:00
838f06323f
fix: datacoord binlog size metrics include dropped segment ( #35338 )
...
issue: #35337
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-07 16:24:16 +08:00
c6253f9c9b
enhance: improve ut cov of clustering compaction task ( #35242 )
...
#34792
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-07 14:44:21 +08:00
bb15ecdc13
enhance: improve ut for compaction_policy_clustering ( #35205 )
...
#34792
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-07 14:42:17 +08:00
1d5bf34e38
fix: init telemetry fails when set otlp as trace exporter ( #35301 )
...
issue: #35298
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-08-07 14:20:21 +08:00
344dc6a9f8
enhance: enable to set load config in cluster level ( #35169 )
...
issue: #35170
This PR enable to set load configs in cluster level, such as replicas
and resource groups. then when load collections will use the load
config.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-07 12:38:21 +08:00
c725416288
enhance: move streaming proto into pkg ( #35284 )
...
issue: #33285
- move streaming related proto into pkg.
- add v2 message type and change flush message into v2 message.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-07 10:34:16 +08:00
b5022b0647
enhance: add metric to record disk quota ( #35305 )
...
https://github.com/milvus-io/milvus/issues/35177
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-07 10:26:16 +08:00
976ceb4a46
fix: comparision operations between incompatible operands ( #35264 )
...
fix : #34139
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-07 10:24:27 +08:00
f13b21c041
fix: error message when input is not aligned ( #35322 )
...
fix: https://github.com/milvus-io/milvus/issues/35321
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-07 09:56:28 +08:00
c19fe95154
fix: support string match for hybrid and bitmap index ( #35294 )
...
#34841
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-07 09:54:22 +08:00
e75efa7359
[automated] Update Knowhere Commit ( #35328 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-07 01:00:16 +08:00
6ff238e88a
fix: Set corresponding DataScope for `loadStreamDelete` ( #35312 )
...
Related to #35311
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-06 22:32:23 +08:00
3c9a47c8db
feat: Encode traceID and spanID as hex string ( #34807 )
...
Issue: https://github.com/zilliztech/knowhere/pull/714
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-08-06 15:20:16 +08:00
91df03afe8
feat: put inverted index into ram ( #35222 )
...
fix: https://github.com/milvus-io/milvus/issues/35224
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-06 11:54:16 +08:00
678018d9ca
enhance: Avoid unnecessary compaction ( #35148 )
...
Estimate the import segment size based on DiskSegmentMaxSize(2G) to
avoid unnecessary compaction after import completed.
issue: https://github.com/milvus-io/milvus/issues/35147
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-06 10:30:21 +08:00
cai.zhang
Zhen Ye
wayblink
wei liu
zhagnlu
smellthemoon
SimFG
sthuang
congqixia
yihao.dai
sre-ci-robot
Buqian Zheng
jaime
Chun Han
Ted Xu
aoiasd
Bingyi Sun
zhenshan.cao
Jiquan Long
XuanYang-cn
CharlesFeng
OxalisCu
cqy123456
Abdullah Ahmed
foxspy
Yinzuo Jiang
Patrick Weizhi Xu
Xiaofan
Gao
presburger
nish112022
Alexander Guzhva
CharlesFeng
yellow-shine
Cai Yudong