2441c92ae0
fix: Query node panic during sending rpc to worker ( #36975 )
...
issue: #36976
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-10-18 12:33:25 +08:00
fbe177d6e7
fix: avoid panic when load segment with pkoracle and idforacle already exist ( #36959 )
...
relate: https://github.com/milvus-io/milvus/issues/36949
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-18 11:57:24 +08:00
903c18ba26
enhance: consider the mmap chunck cache config when resource usage estimate ( #36814 )
...
- issue: #36530
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-10-18 10:17:23 +08:00
3de57ec4fa
enhance: add vector index mgr to remove vector index type dependency ( #36843 )
...
issue: #34298
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-10-17 22:15:25 +08:00
b474374ea5
enhance: use growingMmapEnabled to control the behavior of interim index, not vectorField ( #36500 )
...
issue:https://github.com/milvus-io/milvus/issues/36392
related pr: https://github.com/milvus-io/milvus/pull/36391
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-10-17 20:25:24 +08:00
6851738fd1
fix: fix `make generate-mockery` panic with go1.22 ( #36830 )
...
https://github.com/milvus-io/milvus/issues/36831
Fix `make generate-mockery` panic.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-10-17 12:11:31 +08:00
06b5e186a7
fix: return error if searching against BM25 output field with incorrect metric type ( #36910 )
...
issue: https://github.com/milvus-io/milvus/issues/36835
currently searching BM25 output field using IP will end up in an error
in segcore which is hard to understand. now returning error in query
node delegator and provide more useful error message
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-10-16 19:45:23 +08:00
903450f5c6
enhance: add ts support for iterator( #22718 ) ( #36572 )
...
related: #22718
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-10-16 18:51:23 +08:00
447ff342fb
fix: Direct forward delta exclude l0 segments ( #36899 )
...
Related to #36887
Forward delete to L0 segment will return error and mark l0 segment
offline causing delegator unserviceable
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-16 14:05:23 +08:00
caeab0cc1f
enhance: Fill start pos & level for growing segment ( #36888 )
...
Start position & level info is missing for growing segment loaded in
watch dml channel operation.
Level is important for metrics and start position is crucial for growing
exclude logic.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-16 14:03:31 +08:00
72dc07ba48
fix: bm25 search failed when nq > 1 and remove idf oracle when no bm25 field exist. ( #36886 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-16 12:51:23 +08:00
f3b6792a25
enhance: Enhance segment log ( #36848 )
...
/kind improvement
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-15 20:43:30 +08:00
ba25320aea
fix: Unify loaded partition check to delegator ( #36879 )
...
Related to #36370
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-15 19:15:23 +08:00
5ec4163d0f
feat: support bm25 logs mixcompaction ( #36072 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-14 16:57:22 +08:00
383350c120
feat: added more checks for function creation check ( #36766 )
...
issue: https://github.com/milvus-io/milvus/issues/35853
* BM25 Function now takes no params, k1, b should be passed via index
params
* support BM25 full text search when metric type is not present in
search request
* add more strict validation with functions at collection creation time
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-10-13 17:43:22 +08:00
16b533cbf0
feat: Restful support for BM25 function ( #36713 )
...
issue: https://github.com/milvus-io/milvus/issues/35853
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-10-13 17:41:21 +08:00
a75bb85f3a
feat: support chunked column for sealed segment ( #35764 )
...
This PR splits sealed segment to chunked data to avoid unnecessary
memory copy and save memory usage when loading segments so that loading
can be accelerated.
To support rollback to previous version, we add an option
`multipleChunkedEnable` which is false by default.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-10-12 15:04:52 +08:00
db34572c56
feat: support load and query with bm25 metric ( #36071 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-11 10:23:20 +08:00
3685edb264
enhance: Use common gc config ( #36668 )
...
Use the GC config from `common` and remove the GC config from
`queryNode`.
issue: https://github.com/milvus-io/milvus/issues/36667
related pr: https://github.com/milvus-io/milvus/pull/34949
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-09 19:47:19 +08:00
130a923dec
enhance: the estimate method when loading the collection ( #36307 )
...
- issue: #36530
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
Co-authored-by: xianliang.li <xianliang.li@zilliz.com>
2024-10-09 17:35:19 +08:00
ddc3e76803
fix: Add defer Unpin when error happens ( #36620 )
...
Resolves : #36619
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-30 19:49:17 +08:00
470bb0cc3f
enhance: Enable balance on querynode with different mem capacity ( #36466 )
...
issue: #36464
This PR enable balance on querynode with different mem capacity, for
query node which has more mem capactity will be assigned more records,
and query node with the largest difference between assignedScore and
currentScore will have a higher priority to carry the new segment.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-30 16:15:17 +08:00
ecb2b242e2
enhance: Add sorted for segment info ( #36469 )
...
issue: #33744
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-30 10:01:16 +08:00
4fd9b0a8e3
enhance: Return segment id hint in QueryStream response ( #36487 )
...
Related to #36482
This PR reuses `SealedSegmentIDsRetrieved` field in `RetrieveResults`
struct to store segment id hint.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-26 10:13:14 +08:00
ed95568a05
enhance: Fix PR conflict in reduce unit test ( #36470 )
...
Related to #36433 #36180
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-24 18:01:13 +08:00
98a917c5d4
enhance: [skip e2e] Add unittest for reducing duplicated pk from multi segments ( #36433 )
...
Related to #35505 #36362
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-24 14:11:13 +08:00
df7ae08851
fix: iterator cursor progress too fast( #36179 ) ( #36180 )
...
related: #36179
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-24 11:45:13 +08:00
1833913f44
enhance: Add streaming forward policy switch for delegator ( #36330 )
...
Related to #35303
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-23 18:01:12 +08:00
763fd0dfc5
enhance: Use a separate mmap config for chunk cache ( #36276 )
...
issue: https://github.com/milvus-io/milvus/issues/35273
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-15 16:23:09 +08:00
329fb421cd
fix: fix search/query/count may access same growing and sealed segment ( #36258 )
...
issue: #36257
during syncTargetVersion, sealed segment should be excluded, to avoid
it's growing segment be conusmed from stream again.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-14 14:21:07 +08:00
3bc7d63be9
fix: overwrite correct selection when pk duplicated ( #35826 )
...
Related to #35505
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-14 10:27:08 +08:00
c22a2cebb6
fix: split stream query result to avoid grpc response too large error ( #36090 )
...
relate: https://github.com/milvus-io/milvus/issues/36089
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-13 15:07:09 +08:00
11dbe1e755
enhance: Add L0 forward policy to support remote load ( #36189 )
...
Related to #35303
This PR add a param item to support change l0 forward behavior from bf
filtering and forward to remote load.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-12 12:01:08 +08:00
89bf226f0b
feat: support keyword text match ( #35923 )
...
fix : #35922
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-09-10 15:11:08 +08:00
9d0aa5c202
fix: empty result when having only one subReq( #36098 ) ( #36128 )
...
related: #36098
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-10 14:25:07 +08:00
208c8a2328
fix:support config index offsetcache and fix create same index again ( #35985 )
...
#35971
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-08 18:23:05 +08:00
e480b103bd
feat: supporing hybrid search group_by ( #35982 )
...
related: #35096
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-08 17:09:04 +08:00
7859faf8ea
fix: Change deltalog memory estimation factor to one ( #36033 )
...
See also: #36031
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-09-06 18:21:05 +08:00
5e3f700e5d
enhance: Remove too frequent logs in Delete ( #35980 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-09-05 14:25:03 +08:00
8593c4580a
enhance: Add delete buffer related quota logic ( #35918 )
...
See also #35303
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 11:39:03 +08:00
24fb10114b
enhance: remove cooling off in rate limiter for read requests ( #35935 )
...
issue: #35934
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-04 14:39:10 +08:00
2c9bb4dfa3
feat: Support stats task to sort segment by PK ( #35054 )
...
issue: #33744
This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-02 14:19:03 +08:00
99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode ( #35406 )
...
issue: #33285
- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-29 10:03:08 +08:00
bfd9d86fe9
feat: support groupby size on go-layer( #33544 ) ( #33845 )
...
related: #33544
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-08-27 14:21:00 +08:00
fe83805d56
fix: loss data bug for deprecated querynode DoubleBuffer ( #35128 )
...
relate: https://github.com/milvus-io/milvus/issues/31548
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-08-27 14:10:59 +08:00
f2b83d316b
enhance: Support memory mode chunk cache ( #35347 )
...
Chunk cache supports loading raw vectors into memory.
issue: https://github.com/milvus-io/milvus/issues/35273
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-25 15:42:58 +08:00
e8e3544a11
enhance: add hit segment num metrics for queryHook ( #35577 )
...
issue: #35576
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-08-23 12:49:02 +08:00
a773836b89
enhance: optimize milvus core building ( #35610 )
...
issue: #35549,#35611,#35633
- remove milvus_segcore milvus_indexbuilder..., add libmilvus_core
- core building only link once
- move opendal compilation into cmake
- fix odr
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-23 12:35:02 +08:00
731d45abbe
enhance: provide more general configuration to control mmap behavior ( #35359 )
...
- issue: #35273
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-21 00:22:54 +08:00
41646c8439
feat: integrate new deltalog format ( #35522 )
...
See #34123
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-08-20 19:06:56 +08:00
wei liu
aoiasd
SimFG
foxspy
cqy123456
Bingyi Sun
Buqian Zheng
Chun Han
congqixia
yihao.dai
cai.zhang
Jiquan Long
zhagnlu
XuanYang-cn
jaime
Zhen Ye
Gao
Ted Xu