bfd9d86fe9
feat: support groupby size on go-layer( #33544 ) ( #33845 )
...
related: #33544
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-08-27 14:21:00 +08:00
fe83805d56
fix: loss data bug for deprecated querynode DoubleBuffer ( #35128 )
...
relate: https://github.com/milvus-io/milvus/issues/31548
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-08-27 14:10:59 +08:00
f2b83d316b
enhance: Support memory mode chunk cache ( #35347 )
...
Chunk cache supports loading raw vectors into memory.
issue: https://github.com/milvus-io/milvus/issues/35273
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-25 15:42:58 +08:00
e8e3544a11
enhance: add hit segment num metrics for queryHook ( #35577 )
...
issue: #35576
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-08-23 12:49:02 +08:00
a773836b89
enhance: optimize milvus core building ( #35610 )
...
issue: #35549,#35611,#35633
- remove milvus_segcore milvus_indexbuilder..., add libmilvus_core
- core building only link once
- move opendal compilation into cmake
- fix odr
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-23 12:35:02 +08:00
731d45abbe
enhance: provide more general configuration to control mmap behavior ( #35359 )
...
- issue: #35273
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-21 00:22:54 +08:00
41646c8439
feat: integrate new deltalog format ( #35522 )
...
See #34123
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-08-20 19:06:56 +08:00
2fbc628994
feat: Support field partial load collection ( #35416 )
...
Related to #35415
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-20 16:49:02 +08:00
f87af9bc54
enhance: Exclude L0 segment from readable snapshot ( #35507 )
...
L0 segments now do not contain insert data and may cause confusion for
query hook optimizer if counted as sealed segment number.
This PR add segment level flag in segment entry and exclude L0 segments
while get readable segment snaphsot
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-16 15:28:53 +08:00
6ff238e88a
fix: Set corresponding DataScope for `loadStreamDelete` ( #35312 )
...
Related to #35311
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-06 22:32:23 +08:00
4b553b0333
enhance: revert remove duplicated pk function ( #35103 )
...
issue: #34778
Revert "fix: fix query count(*) concurrently"
Revert "enhance: mark duplicated pk as deleted "
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-05 10:48:17 +08:00
3faef63a25
enhance: add log for partition stats( #30376 ) ( #35219 )
...
related: #30376
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-08-02 19:34:22 +08:00
3641ae6611
fix: Fix index memory estimation ( #35225 )
...
issue: https://github.com/milvus-io/milvus/issues/35229
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-08-02 16:24:15 +08:00
c64a078458
enhance: Support proxy/delegator qn client pooling ( #35194 )
...
See also #35196
Add param item for proxy/delegator query node client pooling and
implement pooling logic
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-02 11:24:19 +08:00
27b6d58981
fix: Set legacy level to l0 segment after qc restart ( #35197 )
...
issue: #35087
after qc restarts, and target is not ready yet, if dist_handler try to
update segment dist, it will set legacy level to l0 segment, which may
cause l0 segment be moved to other node, cause search/query failed.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-02 10:18:13 +08:00
aa247f192d
enhance: remove unused code for StorageV2 ( #35132 )
...
issue: https://github.com/milvus-io/milvus/issues/34168
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-08-01 12:08:13 +08:00
6695c6d0a3
enhance: add channel num for queryHook optimization ( #35104 )
...
At most cases, data in each channel is almost evenly distributed, we
could utilize the channel num info to optimize searh param in queryHook
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-07-31 18:59:49 +08:00
f7f9a729c9
enhance: Pre-allocate space for reduce data structure ( #35118 )
...
Grow slice & map.growWork may cause a lot when segment number is large
for big K query. This PR pre-allocate space for reduce methods to avoid
this cost.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-31 10:35:49 +08:00
de8a266d8a
enhance: Enable linux code checker ( #35084 )
...
See also #34483
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 15:53:51 +08:00
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 ( #34394 )
...
issue: #34252
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
e2e38e98df
fix: nil part stats without l2 compaction( #34923 ) ( #34992 )
...
related: #34923
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-29 11:07:48 +08:00
2372452fac
enhance: Optimized the GC logic to ensure that memory is released in time ( #34949 )
...
issue: #34703
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-28 23:53:47 +08:00
c46c401112
fix: refine handling type for segment pruner( #34923 ) ( #34925 )
...
related: #34923
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-25 13:57:45 +08:00
2ac7164c39
enhance: Remove useless ops when there is no write ( #34767 )
...
Related to #33235
THe querynode pipeline will make map & call ProcessInsert when there is
no write messages. So querynodes will have high CPU usage even when
there is no workload.
This PR check msg length before composing data struct and calling method
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-19 14:31:42 +08:00
804dd5409a
enhance: mark duplicated pk as deleted ( #34586 )
...
fix #34247
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-16 14:25:39 +08:00
531092c031
enhance: Add lint rule to forbid gogo protobuf ( #34594 )
...
github.com/gogo/protobuf is deprecated and could be error prune after
upgrade protobuf message to v2.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-12 10:19:35 +08:00
3b62138c5c
fix: unstable UT for level0 deletion ( #34524 )
...
issue: #34533
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-11 10:02:56 +08:00
d60e628aed
enhance: Avoid use concrete segment type in segments interfaces ( #34521 )
...
See also #34519
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-10 10:18:12 +08:00
eeb03a0d6a
fix: Query may return deleted records ( #34501 )
...
issue: #34500
cause the sort in `GetLevel0Deletions` will broken the corresponed order
between pks and tss, then the pks and tss will be sorted in
segment.Delete() interface.
This PR remove this uncessary and incorrect sort progress to avoid query
may return deleted records.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-09 10:46:11 +08:00
8af187f673
fix: lose partitionIDs when scalar pruning and refine segment prune ratio metrics( #30376 ) ( #34477 )
...
related: #30376
fix: paritionIDs lost when no setting paritions
enhance: refine metrics for segment prune
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-07-08 19:54:15 +08:00
fcafdb6d5f
enhance: reconstruct scalar part's code for segment-pruner( #30376 ) ( #34346 )
...
related: #30376
1. support more complex expr
2. add more ut test for unrelated fields
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-04 16:36:09 +08:00
34bec2ea5e
enhance: add metrics for segment prune latnecy( #30376 ) ( #34094 )
...
related: #30376
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-03 10:04:07 +08:00
b49862d4f3
enhance: Optimize grow slice cost during query ( #34253 )
...
issue: #32252
This PR try to pre-allocate FieldData for Reduce operations in the Query
chain using typeutil.PrepareResultFieldData to avoid the overhead of
dynamically growing the slice during appendFieldData process.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-01 15:18:11 +08:00
45203425fd
enhance: Avoid search querynode return nil status in response ( #34100 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-26 11:50:11 +08:00
9630974fbb
enhance: move rocksmq from internal to pkg module ( #33881 )
...
issue: #33956
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-25 21:18:15 +08:00
f9a0f7bb25
Add an option to enable/disable vector field clustering key ( #34097 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-25 18:52:04 +08:00
fd922d921a
enhance: Add nilness linter and fix some small issues ( #34049 )
...
Add `nilness` for govet linter and fixed some detected issues
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-24 14:52:03 +08:00
ca7ef26e4b
fix: sync part stats task cannot be finished( #30376 ) ( #34027 )
...
related: #30376
also: refine log output for query_coord task by rephrasing action string
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-06-24 10:16:02 +08:00
259a682673
enhance: async search and retrieve in cgo ( #33228 )
...
issue: #30926 , #33132
related pr: #33133
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-22 09:38:02 +08:00
2a1356985d
enhance: support null in go payload ( #32296 )
...
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-06-19 17:08:00 +08:00
a789c60380
enhance: autoindex for multi data type ( #33868 )
...
issue: #22837
contain https://github.com/milvus-io/milvus/pull/33625
https://github.com/milvus-io/milvus/pull/33867
https://github.com/milvus-io/milvus/pull/33911 which already merged to
2.4 branch
Signed-off-by: chasingegg <chao.gao@zilliz.com>
Co-authored-by: foxspy <xianliang.li@zilliz.com>
2024-06-18 21:34:01 +08:00
3fdaae8792
fix: Return record with largest timestamp for entires with same PK ( #33936 )
...
See also #33883
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-18 15:55:59 +08:00
32f685ff12
enhance: growing segment support mmap ( #32633 )
...
issue: https://github.com/milvus-io/milvus/issues/32984
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-18 14:42:00 +08:00
ec64499536
fix: Check nodeID wildcard when removing pkOracle ( #33895 )
...
See also #33894
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-18 14:11:58 +08:00
2a04b0929a
fix: Prevent use captured iteration variable partitionID ( #33906 )
...
See also #33902
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-17 19:11:59 +08:00
9b69601dfb
fix: load operation when segment is on releasing ( #31340 )
...
issue: #30857
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-14 15:35:56 +08:00
4987067375
enhance: Execute bloom filter apply in parallel to speed up segment predict ( #33792 )
...
issue: #33610
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-14 11:37:56 +08:00
ab93d9c23d
enhance: Use BatchPkExist to reduce bloom filter func call cost ( #33611 )
...
issue:#33610
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-13 17:57:56 +08:00
8ca5ced821
fix: async warmup will be blocked by state lock ( #33686 )
...
issue: #33685
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-10 21:59:53 +08:00
a1232fafda
feat: Major compaction ( #33620 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
Chun Han
aoiasd
yihao.dai
Gao
Zhen Ye
SimFG
Ted Xu
congqixia
zhagnlu
Bingyi Sun
wei liu
zhenshan.cao
cai.zhang
jaime
wayblink
smellthemoon
cqy123456