a0e1a1eb31
feat: support enable/disable mmap for index ( #29005 )
...
support enable/disable mmap for index, the user could alter the index's
mode by `AlterIndex` method
related: https://github.com/milvus-io/milvus/issues/21866
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-21 18:07:24 +08:00
61fc822207
fix: creating growing segments may introduce many threads ( #29306 )
...
many growing segments may be created in a short time and there is no
restriction to the process, the CGO call will leave many threads
related: #29282
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-20 10:06:43 +08:00
88b4b8b77c
enhance: make segments SQPool & LoadPool resizable ( #29239 )
...
See also #29223
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-15 20:22:39 +08:00
c4dda3c50f
fix: panic caused by type assert LocalSegment on Segment ( #29018 )
...
related #29017
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-08 01:50:38 +08:00
9b3e06ae86
enhance: add more metrics for level zero segments ( #29029 )
...
- Add SegmentNum metric for level zero segments
- Add level zero segments size metirc
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-07 14:34:35 +08:00
dcb662d9ed
enhance: Refine C.NewSegment response and handle exception ( #28952 )
...
See also #28795
Orignal `C.NewSegment` may panic if some condition is not met, this pr
changes response struct to `CNewSegmentResult`, which contains
`C.CStatus` and may return catched exception
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-07 13:34:35 +08:00
fb089cda8b
enhance: Load raw data while scalar index doesn't have raw data ( #28888 )
...
issue: #28886
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-06 20:36:36 +08:00
1dc086496f
fix: schema->size() check logic with system field ( #28802 )
...
Now segcore load system field info as well, the growing segment
assertion shall not pass with "+ 2" value
This will cause all growing segments load failure
Fix #28801
Related to #28478
See also #28524
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-29 22:40:28 +08:00
3b1b14dd78
fix: update binlog index memory uasge before loading segments ( #28528 )
...
issue: #27678
when interimIndex = true, memory predict should be update with the
memory usage of binlog index build process.
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-11-29 16:42:27 +08:00
d344336a13
fix: Change schema to atomic.Pointer to avoid data race ( #28739 )
...
Fix #28738
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-25 12:42:24 +08:00
cc952e0486
enhance: optimize forwarding level0 deletions by respecting partition ( #28456 )
...
- Cache the level 0 deletions after loading level0 segments
- Divide the level 0 deletions by partition
related: #27349
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 18:24:22 +08:00
0f985c9961
enhance: remove concurrency level in `checkSegmentSize` ( #28533 )
...
`concurrency` is no longer needed in `checkSegmentSize`
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-21 15:00:23 +08:00
13a5b9f64a
fix: query l0 segment bugs ( #28558 )
...
relate: https://github.com/milvus-io/milvus/issues/27675
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-20 17:26:23 +08:00
d2f53aefa5
enhance: improve load speed ( #28518 )
...
This check rejects load request if running out the pool workers, but
small segment would be loaded soon, another segments would been loading
again after a check interval, which leads to slow loading for collection
Block the request by go pool
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-17 19:56:21 +08:00
836f300536
support skip-index based on chunk-metrics to accelerate expr filter( #27925 ) ( #28297 )
...
related: #27925
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-11-15 11:20:19 +08:00
d20ea061d6
Fix panic while forwarding empty deletions to growing segment ( #28213 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-08 16:42:21 +08:00
ece592a42f
Deliver L0 segments delete records ( #27722 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-07 01:44:18 +08:00
90e2c63d9e
Fix getting incorrect CPU num ( #28146 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-11-06 06:02:16 +08:00
dc89730a50
Support collection-level mmap control ( #26901 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-02 23:52:16 +08:00
f8dd589755
Refactor collection's cgo call ( #28055 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-11-02 13:02:13 +08:00
ab6dbf7659
Limit max thread num for pool ( #28018 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-31 13:00:13 +08:00
9b6eeb46f1
Cache segment row num, size, and insert count to reduce CGO calls ( #28007 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-30 17:54:14 +08:00
f93ad6471c
enable query reduce on querynodes for shortcut return( #27306 ) ( #27837 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-10-30 15:08:11 +08:00
36ddeae2cc
Refine log level when request resource fail for loading segments ( #28004 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-30 14:20:14 +08:00
c8415453a1
Support prepare config for index ( #27920 )
...
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-10-25 19:54:12 +08:00
af61a80f73
connect traceID for retrieve ( #27804 )
...
/kind improvement
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-10-19 19:22:10 +08:00
50672f0d60
Skip empty segment while query/search ( #27711 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-15 16:10:04 +08:00
be980fbc38
Refine state check ( #27541 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-11 21:01:35 +08:00
dbdb9e15d8
Update Knowhere version ( #27445 )
...
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Li Liu <li.liu@zilliz.com>
2023-09-29 14:23:28 +08:00
370fdaf50d
Record engine version for segment index ( #27384 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-09-28 18:03:28 +08:00
8c59dba329
Refine queryHook mockery ( #27394 )
...
This PR move `QueryHook` interface to `optimizers` pkg
Update all mockery generated files to latest
Add makefile entry for `QueryHook`
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-28 10:01:26 +08:00
30a6c6072e
Fix bug for loading array data panic ( #27343 )
...
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-09-26 12:23:26 +08:00
5db4a0489e
dynamic index version control ( #27335 )
...
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-25 21:39:27 +08:00
fa06265756
Fix cgo pointer param error ( #27313 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-09-22 18:49:26 +08:00
8f4aaa2da8
Remove ValidateIndexedFieldsData ( #27254 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-22 18:31:25 +08:00
370b6fde58
milvus support multi index engine ( #27178 )
...
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-22 09:59:26 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
0a750408d0
Skip delta logs have been applied ( #26971 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-19 16:21:23 +08:00
cc9974979f
Add staticcheck linter and fix existing problems ( #27174 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-19 10:05:22 +08:00
7939f0e7d5
enable ctx traceId for assignsegment on dc( #26972 ) ( #27108 ) ( #27030 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-09-18 11:39:20 +08:00
488b423e1b
Fix high cpu usage caused by proto.size ( #27054 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-09-17 20:55:21 +08:00
bb6711f28c
Add ChunkCache: support get vector from storage ( #26142 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-15 10:21:20 +08:00
394c898b4c
Discard SyncDistribution set action from legacy querycoord ( #27027 )
...
Since Milvus in lower version (< 2.3.0), there is no load info in set action
which may corrupt data integrity and cause panicking
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-13 09:35:18 +08:00
e107d0794c
support complex delete expression ( #25752 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-09-12 10:19:17 +08:00
ac45af585b
Make segment loaded successful put in manager even ctx done ( #26992 )
...
Leave segment loaded in manager even wait other segment failed
See also #26908
Fix error case in distributed scenario
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-11 21:33:18 +08:00
2a5d574a0d
Fix querynodev2 concurrent load logic ( #26959 )
...
Fix logic error from #26926
function `waitSegmentLoadDone` shall return error when context is done
Make delegator control concurrency for each same segment
Related to #26908
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-10 07:41:18 +08:00
c6116d1819
Remove segment to LocalSegment type assertion ( #26931 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-09 10:35:16 +08:00
c8f9f22c4a
Fix segment loader return false success ( #26926 )
...
`waitSegmentLoadDone` did not check waitCh result is success or failure
after load return without error, delegator will assume all segments are loaded
This PR changes waitCh to loadResult with `sync.Cond` with `atomic.Int32` to represent status
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-08 16:41:16 +08:00
9166011c4a
Add float16 vector ( #25852 )
...
Signed-off-by: Writer-X <1256866856@qq.com>
2023-09-08 10:03:16 +08:00
28681276e2
Improve the retry of the rpc client ( #26795 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-06 17:43:14 +08:00
yah01
congqixia
yah01
cai.zhang
cqy123456
Bingyi Sun
aoiasd
MrPresent-Han
Enwei Jiao
Gao
zhenshan.cao
Jiquan Long
foxspy
xige-16
yihao.dai
SimFG
Xu Tong