7514eece4c
enhance: [2.5] Add mutex and range check preventing concurrent del ( #44128 ) ( #44202 )
...
Cherry-pick from master
pr: #44128
This PR adds a mutex prevent concurrent applying delete on same segment
and check latestDeltaTimestamp to skip overlapping delete range
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-09-04 10:35:54 +08:00
6c29689ca2
enhance: support expr result cache ( #43882 )
...
cherry-pick from pr: #43923
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-08-26 11:19:57 +08:00
7141f96f00
fix: [2.5]add segment lock in LoadTextIndex and LoadJSONKeyIndex ( #43815 )
...
fix: add segment lock in LoadTextIndex and LoadJSONKeyIndex
issue:https://github.com/milvus-io/milvus/issues/43572
pr: https://github.com/milvus-io/milvus/pull/43811
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-08-18 10:21:57 +08:00
f033294dc1
fix: try to get span raw data for variable length data type( #43544 ) ( #43703 )
...
related: #43544
pr: https://github.com/milvus-io/milvus/pull/43705
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2025-08-04 11:25:39 +08:00
33e9b873de
enhance: add logs for count(*) ( #43001 ) ( #43249 )
...
pr: #43001
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-07-11 19:14:48 +08:00
a55c371261
fix: querynode upgrade from 2.5 get stucked ( #42503 )
...
issue: #42492
pr: #42502
- querynode graceful stop can be done if there's only L0 segment exists.
Signed-off-by: chyezh <chyezh@outlook.com>
2025-06-05 15:56:33 +08:00
6e18ededab
fix: [2.5] mockery too unavailable after upgrade golang version ( #41522 )
...
- issue: ##41291
- pr: #41481
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-04-25 14:40:40 +08:00
18eb627533
fix: [2.5] Update logging context and upgrade dependencies ( #41319 )
...
- issue: #41291
- pr: #41318
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-04-24 23:50:40 +08:00
699f8a6ada
fix: [2.5] change log info to debug for collection ref ( #41269 )
...
master: https://github.com/milvus-io/milvus/pull/41267
issue: https://github.com/milvus-io/milvus/issues/41268
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
2025-04-15 17:28:31 +08:00
cb0f984155
enhance: Revert "separate for index completed ( #40873 )" ( #41152 )
...
This reverts commit 23e579e324#40873
issue: #39519
Signed-off-by: Li Liu <li.liu@zilliz.com>
2025-04-08 17:36:30 +08:00
23e579e324
separate for index completed ( #40873 )
...
related: https://github.com/milvus-io/milvus/issues/40781
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2025-04-05 10:20:24 +08:00
705b3c90a5
fix: Failed to rolling upgrade from v2.5.6 to new 2.5 version when enable JsonKeyStats ( #40661 )
...
fix: Failed to rolling upgrade from v2.5.6 to new 2.5 version when
enable JsonKeyStats.The reason is that the file path of the jsonkeyindex
has changed.
issue: https://github.com/milvus-io/milvus/issues/40649
，https://github.com/milvus-io/milvus/issues/40669
https://github.com/milvus-io/milvus/issues/40707
master-pr: https://github.com/milvus-io/milvus/pull/38039
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-03-18 17:32:16 +08:00
f5e9dea2aa
fix: [2.5]fix the garbage cleanup logic of jsonkey stats && improve json key stats filer ( #40039 )
...
fix: fix the garbage collection cleanup logic of jsonkey stats &&
improve json key stats filer
issue: https://github.com/milvus-io/milvus/issues/36995
https://github.com/milvus-io/milvus/issues/40034
https://github.com/milvus-io/milvus/issues/40041
https://github.com/milvus-io/milvus/issues/40106
https://github.com/milvus-io/milvus/issues/40138
pr: https://github.com/milvus-io/milvus/pull/38039
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-03-13 20:18:10 +08:00
cd8f1fe0e4
enhance: [2.5] Remove unnecessary collection and partition label from the metrics ( #40593 )
...
/kind improvement
pr: https://github.com/milvus-io/milvus/pull/39536
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-13 12:24:08 +08:00
683b26ffb7
feat: cherry pick json path index ( #40313 )
...
issue: #35528
pr: #36750
this pr includes json path index pr and some related prs:
1. update tantivy version #39253
2. json path index #36750
3. fall back to brute force #40076
4. term filter #40140
5. bug fix #40336
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-03-10 22:14:05 +08:00
709594f158
enhance: [2.5] Use v2 package name for pkg module ( #40117 )
...
Cherry-pick from master
pr: #39990
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-23 00:46:01 +08:00
f0964f769d
enhance: [2.5]Add json key inverted index in stats for optimization ( #39876 )
...
Add json key inverted index in stats for optimization
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-02-16 20:12:15 +08:00
d121ac3a7a
enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) ( #39180 )
...
issue: https://github.com/milvus-io/milvus/issues/27678
related: https://github.com/milvus-io/milvus/pull/39753
some raw data status will change:
Intermin index has raw data:
<html xmlns:v="urn:schemas-microsoft-com:vml"
xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:x="urn:schemas-microsoft-com:office:excel"
xmlns="http://www.w3.org/TR/REC-html40 ">
<head>
<meta name=ProgId content=Excel.Sheet>
<meta name=Generator content="Microsoft Excel 15">
<link id=Main-File rel=Main-File
href="file:////Users/cqy/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip.htm">
<link rel=File-List
href="file:////Users/cqy/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip_filelist.xml">
<!--table
{mso-displayed-decimal-separator:"\.";
mso-displayed-thousand-separator:"\,";}
@page
{margin:.75in .7in .75in .7in;
mso-header-margin:.3in;
mso-footer-margin:.3in;}
.font5
{color:windowtext;
font-size:9.0pt;
font-weight:400;
font-style:normal;
text-decoration:none;
font-family:等线;
mso-generic-font-family:auto;
mso-font-charset:134;}
tr
{mso-height-source:auto;
mso-ruby-visibility:none;}
col
{mso-width-source:auto;
mso-ruby-visibility:none;}
br
{mso-data-placement:same-cell;}
td
{padding-top:1px;
padding-right:1px;
padding-left:1px;
mso-ignore:padding;
color:black;
font-size:12.0pt;
font-weight:400;
font-style:normal;
text-decoration:none;
font-family:等线;
mso-generic-font-family:auto;
mso-font-charset:134;
mso-number-format:General;
text-align:general;
vertical-align:middle;
border:none;
mso-background-source:auto;
mso-pattern:auto;
mso-protection:locked visible;
white-space:nowrap;
mso-rotate:0;}
ruby
{ruby-align:left;}
rt
{color:windowtext;
font-size:9.0pt;
font-weight:400;
font-style:normal;
text-decoration:none;
font-family:等线;
mso-generic-font-family:auto;
mso-font-charset:134;
mso-char-type:none;
display:none;}
-->
</head>
<body link="#0563C1" vlink="#954F72">
sparse vector | growing segment | sealed segment
-- | -- | --
BM25 | no | no
IP | yes | no
| |
dense vector | growing segment | sealed segment
ivf flat cc | yes | yes
scann_dvr | no | no
</body>
</html>
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2025-02-13 09:56:47 +08:00
1b14f5a7ab
fix: [2.5] Segments return both growing&sealed result ( #39789 )
...
Cherry-pick from master
pr: #39787
Previous PR #38311
Logic bug found in #38348
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-12 14:44:48 +08:00
c0cc8a59f7
fix: [2.5] Return early when skip load pk index ( #39763 )
...
Cherry pick from master
pr: #39762
Previous PR #39437 only print log and add index while load operation is
still executed. This PR return early when segment decides not to load PK
index.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-11 11:04:50 +08:00
758ac5a492
fix: [2.5]Adjust segment loader's memory estimate for intermin indexes ( #39509 )
...
issue: https://github.com/milvus-io/milvus/issues/27678
related 2.4 pr: https://github.com/milvus-io/milvus/pull/39508
related 2.5 pr: https://github.com/milvus-io/milvus/pull/39509
related master pr: https://github.com/milvus-io/milvus/pull/39507
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2025-01-24 11:19:14 +08:00
8475d8b193
fix: cherry pick warmup async ( #39402 ) ( #39474 )
...
related pr: https://github.com/milvus-io/milvus/pull/38690
issue: https://github.com/milvus-io/milvus/issues/38692
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-01-21 15:25:05 +08:00
f912cbf86d
enhance:[Cherry-pick] skip load bm25 sparse row data ( #39207 )
...
pr: https://github.com/milvus-io/milvus/pull/39078
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-01-21 11:05:04 +08:00
8e2e828a78
enhance: [2.5] Skip loading pk index for sorted segment in loader ( #39438 )
...
Cherry pick from master
pr: #39437
Related to #39339
Previous PR #39389 only skips append index into segment
Also related to #39428
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-20 18:07:04 +08:00
c945efa853
enhance: [2.5] Remove mutex from SegmentManger in querynode ( #39051 ) ( #39282 )
...
Remove mutex from SegmentManger in querynode to prevent mutex
contention.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/39051
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-16 13:27:02 +08:00
21df11b37c
enhance: [2.5] Add try-catch and return CStatus for NewCollection ( #39279 ) ( #39303 )
...
Cherry pick from master
pr: #39279
Related to #28795
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-16 10:13:06 +08:00
b658467107
enhance: [2.5] Unify LoadStateLock RLock & PinIf ( #39206 ) ( #39255 )
...
Cherry-pick from master
pr: #39206 #39308
Related to #39205
This PR merge `RLock` & `PinIfNotReleased` into `PinIf` function
preventing segment being released before any Read operation finished.
---------
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-16 00:33:03 +08:00
5b0bb4c04e
enhance: [2.5] Reduce memory usage of BF in DataNode and QueryNode ( #38913 )
...
1. DataNode: Skip generating BF during the insert phase (BF will be
regenerated during the sync phase).
2. QueryNode: Skip generating or maintaining BF for growing segments;
deletion checks will be handled in the segcore.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38129
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 13:59:00 +08:00
b91c0a8079
enhance: [2.5] Optimize GetLocalDiskSize and segment loader mutex ( #38907 )
...
1. Make the segment loader lock protect only the resource.
2. Optimize GetDiskUsage to avoid excessive overhead.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38599
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-14 20:43:06 +08:00
adfc3f945e
enhance: record memory size (uncompressed) item for index ( #38844 )
...
issue: #38715
pr: #38770
- Current milvus use a serialized index size(compressed) for estimate
resource for loading.
- Add a new field MemSize (before compressing) for index to estimate
resource.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-14 10:33:06 +08:00
8fadcde403
fix: [2.5] Fix incorrect memory estimation for small segments ( #38909 )
...
Skip estimation index memory logic for segments without index file.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38813
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-13 21:49:12 +08:00
95809ca767
enhance: make new go package to manage proto ( #39128 )
...
issue: #39095
pr: #39114
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:53:01 +08:00
04f07f00b7
fix: the retrieve plan on heap is used after free when reduce ( #38841 )
...
issue: #38731
pr: #38840
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-30 15:52:51 +08:00
52de43dbeb
enhance: add multiply factor when loading index ( #38722 )
...
issue: #38715
pr: #38716
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-25 10:50:58 +08:00
2afe2eaf3e
feat: support to replicate collection when the services contains the system tt msg ( #37559 )
...
- issue: #37105
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-12-17 09:08:46 +08:00
27229f7907
enhance: refine exists log print with ctx ( #38080 )
...
issue: #35917
Refines exists log print with ctx
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-14 22:36:44 +08:00
8977454311
enhance: support recall estimation ( #38017 )
...
issue: #37899
Only `search` api will be supported
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-12-11 20:40:48 +08:00
7c55649585
enhance: Refine querynode collection number metrics ( #38350 )
...
Related to #37630
Previously the loaded collection metrics was calculated via scanning all
loaded segment in segment manager, which is slow and buggy
implementation.
This PR:
- Move collection num metrics to collection manager
- Remove deprecated loaded partition metrics update logic
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-12-11 11:36:43 +08:00
b8e3795374
enhance: Add secondary index for querynode segment manager ( #38311 )
...
Related to #37630
Add secondary index with vchannel to reduce `GetBy` rlock holding time
when segment number is large.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-12-10 18:18:42 +08:00
051bc280dd
enhance: Make dynamic load/release partition follow targets ( #38059 )
...
Related to #37849
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-12-05 16:24:40 +08:00
618f0cb728
enhance: Put release segment and other misc cgo call into pool ( #38186 )
...
Related to #30273
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-12-05 11:04:40 +08:00
c6dcef7b84
enhance: move segcore codes of segment into one package ( #37722 )
...
issue: #33285
- move most cgo opeartions related to search/query into segcore package
for reusing for streamingnode.
- add go unittest for segcore operations.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-29 10:22:36 +08:00
cb6542339e
enhance: Mark cgo thread with tag name ( #38000 )
...
Related to #37999
This PR add `SetThreadName` API for marking cgo thread and utilize it
when initializing cgo worker.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-26 11:22:35 +08:00
1ed686783f
enhance: Use `PrimaryKeys` to replace interface slice for segment delete ( #37880 )
...
Related to #35303
Reduce temporary memory usage for PK interface for segment delete.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-22 11:52:33 +08:00
92e6ee6285
enhance: Use load pool for `CreateTextIndex` ( #37898 )
...
Related to #37895
Only resolves the starving issue which caused goroutine leakage
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-22 10:06:33 +08:00
ee54a98578
enhance: Add cgo call metrics for load/write API ( #37405 )
...
Cgo API cost is not observerable since not metrics is related to them.
This PR add metrics for some sync cgo call related to load & write
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-07 10:06:25 +08:00
3106384fc4
enhance: Return deltadata for `DeleteCodec.Deserialize` ( #37214 )
...
Related to #35303 #30404
This PR change return type of `DeleteCodec.Deserialize` from
`storage.DeleteData` to `DeltaData`, which
reduces the memory usage of interface header.
Also refine `storage.DeltaData` methods to make it easier to usage.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-29 12:04:24 +08:00
5a0135727d
fix: Check resource when loading deltalogs ( #37195 )
...
Related to #36887
`LoadDeltaLogs` API did not check memory usage. When system is under
high delete load pressure, this could result into OOM quit.
This PR add resource check for `LoadDeltaLogs` actions and separate
internal deltalog loading function with public one.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-29 10:04:25 +08:00
224d797f94
fix: Use singleton delete pool and avoid goroutine leakage ( #37220 )
...
Related to #36887
Previously using newly create pool per request shall cause goroutine
leakage. This PR change this behavior by using singleton delete pool.
This change could also provide better concurrency control over delete
memory usage.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-29 10:02:24 +08:00
7774b7275e
enhance: Replace PrimaryKey slice with PrimaryKeys saving memory ( #37127 )
...
Related to #35303
Slice of `storage.PrimaryKey` will have extra interface cost for each
element, which may cause notable memory usage when delta row count
number is large.
This PR replaces PrimaryKey slice with PrimaryKeys interface saving the
extra interface cost.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-28 10:29:30 +08:00
congqixia
zhagnlu
Xianhui Lin
Chun Han
wei liu
Zhen Ye
SimFG
Spade A
liliu-z
yihao.dai
Bingyi Sun
cqy123456
aoiasd
tinswzy
Gao