milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
22,198 Commits
174 Branches
147 Tags
604 MiB
Commit Graph

426 Commits (2.5)

Author SHA1 Message Date
cai.zhang 78b66a29b6
fix: [2.5] Reduce task slot for standalone to 1/4 of normal datanode (#42809) 
issue: #42129 

master pr: #42808

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-06-20 16:38:46 +08:00
cai.zhang a69ae760e2
fix: Set worker totalSlot in standalone mode is half of cluster mode (#41731) 
issue: #41616 , #41732

master pr: #41730

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-05-09 18:30:58 +08:00
Chun Han 69a80b9ce3
enhance: resize high priority wqthreadpool dynamically(#40838) (#41549) 
related: #40838

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2025-04-28 00:44:39 +08:00
SimFG 18eb627533
fix: [2.5] Update logging context and upgrade dependencies (#41319) 
- issue: #41291
- pr: #41318

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2025-04-24 23:50:40 +08:00
foxspy d5977ec521
enhance: [2.5] add force rebuild index configuration (#41432) 
issue: #41431 
pr: #41473

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2025-04-23 21:44:38 +08:00
Xianhui Lin b3d3a268b3
fix:[2.5]Add debug memory freeing in sortStats (#41285) 
Add debug memory freeing in createJSONKeyIndex function
issue: https://github.com/milvus-io/milvus/issues/41218
pr:https://github.com/milvus-io/milvus/pull/41284

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-04-16 15:00:34 +08:00
liliu-z cb0f984155
enhance: Revert "separate for index completed (#40873)" (#41152) 
This reverts commit 23e579e324. #40873

issue: #39519

Signed-off-by: Li Liu <li.liu@zilliz.com>
 2025-04-08 17:36:30 +08:00
cai.zhang 0db5e0c4f6
enhance: [2.5]Deprecate disk params about indexing (#41078) 
issue: #40863 

master pr: #41045

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-04-07 11:36:34 +08:00
Chun Han 23e579e324
separate for index completed (#40873) 
related: https://github.com/milvus-io/milvus/issues/40781

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2025-04-05 10:20:24 +08:00
cai.zhang 0cfe835cac
enhance: [2.5] Support slot for index task and stats task (#40288) 
issue: #39101 

master pr: #39084

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-03-19 16:50:11 +08:00
Xianhui Lin 705b3c90a5
fix: Failed to rolling upgrade from v2.5.6 to new 2.5 version when enable JsonKeyStats (#40661) 
fix: Failed to rolling upgrade from v2.5.6 to new 2.5 version when
enable JsonKeyStats.The reason is that the file path of the jsonkeyindex
has changed.
issue: https://github.com/milvus-io/milvus/issues/40649
，https://github.com/milvus-io/milvus/issues/40669
https://github.com/milvus-io/milvus/issues/40707
master-pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-03-18 17:32:16 +08:00
Xianhui Lin f5e9dea2aa
fix: [2.5]fix the garbage cleanup logic of jsonkey stats && improve json key stats filer (#40039) 
fix: fix the garbage collection cleanup logic of jsonkey stats &&
improve json key stats filer
issue: https://github.com/milvus-io/milvus/issues/36995
https://github.com/milvus-io/milvus/issues/40034
https://github.com/milvus-io/milvus/issues/40041
https://github.com/milvus-io/milvus/issues/40106
https://github.com/milvus-io/milvus/issues/40138
pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-03-13 20:18:10 +08:00
cai.zhang f6c4ad52f0
enhance: [2.5] Refine task meta with key lock (#40353) 
issue: #39101 

master pr: #39084

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-03-13 19:18:08 +08:00
cai.zhang 2846cc5810
fix: [2.5] Skip executing stats for zero segment (#40449) 
issue : #40241 

master pr: #40448

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-03-07 16:16:03 +08:00
congqixia 709594f158
enhance: [2.5] Use v2 package name for pkg module (#40117) 
Cherry-pick from master
pr: #39990
Related to #39095

https://go.dev/doc/modules/version-numbers

Update pkg version according to golang dep version convention

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-23 00:46:01 +08:00
Xianhui Lin c1de61ff7c
fix: [2.5]Replace the position of EnabledJSONKeyStats (#40108) 
Replace the position of EnabledJSONKeyStats
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-02-22 14:35:54 +08:00
cai.zhang be30f5301d
enhance: Refine index task scheduler policy (#40104) 
master pr: #39084 

issue: #39101

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-02-22 11:25:54 +08:00
cai.zhang f904b73662
fix: [2.5]Skip text index creation when segment is zero after sorting (#39969) 
issue: #39961 

master pr: #39962

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-02-19 10:58:51 +08:00
Xianhui Lin f0964f769d
enhance: [2.5]Add json key inverted index in stats for optimization (#39876) 
Add json key inverted index in stats for optimization
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-02-16 20:12:15 +08:00
cai.zhang 4270174899
fix: [2.5] Add scalar index engine version for compatibility (#39236) 
issue: #39203 

master pr: #39204

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-01-14 21:01:01 +08:00
Zhen Ye adfc3f945e
enhance: record memory size (uncompressed) item for index (#38844) 
issue: #38715 
pr: #38770

- Current milvus use a serialized index size(compressed) for estimate
resource for loading.
- Add a new field MemSize (before compressing) for index to estimate
resource.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-01-14 10:33:06 +08:00
Zhen Ye 95809ca767
enhance: make new go package to manage proto (#39128) 
issue: #39095
pr: #39114

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-01-10 10:53:01 +08:00
cai.zhang 7a05b5bbea
fix: Fix stats task wrong RootPath when upload binlog (#38539) 
issue: #38336

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-12-18 00:18:45 +08:00
tinswzy 27229f7907
enhance: refine exists log print with ctx (#38080) 
issue: #35917 
Refines exists log print with ctx

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
 2024-12-14 22:36:44 +08:00
cai.zhang 0d7a89a4f8
fix: Use the correct RootPath when decompressing binlog in stats task (#38341) 
issue: #38336

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-12-11 16:16:42 +08:00
cai.zhang 9be106dedf
enhance: Refine task scheduler logs (#38334) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-12-11 15:00:44 +08:00
jaime 8ed019735c
enhance: add disk stats within system metrics (#38033) 
issue: ##36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-12-06 16:32:41 +08:00
cai.zhang b9357e4716
fix: Modify the batchsize of writer to timely flushing binlogs (#37692) 
issue: #37579 

If the schema includes large varchar fields, a few thousand rows can
reach hundreds of MB in size. Therefore, if the batch size of the
segment writer is large, it will produce relatively large `binlogs`,
which can cause datanode to run out of memory (OOM) during compaction.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-11-15 10:14:31 +08:00
yihao.dai 81879425e1
enhance: Optimize the performance of stats task (#37374) 
1. Increase the writer's `batchSize` to avoid multiple serialization
operations.
2. Perform asynchronous upload of binlog files to prevent blocking the
data processing flow.
3. Reduce multiple calls to `writer.Flush()`.

issue: https://github.com/milvus-io/milvus/issues/37373

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-11-08 10:08:27 +08:00
foxspy 3224e58c5b
enhance: add unify vector index config management (#36846) 
issue: #34298

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-11-01 06:18:21 +08:00
jaime 4746f47282
feat: management WebUI homepage (#36822) 
issue: #36784
1. Implement an embedded web server for WebUI access.  
2. Complete the homepage development.

Home page demo:
<img width="2177" alt="iShot_2024-10-10_17 57 34"
src="https://github.com/user-attachments/assets/38539917-ce09-4e54-a5b5-7f4f7eaac353">

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-10-23 11:29:28 +08:00
foxspy 3de57ec4fa
enhance: add vector index mgr to remove vector index type dependency (#36843) 
issue: #34298

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-10-17 22:15:25 +08:00
aoiasd 5ec4163d0f
feat: support bm25 logs mixcompaction (#36072) 
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-10-14 16:57:22 +08:00
Rijin-N a05a37a583
enhance: GCS native support (GCS implemented using Google Cloud Storage libraries) (#36214) 
Native support for Google cloud storage using the Google Cloud Storage
libraries. Authentication is performed using GCS service account
credentials JSON.

Currently, Milvus supports Google Cloud Storage using S3-compatible APIs
via the AWS SDK. This approach has the following limitations:

1. Overhead: Translating requests between S3-compatible APIs and GCS can
introduce additional overhead.
2. Compatibility Limitations: Some features of the original S3 API may
not fully translate or work as expected with GCS.

To address these limitations, This enhancement is needed.

Related Issue: #36212
 2024-09-30 13:23:32 +08:00
cai.zhang 517f8b3755
enhance: Refine the code for returning error (#36103) 
issue: #36023

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-15 15:23:14 +08:00
cai.zhang 8395c8a8db
enhance: Update stats task to optional (#35947) 
issue: #33744

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-12 20:37:08 +08:00
Jiquan Long 89bf226f0b
feat: support keyword text match (#35923) 
fix: #35922

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-09-10 15:11:08 +08:00
CharlesFeng 4850641943
fix: BinlogDeserializeReader leak (#36087) 
https://github.com/milvus-io/milvus/issues/36086

Signed-off-by: fengjun2016 <jornfeng@gmail.com>
 2024-09-10 12:43:07 +08:00
cai.zhang 2c9bb4dfa3
feat: Support stats task to sort segment by PK (#35054) 
issue: #33744 

This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-02 14:19:03 +08:00
Zhen Ye a773836b89
enhance: optimize milvus core building (#35610) 
issue: #35549,#35611,#35633

- remove milvus_segcore milvus_indexbuilder..., add libmilvus_core
- core building only link once
- move opendal compilation into cmake
- fix odr

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-08-23 12:35:02 +08:00
zhenshan.cao aa247f192d
enhance: remove unused code for StorageV2 (#35132) 
issue: https://github.com/milvus-io/milvus/issues/34168

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-08-01 12:08:13 +08:00
cai.zhang 575ce91039
fix: Get current index version from knowhere before building index (#34901) 
issue: #34900

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-23 10:23:42 +08:00
Patrick Weizhi Xu 104d0966b7
feat: support partition key isolation (#34336) 
issue: #34332

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-07-11 19:01:35 +08:00
wei liu ebc68d2774
fix: Indexnode stuck at stopping progress cause by wrong lifetime control (#34558) 
issue: #34557

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-10 15:12:13 +08:00
congqixia fd922d921a
enhance: Add nilness linter and fix some small issues (#34049) 
Add `nilness` for govet linter and fixed some detected issues

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-24 14:52:03 +08:00
cai.zhang b69e9093c8
fix: Fallback field type when it isn't in request (#33832) 
issue: #33432

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-14 09:55:56 +08:00
congqixia 512ea6be5f
enhance: Avoid merging insert data when buffering insert msgs (#33562) 
See also #33561

This PR:
- Use zero copy when buffering insert messages
- Make `storage.InsertCodec` support serialize multiple insert data
chunk into same batch binlog files

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-13 11:15:56 +08:00
cai.zhang 27cc9f2630
enhance: Support analyze data (#33651) 
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
 2024-06-06 17:37:51 +08:00
cai.zhang 412ccfbb20
enhance: Refine IndexNode code and ensure compatibility (#33458) 
issue: #33432 , #33183

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-05 19:35:52 +08:00
cai.zhang be77ceba84
enhance: Use proto for passing info in cgo (#33184) 
issue: #33183

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-05-23 10:31:40 +08:00