milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
21,884 Commits
170 Branches
144 Tags
577 MiB
Commit Graph

32 Commits (eb046863485fdf3e130fc60484485c901b81276b)

Author SHA1 Message Date
congqixia cb7f2fa6fd
enhance: Use v2 package name for pkg module (#39990) 
Related to #39095

https://go.dev/doc/modules/version-numbers

Update pkg version according to golang dep version convention

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-22 23:15:58 +08:00
smellthemoon 8b974c5742
enhance: support compact if lack of binlog (#40000) 
https://github.com/milvus-io/milvus/issues/39718

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2025-02-22 10:51:56 +08:00
Ted Xu 2978b0890e
enhance: iterative download data during compaction to reduce memory cost (#39724) 
See #37234

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2025-02-13 10:36:47 +08:00
sthuang 15c8798b93
feat: storage v2 serde reader and writer (#39667) 
related: https://github.com/milvus-io/milvus/issues/39173

---------

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2025-02-11 16:00:46 +08:00
Ted Xu 427b6a4c94
enhance: reduce stats task cost by skipping ser/de (#39568) 
See #37234

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2025-02-06 17:14:45 +08:00
XuanYang-cn 7261128df0
enhance: Remove not inuse binlog iterators (#39243) 
1. Remove datanode/iterators/binlog_iterator
2. Remove storage/binlog_iterator/MergeIterator

See also: #39242

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2025-01-21 11:15:04 +08:00
Zhen Ye bb8d1ab3bf
enhance: make new go package to manage proto (#39114) 
issue: #39095

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-01-10 10:49:01 +08:00
Ted Xu 262a994d6d
enhance: generally improve the performance of mix compactions (#37163) 
See #37234

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-10-29 18:12:20 +08:00
congqixia 4ee6c69217
enhance: Add Segment Level in milvus segment info APIs (#34763) 
See also #34746

This PR add segment level field in response of
`GetPersistentSegmentInfo` and `GetQuerySegmentInfo`

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-26 10:01:46 +08:00
shaoting-huang 88b373b024
enhance: binlog primary key turn off dict encoding (#34358) 
issue: #34357 

Go Parquet uses dictionary encoding by default, and it will fall back to
plain encoding if the dictionary size exceeds the dictionary size page
limit. Users can specify custom fallback encoding by using
`parquet.WithEncoding(ENCODING_METHOD)` in writer properties. However,
Go Parquet [fallbacks to plain
encoding](e65c1e295d/go/parquet/file/column_writer_types.gen.go.tmpl (L238))
rather than custom encoding method users provide. Therefore, this patch
only turns off dictionary encoding for the primary key.

With a 5 million auto ID primary key benchmark, the parquet file size
improves from 13.93 MB to 8.36 MB when dictionary encoding is turned
off, reducing primary key storage space by 40%.

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2024-07-17 17:47:44 +08:00
Buqian Zheng 8a1017a152
enhance: add helpers to parse sparse float vector in JSON (#32543) 
issue: #29419

added helper functions to parse JSON representation of sparse float
vectors, will be used by both the restful server and the import utils.

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-04-25 14:47:24 +08:00
Ted Xu dc5ea6f17c
feat: adding binlog streaming writer (#31537) 
See #31679

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-04-11 10:33:20 +08:00
Buqian Zheng d7dbc3c9d8
fix: [sparse float vector] support the new streaming deserialize reader (#31325) 
issue: https://github.com/milvus-io/milvus/issues/31324

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-03-17 13:59:04 +08:00
Ted Xu 987d9023a5
enhance: Enable binlog deserialize reader in datanode compaction (#31036) 
See #30863

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-03-08 18:25:02 +08:00
Ted Xu 71adafa933
enhance: adding a streaming deserialize reader for binlogs (#30860) 
See #30863

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-03-04 19:31:09 +08:00
SimFG 26f06dd732
Format the code (#27275) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-21 09:45:27 +08:00
congqixia 41af0a98fa
Use go-api/v2 for milvus-proto (#24770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
yah01 ebd0279d3f
Check error by Error() and NoError() for better report message (#24736) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-08 15:36:36 +08:00
aoiasd c84bdcea49
merge stats log when segment flushing or compacting (#23570) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-05-29 10:21:28 +08:00
jaime c9d0c157ec
Move some modules from internal to public package (#22572) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-04-06 19:14:32 +08:00
yah01 081572d31c
Refactor QueryNode (#21625) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-03-27 00:42:00 +08:00
Xiaofan 949d5d078f
Fix memory calculation in dataCodec (#21800) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-01-28 11:09:52 +08:00
SimFG a55f739608
Separate public proto files (#19782) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-10-16 20:49:27 +08:00
SimFG d7f38a803d
Separate some proto files (#19218) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-09-16 16:56:49 +08:00
godchen bb7a0766fe
Add dependency factory (#16204) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-04-07 22:05:32 +08:00
xige-16 99984b88e1
Support delete varChar value (#16229) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-04-02 17:43:29 +08:00
XuanYang-cn d2c83c24f4
[skip e2e]Update license for storage iterator (#13890) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-21 18:48:57 +08:00
XuanYang-cn 6bf0c4326d
Fix compaction generates wrong insert binlogs (#12204) 
This PR
- changes Get() to GetRow() in BinlogIterator
- removes Get() and Length() of FieldData
- adds ut in BinlogIterator for floatvector and binaryvector
- fix a log bug

Fixes: #12146

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-23 17:09:15 +08:00
godchen 9d5bcd3e3a
Close event and binlog reader (#12173) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-22 17:27:14 +08:00
XuanYang-cn 9bd3664672
Fix compaction bug (#11816) 
This PR fixes:
- timeout in compaction
- BinlogIterator using rawID as PKField bug

See also: #11757

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-17 10:07:13 +08:00
Xiangyu Wang 0b4d7dc7f6
[skip ci]Improve import statements (#11713) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-11-12 11:10:47 +08:00
sunby db53eccec9
Add binlog iterators (#10173) 
issue: #9530
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-20 11:52:35 +08:00