2a1356985d
enhance: support null in go payload ( #32296 )
...
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-06-19 17:08:00 +08:00
0ecd694305
enhance: legacy code clean up ( #33838 )
...
issue: #33839
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-06-14 14:25:56 +08:00
976928ecd1
fix: fix fp16/bf16 some code missing and add more fp16/bf16 test ( #31612 )
...
issue: #31534
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-03-28 14:11:10 +08:00
b1a1cca10b
feat: add more operation detail info for better allocation ( #30438 )
...
issue: #30436
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-03-28 06:33:11 +08:00
31cf849f68
enhance: Support retriving file size from importutilv2.Reader ( #31533 )
...
To reduce the overhead caused by listing the S3 objects, add an
interface to importutil.Reader to retrieve file sizes.
issue: https://github.com/milvus-io/milvus/issues/31532 ,
https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-25 20:29:07 +08:00
c3264ca3e3
feat: support segment pruner ( #31003 )
...
related: #30376
2024-03-22 13:57:06 +08:00
3c80083f51
feat: [Sparse Float Vector] add sparse vector support to milvus components ( #30630 )
...
add sparse float vector support to different milvus components,
including proxy, data node to receive and write sparse float vectors to
binlog, query node to handle search requests, index node to build index
for sparse float column, etc.
https://github.com/milvus-io/milvus/issues/29419
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-13 14:32:54 -07:00
a0537156c0
enhance: delete codc deserialize data by stream batch ( #30407 )
...
relate: https://github.com/milvus-io/milvus/issues/30404
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-02-06 17:04:25 +08:00
e429965f32
Add float16 approve for multi-type part ( #28427 )
...
issue：https://github.com/milvus-io/milvus/issues/22837
Add bfloat16 vector, add the index part of float16 vector.
Signed-off-by: Writer-X <1256866856@qq.com>
2024-01-11 15:48:51 +08:00
f18a7191f2
enhance: make `ColumnBasedInsertMsgToInsertData` check field missing ( #29758 )
...
fix : #29757
In previous code, `ColumnBasedInsertMsgToInsertData` adds empty field if
the insertMsg parameter does not have the column schema defined. This
may lead to unexpected behavior of caller functions.
This PR:
- Add column missing check
- Add column length check
- Generate BlobInfo for ColumnBasedInsertMsgToInsertData result
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-09 11:50:48 +08:00
e576271a24
Fix buffer FieldData has no `ElementType` and array logsize always zero ( #28295 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-09 14:16:20 +08:00
9658367a3c
Refine chunk manager errors ( #27590 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-31 12:18:15 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
9166011c4a
Add float16 vector ( #25852 )
...
Signed-off-by: Writer-X <1256866856@qq.com>
2023-09-08 10:03:16 +08:00
548c82eca5
Refactor storage.MergeInsertData() to optimize the merging process ( #26839 )
...
Benchmark Milvus with https://github.com/qdrant/vector-db-benchmark and
specify the datasets as 'deep-image-96-angular'. Meanwhile, do perf
profiling during 'upload + index' stage of vector-db-benchmark and see
the following hot spots.
39.59%--github.com/milvus-io/milvus/internal/storage.MergeInsertData
|
|--21.43%--github.com/milvus-io/milvus/internal/storage.MergeFieldData
| |
| |--17.22%--runtime.memmove
| |
| |--1.53%--asm_exc_page_fault
| ......
|
|--18.16%--runtime.memmove
|
|--1.66%--asm_exc_page_fault
......
The hot code path is in storage.MergeInsertData() which updates
buffer.buffer by creating a new 'InsertData' instance and merging both
the old buffer.buffer and addedBuffer into it. When it calls golang
runtime.memmove to move buffer.buffer which is with big size (>1M), the
hot spots appear.
To avoid the above overhead, update storage.MergeInsertData() by
appending addedBuffer to buffer.buffer, instead of moving buffer.buffer
and addedBuffer to a new 'InsertData'. This change removes the hot spots
'runtime.memmove' from perf profiling output. Additionally, the 'upload
+ index' time, which is one performance metric of vector-db-benchmark,
is reduced around 60% with this change.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
2023-09-05 21:41:48 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
73a181d226
Fix get vector it timeout and improve some string const usage ( #24141 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-16 17:41:22 +08:00
967a97b9bd
Support json & array types ( #23408 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
2023-04-20 11:32:31 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
697dedac7e
Use cockroachdb/errors to replace other error pkg ( #22390 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
949d5d078f
Fix memory calculation in dataCodec ( #21800 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-28 11:09:52 +08:00
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
d7f38a803d
Separate some proto files ( #19218 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
68a6587374
Set insert&stats binlog timestamp range ( #19005 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-04 09:05:09 +08:00
515d0369de
Support string type in segcore ( #16546 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
2022-04-29 13:35:49 +08:00
68b1b82faf
Remove DataKV interface ( #16692 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-04-28 21:03:47 +08:00
205c92e54b
Support insert string data ( #15993 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-03-25 14:27:25 +08:00
3121619758
Chunk manager support scalar data ( #16010 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-03-11 14:39:59 +08:00
f71651e294
Support column-based insert data in message stream ( #15802 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-03-04 15:09:56 +08:00
dd860a76cf
[skip e2e]Update license for storage util ( #14453 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-12-28 20:11:55 +08:00
89a26b2679
Fix redundant init declaration in storage/util.go ( #13381 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-15 11:13:10 +08:00
001752eb91
Fix zero length slice declaration in storage/util.go ( #12412 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-30 09:57:44 +08:00
93149c5ad9
Load growing segment in query node ( #11664 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-11-12 18:27:10 +08:00
5b42a3223c
Increase compatibility for EstimateMemorySize interface ( #10603 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-10-26 15:34:21 +08:00
f85271cf3f
Estimate memory size by descriptor event ( #9688 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-10-12 17:00:34 +08:00
7daa319dc2
[skip ci] Rename EstimateMemorySize to GetBinlogSize ( #9651 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-10-11 18:20:30 +08:00
1bc4b36617
Estimate the memory size of binlog file ( #9612 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-10-11 14:10:48 +08:00
smellthemoon
shaoting-huang
cqy123456
SimFG
yihao.dai
Chun Han
Buqian Zheng
aoiasd
Xu Tong
congqixia
yah01
bjzhjing
Enwei Jiao
jaime
Xiaofan
xige-16
Jiquan Long
XuanYang-cn
cai.zhang