9166011c4a
Add float16 vector ( #25852 )
...
Signed-off-by: Writer-X <1256866856@qq.com>
2023-09-08 10:03:16 +08:00
548c82eca5
Refactor storage.MergeInsertData() to optimize the merging process ( #26839 )
...
Benchmark Milvus with https://github.com/qdrant/vector-db-benchmark and
specify the datasets as 'deep-image-96-angular'. Meanwhile, do perf
profiling during 'upload + index' stage of vector-db-benchmark and see
the following hot spots.
39.59%--github.com/milvus-io/milvus/internal/storage.MergeInsertData
|
|--21.43%--github.com/milvus-io/milvus/internal/storage.MergeFieldData
| |
| |--17.22%--runtime.memmove
| |
| |--1.53%--asm_exc_page_fault
| ......
|
|--18.16%--runtime.memmove
|
|--1.66%--asm_exc_page_fault
......
The hot code path is in storage.MergeInsertData() which updates
buffer.buffer by creating a new 'InsertData' instance and merging both
the old buffer.buffer and addedBuffer into it. When it calls golang
runtime.memmove to move buffer.buffer which is with big size (>1M), the
hot spots appear.
To avoid the above overhead, update storage.MergeInsertData() by
appending addedBuffer to buffer.buffer, instead of moving buffer.buffer
and addedBuffer to a new 'InsertData'. This change removes the hot spots
'runtime.memmove' from perf profiling output. Additionally, the 'upload
+ index' time, which is one performance metric of vector-db-benchmark,
is reduced around 60% with this change.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
2023-09-05 21:41:48 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
ebd0279d3f
Check error by Error() and NoError() for better report message ( #24736 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-08 15:36:36 +08:00
73a181d226
Fix get vector it timeout and improve some string const usage ( #24141 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-16 17:41:22 +08:00
967a97b9bd
Support json & array types ( #23408 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
2023-04-20 11:32:31 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
732986aa04
Remove fmt.Print from internal package ( #22722 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-14 17:36:05 +08:00
d126f06946
Decouple mq module from internal proto definition ( #22536 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-03-04 23:21:50 +08:00
949d5d078f
Fix memory calculation in dataCodec ( #21800 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-28 11:09:52 +08:00
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
d7f38a803d
Separate some proto files ( #19218 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
4de1bfe5bc
Add cpp data codec ( #18538 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: zhagnlu lu.zhang@zilliz.com
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-09-09 22:12:34 +08:00
68a6587374
Set insert&stats binlog timestamp range ( #19005 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-04 09:05:09 +08:00
68b1b82faf
Remove DataKV interface ( #16692 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-04-28 21:03:47 +08:00
205c92e54b
Support insert string data ( #15993 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-03-25 14:27:25 +08:00
3121619758
Chunk manager support scalar data ( #16010 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-03-11 14:39:59 +08:00
f71651e294
Support column-based insert data in message stream ( #15802 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-03-04 15:09:56 +08:00
92c8e32ebd
Let MemoryKV.Load return error when key not exist ( #15814 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-02 18:51:55 +08:00
dd860a76cf
[skip e2e]Update license for storage util ( #14453 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-12-28 20:11:55 +08:00
9d5bcd3e3a
Close event and binlog reader ( #12173 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-11-22 17:27:14 +08:00
93149c5ad9
Load growing segment in query node ( #11664 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-11-12 18:27:10 +08:00
a0a3a889e7
Add common endian for global use ( #11092 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-11-02 18:16:32 +08:00
5b42a3223c
Increase compatibility for EstimateMemorySize interface ( #10603 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-10-26 15:34:21 +08:00
a63ef91c74
Fix static-check ( #9776 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-10-13 13:22:33 +08:00
1f224c4b2e
Optimize the ut of storage utils ( #9740 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-10-12 19:47:08 +08:00
f85271cf3f
Estimate memory size by descriptor event ( #9688 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-10-12 17:00:34 +08:00
7daa319dc2
[skip ci] Rename EstimateMemorySize to GetBinlogSize ( #9651 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-10-11 18:20:30 +08:00
1bc4b36617
Estimate the memory size of binlog file ( #9612 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-10-11 14:10:48 +08:00
Xu Tong
bjzhjing
congqixia
yah01
Enwei Jiao
jaime
Xiaofan
SimFG
xige-16
Jiquan Long
Cai Yudong
XuanYang-cn
godchen
bigsheeper
cai.zhang