milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
20,270 Commits
152 Branches
122 Tags
588 MiB
Commit Graph

13 Commits (531092c0313cc5dafed58710d78a60c412020a5a)

Author SHA1 Message Date
Ted Xu eae4dfca7b
fix: reuse deserialize result to help improve memory management (#34507) 
Fixed #33268
The original reuse is broken by #33359

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-07-09 14:12:10 +08:00
shaoting-huang f4dd7c7efb
enhance: add delta log stream new format reader and writer (#34116) 
issue: #34123

Benchmark case: The benchmark run the go benchmark function
`BenchmarkDeltalogFormat` which is put in the Files changed. It tests
the performance of serializing and deserializing from two different data
formats under a 10 million delete log dataset.

Metrics: The benchmarks measure the average time taken per operation
(ns/op), memory allocated per operation (MB/op), and the number of
memory allocations per operation (allocs/op).
| Test Name | Avg Time (ns/op) | Time Comparison | Memory Allocation
(MB/op) | Memory Comparison | Allocation Count (allocs/op) | Allocation
Comparison |

|---------------------------------|------------------|-----------------|---------------------------|-------------------|------------------------------|------------------------|
| one_string_format_reader | 2,781,990,000 | Baseline | 2,422 | Baseline
| 20,336,539 | Baseline |
| pk_ts_separate_format_reader | 480,682,639 | -82.72% | 1,765 | -27.14%
| 20,396,958 | +0.30% |
| one_string_format_writer | 5,483,436,041 | Baseline | 13,900 |
Baseline | 70,057,473 | Baseline |
| pk_and_ts_separate_format_writer| 798,591,584 | -85.43% | 2,178 |
-84.34% | 30,270,488 | -56.78% |

Both read and write operations show significant improvements in both
speed and memory allocation.

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2024-07-06 09:08:09 +08:00
Ted Xu 78885a44c4
fix: turn on compression on stream writers (#34067) 
See #31679

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-06-24 10:08:02 +08:00
wayblink 380d3f4469
fix: Fix memory buffer error & some renaming (#33850) 
#30633

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-21 17:30:01 +08:00
Ted Xu 6d5747cb3e
feat: adding deltalog stream reader and writer (#33844) 
See #31679

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-06-19 14:42:01 +08:00
Ted Xu 066c8ea175
feat: stream reader/writer to support nulls (#33080) 
See: #31728

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-05-27 16:27:42 +08:00
congqixia 970bf18a49
fix: Allocate new slice for each batch in streaming reader (#33359) 
Related to #33268

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-24 18:07:41 +08:00
Ted Xu a8bd9bea39
fix: adding blob memory size in binlog serde (#33324) 
See: #33280

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-05-24 10:33:40 +08:00
Ted Xu a9c7ce72b8
enhance: enable stream writer in compactions (#32612) 
See #31679

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-05-17 15:05:37 +08:00
Ted Xu dc5ea6f17c
feat: adding binlog streaming writer (#31537) 
See #31679

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-04-11 10:33:20 +08:00
Buqian Zheng d7dbc3c9d8
fix: [sparse float vector] support the new streaming deserialize reader (#31325) 
issue: https://github.com/milvus-io/milvus/issues/31324

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-03-17 13:59:04 +08:00
Ted Xu 987d9023a5
enhance: Enable binlog deserialize reader in datanode compaction (#31036) 
See #30863

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-03-08 18:25:02 +08:00
Ted Xu 71adafa933
enhance: adding a streaming deserialize reader for binlogs (#30860) 
See #30863

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-03-04 19:31:09 +08:00