14 Commits (eb046863485fdf3e130fc60484485c901b81276b)

Author	SHA1	Message	Date
sthuang	90acc8a58f	enhance: upgrade go arrow version from 12.0.1 to 17.0.0 (#39916) ... related: https://github.com/milvus-io/milvus/issues/39915 Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>	2025-02-25 10:30:02 +08:00
congqixia	cb7f2fa6fd	enhance: Use v2 package name for pkg module (#39990) ... Related to #39095 https://go.dev/doc/modules/version-numbers Update pkg version according to golang dep version convention --------- Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2025-02-22 23:15:58 +08:00
Ted Xu	2978b0890e	enhance: iterative download data during compaction to reduce memory cost (#39724) ... See #37234 --------- Signed-off-by: Ted Xu <ted.xu@zilliz.com>	2025-02-13 10:36:47 +08:00
Ted Xu	427b6a4c94	enhance: reduce stats task cost by skipping ser/de (#39568) ... See #37234 --------- Signed-off-by: Ted Xu <ted.xu@zilliz.com>	2025-02-06 17:14:45 +08:00
zhagnlu	6ee94d00b9	fix:fix calculate arrow nest type and add ut (#38527) ... #37767 Signed-off-by: luzhang <luzhang@zilliz.com> Co-authored-by: luzhang <luzhang@zilliz.com>	2024-12-18 11:54:44 +08:00
shaoting-huang	f4dd7c7efb	enhance: add delta log stream new format reader and writer (#34116) ... issue: #34123 Benchmark case: The benchmark run the go benchmark function `BenchmarkDeltalogFormat` which is put in the Files changed. It tests the performance of serializing and deserializing from two different data formats under a 10 million delete log dataset. Metrics: The benchmarks measure the average time taken per operation (ns/op), memory allocated per operation (MB/op), and the number of memory allocations per operation (allocs/op). | Test Name | Avg Time (ns/op) | Time Comparison | Memory Allocation (MB/op) | Memory Comparison | Allocation Count (allocs/op) | Allocation Comparison | |---------------------------------|------------------|-----------------|---------------------------|-------------------|------------------------------|------------------------| | one_string_format_reader | 2,781,990,000 | Baseline | 2,422 | Baseline | 20,336,539 | Baseline | | pk_ts_separate_format_reader | 480,682,639 | -82.72% | 1,765 | -27.14% | 20,396,958 | +0.30% | | one_string_format_writer | 5,483,436,041 | Baseline | 13,900 | Baseline | 70,057,473 | Baseline | | pk_and_ts_separate_format_writer| 798,591,584 | -85.43% | 2,178 | -84.34% | 30,270,488 | -56.78% | Both read and write operations show significant improvements in both speed and memory allocation. Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>	2024-07-06 09:08:09 +08:00
Ted Xu	6d5747cb3e	feat: adding deltalog stream reader and writer (#33844) ... See #31679 --------- Signed-off-by: Ted Xu <ted.xu@zilliz.com>	2024-06-19 14:42:01 +08:00
Ted Xu	066c8ea175	feat: stream reader/writer to support nulls (#33080) ... See: #31728 --------- Signed-off-by: Ted Xu <ted.xu@zilliz.com>	2024-05-27 16:27:42 +08:00
Ted Xu	a8bd9bea39	fix: adding blob memory size in binlog serde (#33324) ... See: #33280 Signed-off-by: Ted Xu <ted.xu@zilliz.com>	2024-05-24 10:33:40 +08:00
Ted Xu	a9c7ce72b8	enhance: enable stream writer in compactions (#32612) ... See #31679 --------- Signed-off-by: Ted Xu <ted.xu@zilliz.com>	2024-05-17 15:05:37 +08:00
Ted Xu	dc5ea6f17c	feat: adding binlog streaming writer (#31537) ... See #31679 --------- Signed-off-by: Ted Xu <ted.xu@zilliz.com>	2024-04-11 10:33:20 +08:00
Buqian Zheng	d7dbc3c9d8	fix: [sparse float vector] support the new streaming deserialize reader (#31325) ... issue: https://github.com/milvus-io/milvus/issues/31324 Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>	2024-03-17 13:59:04 +08:00
Ted Xu	987d9023a5	enhance: Enable binlog deserialize reader in datanode compaction (#31036) ... See #30863 Signed-off-by: Ted Xu <ted.xu@zilliz.com>	2024-03-08 18:25:02 +08:00
Ted Xu	71adafa933	enhance: adding a streaming deserialize reader for binlogs (#30860) ... See #30863 --------- Signed-off-by: Ted Xu <ted.xu@zilliz.com>	2024-03-04 19:31:09 +08:00