milvus/internal/util/importutilv2
yihao.dai ad950368fe
enhance: Fix parquet import OOM (#43756)
Each ColumnReader consumes ReaderProperties.BufferSize memory
independently. Therefore, the bufferSize should be divided by the number
of columns to ensure total memory usage stays within the intended limit.

issue: https://github.com/milvus-io/milvus/issues/43755

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-08-08 18:57:40 +08:00
..
binlog enhance: Add timestamp filtering support to L0Reader (#43747) 2025-08-06 16:49:39 +08:00
common enhance: bulkinsert handle nullable/defaultValue/functionOutput fields (#42956) 2025-07-04 14:20:44 +08:00
csv enhance: bulkinsert handle nullable/defaultValue/functionOutput fields (#42956) 2025-07-04 14:20:44 +08:00
json enhance: bulkinsert handle nullable/defaultValue/functionOutput fields (#42956) 2025-07-04 14:20:44 +08:00
numpy enhance: bulkinsert handle nullable/defaultValue/functionOutput fields (#42956) 2025-07-04 14:20:44 +08:00
parquet enhance: Fix parquet import OOM (#43756) 2025-08-08 18:57:40 +08:00
mock_reader.go enhance: Support retriving file size from importutilv2.Reader (#31533) 2024-03-25 20:29:07 +08:00
option.go fix: Fix storagev2 binlog import (#43221) 2025-07-13 22:52:49 +08:00
option_test.go fix: Fix storagev2 binlog import (#43221) 2025-07-13 22:52:49 +08:00
reader.go fix: Adjust binlog and parquet reader buffer size for import (#43495) 2025-07-23 21:28:54 +08:00
reader_test.go enhance: [StorageV2] Refine storage rw option usage & validation (#43175) 2025-07-11 01:14:48 +08:00
util.go enhance: Use v2 package name for pkg module (#39990) 2025-02-22 23:15:58 +08:00