milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
milvus/internal/storage
History
shaoting-huang 88b373b024
enhance: binlog primary key turn off dict encoding (#34358) 
issue: #34357 

Go Parquet uses dictionary encoding by default, and it will fall back to
plain encoding if the dictionary size exceeds the dictionary size page
limit. Users can specify custom fallback encoding by using
`parquet.WithEncoding(ENCODING_METHOD)` in writer properties. However,
Go Parquet [fallbacks to plain
encoding](e65c1e295d/go/parquet/file/column_writer_types.gen.go.tmpl (L238))
rather than custom encoding method users provide. Therefore, this patch
only turns off dictionary encoding for the primary key.

With a 5 million auto ID primary key benchmark, the parquet file size
improves from 13.93 MB to 8.36 MB when dictionary encoding is turned
off, reducing primary key storage space by 40%.

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 		2024-07-17 17:47:44 +08:00
..
aliyun Identify service providers based on addresses (#27907) 2023-10-25 17:28:10 +08:00
gcp Format the code (#27275) 2023-09-21 09:45:27 +08:00
tencent feat: Support tencent cloud object storage for milvus (#30163) 2024-01-23 11:28:56 +08:00
OWNERS [skip ci]Update OWNERS files (#11898) 2021-11-16 15:41:11 +08:00
azure_object_storage.go enhance: Add nilness linter and fix some small issues (#34049) 2024-06-24 14:52:03 +08:00
azure_object_storage_test.go enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740) 2024-04-25 20:41:27 +08:00
binlog_iterator.go enhance: legacy code clean up (#33838) 2024-06-14 14:25:56 +08:00
binlog_iterator_test.go enhance: binlog primary key turn off dict encoding (#34358) 2024-07-17 17:47:44 +08:00
binlog_reader.go fix: descriptor event in previous version not has nullable to parse error (#34235) 2024-07-01 16:38:06 +08:00
binlog_test.go enhance: binlog primary key turn off dict encoding (#34358) 2024-07-17 17:47:44 +08:00
binlog_util.go Move some modules from internal to public package (#22572) 2023-04-06 19:14:32 +08:00
binlog_util_test.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
binlog_writer.go enhance: binlog primary key turn off dict encoding (#34358) 2024-07-17 17:47:44 +08:00
binlog_writer_test.go enhance: binlog primary key turn off dict encoding (#34358) 2024-07-17 17:47:44 +08:00
data_codec.go enhance: binlog primary key turn off dict encoding (#34358) 2024-07-17 17:47:44 +08:00
data_codec_test.go enhance: binlog primary key turn off dict encoding (#34358) 2024-07-17 17:47:44 +08:00
data_sorter.go feat: [Sparse Float Vector] add sparse vector support to milvus components (#30630) 2024-03-13 14:32:54 -07:00
data_sorter_test.go enhance: add helpers to parse sparse float vector in JSON (#32543) 2024-04-25 14:47:24 +08:00
delta_data.go enhance: Unify DeleteLog parsing code (#34009) 2024-06-21 16:54:01 +08:00
delta_data_test.go enhance: Add unittest for `storage.DeleteLog` (#34190) 2024-06-26 17:14:04 +08:00
event_data.go enhance: Fix lint issues from recent PRs (#34482) 2024-07-09 10:06:24 +08:00
event_header.go Move some modules from internal to public package (#22572) 2023-04-06 19:14:32 +08:00
event_reader.go enhance: support null in go payload (#32296) 2024-06-19 17:08:00 +08:00
event_test.go enhance: binlog primary key turn off dict encoding (#34358) 2024-07-17 17:47:44 +08:00
event_writer.go enhance: binlog primary key turn off dict encoding (#34358) 2024-07-17 17:47:44 +08:00
event_writer_test.go enhance: binlog primary key turn off dict encoding (#34358) 2024-07-17 17:47:44 +08:00
factory.go enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740) 2024-04-25 20:41:27 +08:00
field_stats.go feat: Major compaction (#33620) 2024-06-10 21:34:08 +08:00
field_stats_test.go feat: Major compaction (#33620) 2024-06-10 21:34:08 +08:00
field_value.go enhance: reconstruct scalar part's code for segment-pruner(#30376) (#34346) 2024-07-04 16:36:09 +08:00
field_value_test.go feat: Define FieldValue, FieldStats and PartitionStats (#30286) 2024-03-06 20:42:37 -08:00
index_data_codec.go enhance: support null in go payload (#32296) 2024-06-19 17:08:00 +08:00
index_data_codec_test.go enhance: Add memory size for binlog (#33025) 2024-05-15 12:59:34 +08:00
insert_data.go enhance: Add lint rule to forbid gogo protobuf (#34594) 2024-07-12 10:19:35 +08:00
insert_data_test.go enhance: support null in go payload (#32296) 2024-06-19 17:08:00 +08:00
local_chunk_manager.go enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740) 2024-04-25 20:41:27 +08:00
local_chunk_manager_test.go enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740) 2024-04-25 20:41:27 +08:00
minio_object_storage.go enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740) 2024-04-25 20:41:27 +08:00
minio_object_storage_test.go fix: Removed minio bucket after use in test (#32624) 2024-04-28 13:51:26 +08:00
options.go enhance: Support MinIO TLS connection (#31311) 2024-03-21 11:15:20 +08:00
partition_stats.go fix: sync part stats task cannot be finished(#30376) (#34027) 2024-06-24 10:16:02 +08:00
partition_stats_test.go feat: Define FieldValue, FieldStats and PartitionStats (#30286) 2024-03-06 20:42:37 -08:00
payload.go enhance: support null in go payload (#32296) 2024-06-19 17:08:00 +08:00
payload_reader.go enhance: support null in go payload (#32296) 2024-06-19 17:08:00 +08:00
payload_reader_test.go enhance: support null in go payload (#32296) 2024-06-19 17:08:00 +08:00
payload_test.go enhance: binlog primary key turn off dict encoding (#34358) 2024-07-17 17:47:44 +08:00
payload_writer.go enhance: binlog primary key turn off dict encoding (#34358) 2024-07-17 17:47:44 +08:00
payload_writer_test.go enhance: binlog primary key turn off dict encoding (#34358) 2024-07-17 17:47:44 +08:00
pk_statistics.go enhance: Use BatchPkExist to reduce bloom filter func call cost (#33611) 2024-06-13 17:57:56 +08:00
primary_key.go enhance: Remove StringPrimaryKey to reduce unnecessary copy and function call cost (#33486) 2024-05-31 15:41:45 +08:00
primary_key_test.go Use go-api/v2 for milvus-proto (#24770) 2023-06-09 01:28:37 +08:00
primary_keys.go enhance: Add PrimaryKeys interface to reduce memory usage (#30405) 2024-02-01 09:57:11 +08:00
primary_keys_test.go enhance: Add PrimaryKeys interface to reduce memory usage (#30405) 2024-02-01 09:57:11 +08:00
print_binlog.go enhance: support null in go payload (#32296) 2024-06-19 17:08:00 +08:00
print_binlog_test.go enhance: binlog primary key turn off dict encoding (#34358) 2024-07-17 17:47:44 +08:00
remote_chunk_manager.go enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740) 2024-04-25 20:41:27 +08:00
remote_chunk_manager_test.go enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740) 2024-04-25 20:41:27 +08:00
serde.go enhance: binlog primary key turn off dict encoding (#34358) 2024-07-17 17:47:44 +08:00
serde_events.go enhance: binlog primary key turn off dict encoding (#34358) 2024-07-17 17:47:44 +08:00
serde_events_test.go enhance: binlog primary key turn off dict encoding (#34358) 2024-07-17 17:47:44 +08:00
serde_test.go enhance: add delta log stream new format reader and writer (#34116) 2024-07-06 09:08:09 +08:00
stats.go enhance: Use Blocked Bloom Filter instead of basic bloom fitler impl. (#33405) 2024-05-31 17:49:45 +08:00
stats_test.go enhance: Use Blocked Bloom Filter instead of basic bloom fitler impl. (#33405) 2024-05-31 17:49:45 +08:00
storage_test.go enhance: Remove vector chunk manager (#28569) 2023-11-30 18:00:33 +08:00
types.go enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740) 2024-04-25 20:41:27 +08:00
unsafe.go enhance: support null in go payload (#32296) 2024-06-19 17:08:00 +08:00
unsafe_test.go [skip e2e]Update license for storage unsafe (#14452) 2021-12-28 20:03:56 +08:00
utils.go enhance: binlog primary key turn off dict encoding (#34358) 2024-07-17 17:47:44 +08:00
utils_test.go enhance: support null in go payload (#32296) 2024-06-19 17:08:00 +08:00