milvus

Commit Graph

Author	SHA1	Message	Date
shaoting-huang	88b373b024	enhance: binlog primary key turn off dict encoding (#34358 ) issue: #34357 Go Parquet uses dictionary encoding by default, and it will fall back to plain encoding if the dictionary size exceeds the dictionary size page limit. Users can specify custom fallback encoding by using `parquet.WithEncoding(ENCODING_METHOD)` in writer properties. However, Go Parquet [fallbacks to plain encoding](`e65c1e295d/go/parquet/file/column_writer_types.gen.go.tmpl (L238)`) rather than custom encoding method users provide. Therefore, this patch only turns off dictionary encoding for the primary key. With a 5 million auto ID primary key benchmark, the parquet file size improves from 13.93 MB to 8.36 MB when dictionary encoding is turned off, reducing primary key storage space by 40%. Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>	2024-07-17 17:47:44 +08:00
smellthemoon	2a1356985d	enhance: support null in go payload (#32296 ) #31728 --------- Signed-off-by: lixinguo <xinguo.li@zilliz.com> Co-authored-by: lixinguo <xinguo.li@zilliz.com>	2024-06-19 17:08:00 +08:00
SimFG	26f06dd732	Format the code (#27275 ) Signed-off-by: SimFG <bang.fu@zilliz.com>	2023-09-21 09:45:27 +08:00
congqixia	41af0a98fa	Use go-api/v2 for milvus-proto (#24770 ) Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2023-06-09 01:28:37 +08:00
yah01	ebd0279d3f	Check error by Error() and NoError() for better report message (#24736 ) Signed-off-by: yah01 <yang.cen@zilliz.com>	2023-06-08 15:36:36 +08:00
SimFG	a55f739608	Separate public proto files (#19782 ) Signed-off-by: SimFG <bang.fu@zilliz.com> Signed-off-by: SimFG <bang.fu@zilliz.com>	2022-10-16 20:49:27 +08:00
SimFG	d7f38a803d	Separate some proto files (#19218 ) Signed-off-by: SimFG <bang.fu@zilliz.com> Signed-off-by: SimFG <bang.fu@zilliz.com>	2022-09-16 16:56:49 +08:00
Xiaofan	801eeffbcc	Replace cgo parquet reader to go parquet reader (#16199 ) Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>	2022-03-30 15:21:28 +08:00
XuanYang-cn	6eb0de1640	[skip e2e]Update license for storage writer (#14040 ) Signed-off-by: yangxuan <xuan.yang@zilliz.com>	2021-12-23 12:03:28 +08:00
godchen	febdda90f4	Change binlog writer close behavior (#13046 ) Signed-off-by: godchen <qingxiang.chen@zilliz.com>	2021-12-09 12:37:06 +08:00
Xiangyu Wang	95ac530f5c	[skip ci]Improve import statements (#11715 ) Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>	2021-11-12 11:14:44 +08:00
dragondriver	abb8c2b9ed	Fix the way that binlog store the original size (#9681 ) Signed-off-by: dragondriver <jiquan.long@zilliz.com>	2021-10-11 21:02:37 +08:00
dragondriver	9a7a060484	Store original memory size of binlog file to extra information (#9628 ) Signed-off-by: dragondriver <jiquan.long@zilliz.com>	2021-10-11 17:28:30 +08:00
groot	314eed2172	Update storage unittest (#7637 ) Signed-off-by: yhmo <yihua.mo@zilliz.com>	2021-09-09 17:12:55 +08:00
Cai Yudong	3387b07dfd	Optimize code under storage (#6335 ) * rename AddOneStringToPayload/GetOneStringFromPayload to AddStringToPayload/GetStringFromPayload Signed-off-by: yudong.cai <yudong.cai@zilliz.com> * code optimize Signed-off-by: yudong.cai <yudong.cai@zilliz.com> * rename print_binglog_test to print_binlog_test Signed-off-by: yudong.cai <yudong.cai@zilliz.com> * update chap08_binlog.md Signed-off-by: yudong.cai <yudong.cai@zilliz.com> * fix unittest Signed-off-by: yudong.cai <yudong.cai@zilliz.com> * use SetEventTimestamp() to replace SetStartTimestamp() and SetEndTimestamp() Signed-off-by: yudong.cai <yudong.cai@zilliz.com> * code optimize Signed-off-by: yudong.cai <yudong.cai@zilliz.com> * rename AddStringToPayload/GetStringFromPayload to AddOneStringToPayload/GetOneStringFromPayload Signed-off-by: yudong.cai <yudong.cai@zilliz.com>	2021-07-07 19:10:07 +08:00
Xiangyu Wang	82ccd4cec0	Rename module (#4988 ) * Rename module Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>	2021-04-22 14:45:57 +08:00
godchen	0dfcb90881	Add storage copyright Signed-off-by: godchen <qingxiang.chen@zilliz.com>	2021-04-19 11:32:24 +08:00
godchen	a5ad70a5ab	Add unittest for storage Signed-off-by: godchen <qingxiang.chen@zilliz.com>	2021-04-19 10:36:19 +08:00
godchen	8204546bcd	Add unit test for binlog module Signed-off-by: godchen <qingxiang.chen@zilliz.com>	2021-04-12 14:07:46 +08:00
godchen	f3649f0419	Refactor interface and proto Signed-off-by: godchen <qingxiang.chen@zilliz.com>	2021-03-12 14:22:09 +08:00
cai.zhang	54f2b79f1b	Add indexbuilder client Signed-off-by: cai.zhang <cai.zhang@zilliz.com>	2020-12-12 17:10:42 +08:00
bigsheeper	585d3f9831	Refactor param table, and add ddNode Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2020-12-11 17:20:14 +08:00
neza2017	8df05714a8	Add binlog unittest Signed-off-by: neza2017 <yefu.chen@zilliz.com>	2020-12-11 12:01:38 +08:00
cai.zhang	62e9002348	Add git-hooks for check code Signed-off-by: cai.zhang <cai.zhang@zilliz.com>	2020-12-11 11:29:07 +08:00
neza2017	9585819154	Add binlog unittest Signed-off-by: neza2017 <yefu.chen@zilliz.com>	2020-12-10 15:50:09 +08:00
neza2017	de12fa5a10	Add binlog event Signed-off-by: neza2017 <yefu.chen@zilliz.com>	2020-12-09 11:18:13 +08:00
sunby	ed54d3e26c	Fix binlog reader bug Signed-off-by: sunby <bingyi.sun@zilliz.com>	2020-12-09 09:55:56 +08:00
quicksilver	d09ad77fb2	Add archiving file workflow for cache thirdparty Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>	2020-12-08 19:53:22 +08:00
sunby	8ace1c3837	Add binlog serialization and deserialization Signed-off-by: sunby <bingyi.sun@zilliz.com>	2020-12-08 19:13:39 +08:00

29 Commits (ecdbc12678d715c039433ba24f8a20dc6457787b)