milvus

Commit Graph

Author	SHA1	Message	Date
shaoting-huang	88b373b024	enhance: binlog primary key turn off dict encoding (#34358 ) issue: #34357 Go Parquet uses dictionary encoding by default, and it will fall back to plain encoding if the dictionary size exceeds the dictionary size page limit. Users can specify custom fallback encoding by using `parquet.WithEncoding(ENCODING_METHOD)` in writer properties. However, Go Parquet [fallbacks to plain encoding](`e65c1e295d/go/parquet/file/column_writer_types.gen.go.tmpl (L238)`) rather than custom encoding method users provide. Therefore, this patch only turns off dictionary encoding for the primary key. With a 5 million auto ID primary key benchmark, the parquet file size improves from 13.93 MB to 8.36 MB when dictionary encoding is turned off, reducing primary key storage space by 40%. Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>	2024-07-17 17:47:44 +08:00
smellthemoon	2a1356985d	enhance: support null in go payload (#32296 ) #31728 --------- Signed-off-by: lixinguo <xinguo.li@zilliz.com> Co-authored-by: lixinguo <xinguo.li@zilliz.com>	2024-06-19 17:08:00 +08:00
SimFG	26f06dd732	Format the code (#27275 ) Signed-off-by: SimFG <bang.fu@zilliz.com>	2023-09-21 09:45:27 +08:00
congqixia	41af0a98fa	Use go-api/v2 for milvus-proto (#24770 ) Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2023-06-09 01:28:37 +08:00
yah01	ebd0279d3f	Check error by Error() and NoError() for better report message (#24736 ) Signed-off-by: yah01 <yang.cen@zilliz.com>	2023-06-08 15:36:36 +08:00
jaime	c9d0c157ec	Move some modules from internal to public package (#22572 ) Signed-off-by: jaime <yun.zhang@zilliz.com>	2023-04-06 19:14:32 +08:00
SimFG	a55f739608	Separate public proto files (#19782 ) Signed-off-by: SimFG <bang.fu@zilliz.com> Signed-off-by: SimFG <bang.fu@zilliz.com>	2022-10-16 20:49:27 +08:00
SimFG	d7f38a803d	Separate some proto files (#19218 ) Signed-off-by: SimFG <bang.fu@zilliz.com> Signed-off-by: SimFG <bang.fu@zilliz.com>	2022-09-16 16:56:49 +08:00
XuanYang-cn	e580bb9ccb	[skip e2e]Update license for storage event writer (#14043 ) Signed-off-by: yangxuan <xuan.yang@zilliz.com>	2021-12-23 12:07:10 +08:00
godchen	febdda90f4	Change binlog writer close behavior (#13046 ) Signed-off-by: godchen <qingxiang.chen@zilliz.com>	2021-12-09 12:37:06 +08:00
godchen	a0a3a889e7	Add common endian for global use (#11092 ) Signed-off-by: godchen <qingxiang.chen@zilliz.com>	2021-11-02 18:16:32 +08:00
dragondriver	f85271cf3f	Estimate memory size by descriptor event (#9688 ) Signed-off-by: dragondriver <jiquan.long@zilliz.com>	2021-10-12 17:00:34 +08:00
godchen	10c07a3041	Add description event extras. (#8264 ) Signed-off-by: godchen <qingxiang.chen@zilliz.com>	2021-09-23 17:23:54 +08:00
groot	314eed2172	Update storage unittest (#7637 ) Signed-off-by: yhmo <yihua.mo@zilliz.com>	2021-09-09 17:12:55 +08:00
Cai Yudong	3387b07dfd	Optimize code under storage (#6335 ) * rename AddOneStringToPayload/GetOneStringFromPayload to AddStringToPayload/GetStringFromPayload Signed-off-by: yudong.cai <yudong.cai@zilliz.com> * code optimize Signed-off-by: yudong.cai <yudong.cai@zilliz.com> * rename print_binglog_test to print_binlog_test Signed-off-by: yudong.cai <yudong.cai@zilliz.com> * update chap08_binlog.md Signed-off-by: yudong.cai <yudong.cai@zilliz.com> * fix unittest Signed-off-by: yudong.cai <yudong.cai@zilliz.com> * use SetEventTimestamp() to replace SetStartTimestamp() and SetEndTimestamp() Signed-off-by: yudong.cai <yudong.cai@zilliz.com> * code optimize Signed-off-by: yudong.cai <yudong.cai@zilliz.com> * rename AddStringToPayload/GetStringFromPayload to AddOneStringToPayload/GetOneStringFromPayload Signed-off-by: yudong.cai <yudong.cai@zilliz.com>	2021-07-07 19:10:07 +08:00
Xiangyu Wang	82ccd4cec0	Rename module (#4988 ) * Rename module Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>	2021-04-22 14:45:57 +08:00
godchen	0dfcb90881	Add storage copyright Signed-off-by: godchen <qingxiang.chen@zilliz.com>	2021-04-19 11:32:24 +08:00
godchen	f3649f0419	Refactor interface and proto Signed-off-by: godchen <qingxiang.chen@zilliz.com>	2021-03-12 14:22:09 +08:00
cai.zhang	62e9002348	Add git-hooks for check code Signed-off-by: cai.zhang <cai.zhang@zilliz.com>	2020-12-11 11:29:07 +08:00
neza2017	9585819154	Add binlog unittest Signed-off-by: neza2017 <yefu.chen@zilliz.com>	2020-12-10 15:50:09 +08:00
neza2017	5512ce8d8a	Add binlog unittest Signed-off-by: neza2017 <yefu.chen@zilliz.com>	2020-12-10 09:31:29 +08:00
neza2017	de12fa5a10	Add binlog event Signed-off-by: neza2017 <yefu.chen@zilliz.com>	2020-12-09 11:18:13 +08:00
sunby	ed54d3e26c	Fix binlog reader bug Signed-off-by: sunby <bingyi.sun@zilliz.com>	2020-12-09 09:55:56 +08:00
quicksilver	d09ad77fb2	Add archiving file workflow for cache thirdparty Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>	2020-12-08 19:53:22 +08:00
sunby	8ace1c3837	Add binlog serialization and deserialization Signed-off-by: sunby <bingyi.sun@zilliz.com>	2020-12-08 19:13:39 +08:00

25 Commits (b48f531e63041de8c9400ae82e35c2a237124985)