Some data shapes would cause files to grow larger than the max size more
quickly which resulted in them getting skipped by the full compaction planner
at times. Some datasets that could make this happen are very large keys or
very large numbers of keys (10M). When this happened, multiple max sized
files would accumulate but the blocks would not be full. When the shard went
cold for writes, these files would get recompacted down to the optimal size, but
a lot of space would be wasted in the mean time.
- Re-enabling support for ARM builds in `build.py`.
- Improved `build.py`'s iteration and package version.
- Fixed minor bug with logrotate file path in `build.py`.
- Improved installation pre and post install/uninstall scripts by adding distribution-specific logic to account for quirks between the different package managers.
- Added post-install as post-trans script for RPM builds due to the order of RPM upgrade commands (issue where init scripts were removed when upgrading).
- Added explicit backup to pre-install script to ensure legacy configuration survive an upgrade.
byte.Replace will return a copy of the input even when nothing has
been replaced. This is called in lower level query execution and
create some garbage that isn't necessary.
This is contributing to some of the high memory usage on queries and possibly
some OOMs. This is slightly slower, but removing it allows some fairly large
count queries over 5M series to complete instead of crashing the process using
tsm1 engine.
Key() returned the key and the entries. We did not always need the
entries so they would be allocated and ignored. Added a KeyAt func
that just returns the key to avoid the unnecesary entries allocation.
Use sync.Pool for some temporary buffers used while encoding instead of
allocatin new ones each time. Also increased the default buffer size which
might be too small. Probably need to make this a config var.
We were buffering up the data to write into byte slices to reduce
IO calls but at larger sizes, this causes memory to spike. The TSMWriter
was switched to use a bufio.Writer internally so this byte slice buffering
is unnecessary and costly now.
Philip O'Toole
Ross McDonald
Jason Wilder
Philip O'Toole
Edd Robinson
Jason Wilder
Paulo Pires
Ross McDonald
Mark Rushakoff