Will make it less error-prone to add new encodings int the future
since each encoder has it's set of constants. There are some placeholder
contants for uncompressed encodings which are not in all encoder currently.
* Fix bug with locking when the interval completely covers or is totally inside another one.
* Fix bug with full compactions running when the index is actively being written to.
If reading into fixed sized buffer using io.ReadFull, the func can
return io.ErrUnexpectedEOF if the read was short. This was slipping
through the error handling causing the shard to fail to load.
The defer tx.Rollback() tries to free the queryLock but the defer e.Cleanup() runs
before it and tries to take a write lock on the query lock (which blocks) and prevents
tx.Rollback() from acquring the read lock.
Previously were using a frame of reference approach where we would
transform the (possibly negative) deltas into positive values from
the minimum. That required an extra pass over the values as well
as a large slice allocation so we could encode the originals in uncompressed
form if they were too large.
This switches the encoding to use zigzag encoding for the deltas which
removes the extra slice allocation as well as the extra loops.
Improves encoding performane by ~4x.
This is using zig zag encoding to convert int64 to uint64s and then using simple8b
to compress them, falling back to uncompressed if the value exceeds 1 << 60. A
patched encoding scheme would likely be better in general but this provides decent
compression for integers that are not at the ends of the int64 range.
Time compression uses an adaptive approach using delta-encoding,
frame-of-reference, run length encoding as well as compressed integer
encoding.
Float compression uses an implementation of the Gorilla paper encoding
for timestamps based on XOR deltas and leading and trailing null suppression.
Paul Dix
Jason Wilder
Jason Wilder