For aggregate queries, having a null result means that you haven't
got any data for that time period. CQs used this as a signal that
the measurement was not created and dropped the entire write.
INTO queries can have any structure, including wildcards, so dropping
the entire query isn't going to work. Instead, just drop the nulls
returned.
This will help large integer counters type fields that increment by
small amounts over time. Instead of storing the larger raw value
in a compressed format, we store the difference from the prior value
in compressed format which allows the value to be stored using
fewer bits.
Since INTO queries need to have absolute information about the database
to work, we need to create a loopback interface back to the cluster
in order to perform them.
influx_inpsect uncovered some scenarios where timestamps could be stored using
run-length encoding but were being stored using simple8 which uses more space.
If DecodeSameTypeBlock is called on on an empty Values slice, it would
panic with an index out of bounds error. This func can actually be removed
because DecodeBlock can determine what type of values are encoded already.
This will still panic if the block cannot be decoded due to other reasons.
Fixes#4365
If similar float values were encoded, the number of leading bits would
overflow the 5 available bits to store them (e.g. store 33 in 5 bits). When
decoding, the values after the overflowed value would spike to very large and
small values.
To prevent the overflow, we clamp the value to 31 which is the maximum
number of leading zero bits we can encoded.
Fixes#4357
Closing the store did not properly return an error for in-flight
writes because the closing channel was set to nil when closed. A
nil channel is not selectable so writes continue on past the guard
checks and trigger panics.
Will make it less error-prone to add new encodings int the future
since each encoder has it's set of constants. There are some placeholder
contants for uncompressed encodings which are not in all encoder currently.
* Fix bug with locking when the interval completely covers or is totally inside another one.
* Fix bug with full compactions running when the index is actively being written to.
If reading into fixed sized buffer using io.ReadFull, the func can
return io.ErrUnexpectedEOF if the read was short. This was slipping
through the error handling causing the shard to fail to load.
The defer tx.Rollback() tries to free the queryLock but the defer e.Cleanup() runs
before it and tries to take a write lock on the query lock (which blocks) and prevents
tx.Rollback() from acquring the read lock.
Previously were using a frame of reference approach where we would
transform the (possibly negative) deltas into positive values from
the minimum. That required an extra pass over the values as well
as a large slice allocation so we could encode the originals in uncompressed
form if they were too large.
This switches the encoding to use zigzag encoding for the deltas which
removes the extra slice allocation as well as the extra loops.
Improves encoding performane by ~4x.
This is using zig zag encoding to convert int64 to uint64s and then using simple8b
to compress them, falling back to uncompressed if the value exceeds 1 << 60. A
patched encoding scheme would likely be better in general but this provides decent
compression for integers that are not at the ends of the int64 range.
Time compression uses an adaptive approach using delta-encoding,
frame-of-reference, run length encoding as well as compressed integer
encoding.
Float compression uses an implementation of the Gorilla paper encoding
for timestamps based on XOR deltas and leading and trailing null suppression.
If an aggregate derivative query did not have a value in the first
time bucket, it would abort early and return a single row with value
of 0. Similarly, if either the current or previous value was nil,
it would skip the row and not append any values causing gaps and
no data to show up.
Instead, this will append a nil value if either the current or previous
valis is nil. This essentially allows nil values to carry through the
results as well as gives a more sensible value for rows where we cannot
compute a difference (instead of dropping the row as before).
Fixes#4237#4263
This commit changes `tsdb.mapFunc` to use `tsdb.MapInput` instead
of an iterator. This will make it easier and faster to pass blocks
of values from the new storage engine into the engine.
Jason Wilder
Jason Wilder
Nathaniel Cook
Daniel Morsing
Sean Beckett
Ben Johnson
Daniel Morsing
Philip O'Toole
Cory LaNou
Paul Dix
David Norton
Nathaniel Cook
Jason Wilder