01b5b9268e
Fix compaction and multi-write bugs.
...
* Fix bug with locking when the interval completely covers or is totally inside another one.
* Fix bug with full compactions running when the index is actively being written to.
2015-09-30 16:39:27 -04:00
c1accf7e59
Handle partial reads when loading WAL
...
If reading into fixed sized buffer using io.ReadFull, the func can
return io.ErrUnexpectedEOF if the read was short. This was slipping
through the error handling causing the shard to fail to load.
2015-09-30 09:51:05 -06:00
9b84a20dec
Fix bugs with writing old data and compaction.
2015-09-30 11:48:26 -04:00
c27de6fbb8
Add test with duplicate timestamps
...
Should not happen but makes sure that the same values are encoded
and decoded correctly.
2015-09-29 20:02:22 -06:00
f29c4c8cf9
Add compressed string encoding
...
Uses snappy to compress multiple strings into a block
2015-09-29 20:02:22 -06:00
4937ae8fbf
Fix panic when data file has small index
2015-09-29 21:51:16 -04:00
6c2bef6073
Add logging to pd1
2015-09-29 20:03:37 -04:00
72fa3dd5a4
Update WAL to deduplicate values on Cursor query.
...
Added test and have failing section for single value encoding.
2015-09-29 16:17:58 -04:00
c0eba04d9d
Rename float encoding tests
2015-09-29 10:47:35 -06:00
24922181c6
Add documentation about compression
2015-09-29 10:46:19 -06:00
3ea1b5e795
Fix deadlock in pd1_test.go
...
The defer tx.Rollback() tries to free the queryLock but the defer e.Cleanup() runs
before it and tries to take a write lock on the query lock (which blocks) and prevents
tx.Rollback() from acquring the read lock.
2015-09-29 10:44:24 -06:00
3cbd4b198f
Update simple8b api usage
2015-09-29 09:39:01 -06:00
6c0f53dfe4
Add a compressed boolean encoding
...
Packs booleans into bytes using 1 bit per value.
2015-09-29 09:39:01 -06:00
7fe9673bbc
Keep track of the type of the block encoded
...
Allowes decode to decode an arbitrary block correctly.
2015-09-29 09:39:00 -06:00
8d2ecb5df5
Remove unnecessary allocations from int64 decoder
...
The decoder was creating a large slice and decoding all values when
instead, it could decode one packed value as needed.
2015-09-29 09:39:00 -06:00
5a49e1a04b
Add test assertions for time encoding type
2015-09-29 09:39:00 -06:00
071739b960
Fix run length encoding check
...
Values were run length encoded even when they should not have been
2015-09-29 09:39:00 -06:00
8ce3d7564d
Reduce memory allocations
...
Converting between different encoders is wasting a lot of memory allocating different
typed slices.
2015-09-29 09:39:00 -06:00
fce01a2466
Use zigzag encoding for timestamp deltas
...
Previously were using a frame of reference approach where we would
transform the (possibly negative) deltas into positive values from
the minimum. That required an extra pass over the values as well
as a large slice allocation so we could encode the originals in uncompressed
form if they were too large.
This switches the encoding to use zigzag encoding for the deltas which
removes the extra slice allocation as well as the extra loops.
Improves encoding performane by ~4x.
2015-09-29 09:39:00 -06:00
bf74c7cf86
Add int64 compression
...
This is using zig zag encoding to convert int64 to uint64s and then using simple8b
to compress them, falling back to uncompressed if the value exceeds 1 << 60. A
patched encoding scheme would likely be better in general but this provides decent
compression for integers that are not at the ends of the int64 range.
2015-09-29 09:39:00 -06:00
19877a6d85
Add time and float compression
...
Time compression uses an adaptive approach using delta-encoding,
frame-of-reference, run length encoding as well as compressed integer
encoding.
Float compression uses an implementation of the Gorilla paper encoding
for timestamps based on XOR deltas and leading and trailing null suppression.
2015-09-29 09:39:00 -06:00
57b0a276da
Allow influx_stress to use small data sizes
...
Prevents a panics when response size is less than 100. Also allows
data to be posted when it is less than the batch size.
2015-09-29 09:39:00 -06:00
938bae97d4
Fix go vet errors
2015-09-29 09:39:00 -06:00
4fcc61c766
Ensure we have files when iterating in cursor
...
Prevents index out of bounds panic
2015-09-29 09:39:00 -06:00
ca2a13c76e
Update stress to use second timestamps and less random floats.
2015-09-29 11:37:28 -04:00
9031804dcc
Update tests to use transactions. Add test for single series 10k points.
2015-09-29 11:27:02 -04:00
eb24c0daa6
Add recover to maintenance. Change snapshot writer to not use bolt on shard.
2015-09-29 10:56:11 -04:00
9e630f8b8b
Ensure data files can't be deleted while query is running.
...
Also ensure that queries don't try to use files that have been deleted.
2015-09-29 08:59:25 -04:00
68d03e8ae0
Make compaction run at most at set duration.
2015-09-29 08:35:05 -04:00
8444e0546d
Add PerformMaintenance to store for flushes and compactions.
...
Also fixed shard to work again with b1 and bz1 engines.
2015-09-28 22:50:26 -04:00
f1ba618b23
Fix wal flushing, compacting, and write lock
2015-09-28 21:04:02 -04:00
17ed6932ae
Add support for multiple fields
2015-09-27 19:37:03 -04:00
a103432598
Handle hash collisions on keys
2015-09-27 16:44:57 -04:00
4db2e5c2d4
Add compaction and time range based write locks.
2015-09-27 14:45:55 -04:00
3aba709440
Add multicursor to combine wal and index
2015-09-26 10:47:36 -04:00
33d28e18ec
Ensure we don't have duplicate values. Fix panic in compaction.
2015-09-26 09:02:25 -04:00
0aa684728c
Make writes to historical areas possible
2015-09-25 17:11:27 -04:00
ed7055146a
Update to work with new cursor definitiono and Point in models
2015-09-25 10:49:26 -04:00
ea85f8042d
Update wal to only open new segment file on flush if its not an idle flush
2015-09-24 15:42:54 -07:00
1b57b80fdb
Add test for close and restart of engine and fix errors.
2015-09-24 15:42:54 -07:00
38f9b29925
Update engine to put index at the end of data files
2015-09-24 15:42:54 -07:00
750856836e
Add memory settings and WAL backpressure
2015-09-24 15:42:54 -07:00
95f9e1619e
Update encoding test to work with new interface.
2015-09-24 15:38:33 -07:00
318bc7281d
Add full durability to WAL and flush on startup
2015-09-24 15:38:33 -07:00
f37df11339
WIP: more WAL work
2015-09-24 15:38:33 -07:00
4e2dfd77c3
WIP: finish basics of PD1. IT WORKS! (kind of)
2015-09-24 15:38:07 -07:00
041e7bead3
WIP: engine work
2015-09-24 15:36:21 -07:00
8d16c96060
Add comment about encoding float
2015-09-24 15:36:21 -07:00
3446c6024d
WIP: skeleton for encoding for new engine
2015-09-24 15:36:21 -07:00
d02817ca6d
Merge pull request #4166 from influxdb/correct_show_error
...
Update, and ease maintainabilty of, SHOW errors
2015-09-24 13:24:00 -07:00
Paul Dix
Jason Wilder
Jason Wilder
Philip O'Toole