f4d8271006
Merge pull request #4402 from influxdb/pd-tsm-file-size-limit
...
TSM1 fixes
2015-10-16 07:46:08 -04:00
ba41d6fd92
remove failing test
2015-10-16 07:25:33 -04:00
a2102e19ba
always return time, never strings. fixes #4415
2015-10-15 20:19:58 -05:00
bc2a95a877
Merge pull request #4455 from influxdb/nc-mr-signatures
...
update reduce func signatures to not have influxql.Call arguments
2015-10-15 15:16:39 -06:00
c27f8ae3a4
tsm1 meta lint
2015-10-15 15:03:10 -06:00
ae925625ce
Merge pull request #4451 from influxdb/jw-int64
...
Int64 encoding enhancements
2015-10-15 13:44:55 -06:00
3db28cfe99
Fix typos
2015-10-15 13:36:42 -06:00
d338f38d65
update reduce func signatures to not have influxql.Call arguments
2015-10-15 09:24:16 -06:00
13f6fd6238
Add measurement reference back to CQs
...
CQs has a feature where you can tell it to use the name of the originating measurement as the target name. This was missed when I reimplemented it.
2015-10-15 13:10:38 +00:00
d990b5f28d
fix into queries when encountering nil values.
...
For aggregate queries, having a null result means that you haven't
got any data for that time period. CQs used this as a signal that
the measurement was not created and dropped the entire write.
INTO queries can have any structure, including wildcards, so dropping
the entire query isn't going to work. Instead, just drop the nulls
returned.
2015-10-15 12:12:40 +00:00
c037ae1416
Add run-length encoding to int64 encoder
...
This encoding can really help counter values with large runs of the same
value (e.g. a counter that is 0 for long periods of time).
2015-10-14 19:23:58 -06:00
014c51f32a
Delta-encode int64 before bit packing
...
This will help large integer counters type fields that increment by
small amounts over time. Instead of storing the larger raw value
in a compressed format, we store the difference from the prior value
in compressed format which allows the value to be stored using
fewer bits.
2015-10-14 19:23:51 -06:00
82f104a8b1
Merge pull request #4436 from influxdb/tag-names-to-keys
...
WIP tag name --> tag key, field name --> field key
2015-10-14 16:02:46 -07:00
f2d23b070b
add tsm1 wal quickcheck
...
This commit adds quickcheck testing for the tsm1 WAL.
2015-10-14 09:38:38 -06:00
6d188d9703
Merge pull request #4409 from influxdb/intoq
...
wire up INTO queries.
2015-10-14 15:29:54 +01:00
6d38559f24
RLock access to store's shards
...
Fix issue #4442 .
2015-10-13 20:28:19 -07:00
1d83c8c427
Update meta.go
2015-10-13 16:46:59 -07:00
62dff895e2
wire up INTO queries.
...
Since INTO queries need to have absolute information about the database
to work, we need to create a loopback interface back to the cluster
in order to perform them.
2015-10-13 15:00:36 +00:00
6787525912
fixes multiple selectors overwriting each other. fixes #4360
2015-10-12 21:40:57 -05:00
16b3084ca9
Merge pull request #4397 from influxdb/jw-tsmdump
...
tsm1 file dump
2015-10-12 08:12:23 -06:00
a99c9ec5af
Ensure rewrite index doesn't write corrupt file.
...
Fixes #4401
2015-10-10 16:58:25 -04:00
629219951a
Fix timestamp encoding not using run-length encoding when possible
...
influx_inpsect uncovered some scenarios where timestamps could be stored using
run-length encoding but were being stored using simple8 which uses more space.
2015-10-09 22:38:17 -06:00
9627f5ab84
fix #4280 : add comments based on code review
2015-10-09 18:38:33 -04:00
512d6ac050
fix #4280 : only drop points matching WHERE clause
2015-10-09 18:34:32 -04:00
758359accc
Prevent panic in DecodeSameTypeBlock
...
If DecodeSameTypeBlock is called on on an empty Values slice, it would
panic with an index out of bounds error. This func can actually be removed
because DecodeBlock can determine what type of values are encoded already.
This will still panic if the block cannot be decoded due to other reasons.
Fixes #4365
2015-10-09 12:52:23 -06:00
2b3bb5336d
add tsm1 quickcheck tests
2015-10-08 11:59:57 -06:00
79185fc1dc
Fix index out of bounds panic in int64Decoder
...
Code was missing a check for when we did not have anymore bytes to decode
so it panic when we tried to decode the empty slice.
2015-10-08 11:21:19 -06:00
b3343a6d2a
Fix similar float values encoding overflow
...
If similar float values were encoded, the number of leading bits would
overflow the 5 available bits to store them (e.g. store 33 in 5 bits). When
decoding, the values after the overflowed value would spike to very large and
small values.
To prevent the overflow, we clamp the value to 31 which is the maximum
number of leading zero bits we can encoded.
Fixes #4357
2015-10-07 15:05:56 -06:00
6f1a44bd07
Merge pull request #4345 from influxdb/remove_iterator
...
tsdb.Iterator is no longer used. Removing
2015-10-06 17:00:50 -06:00
f041939a1c
Merge pull request #4308 from influxdb/pd-storage-engine
...
The TSM storage engine
2015-10-06 15:54:56 -07:00
b11308133a
Only limit field count for non-tsm engines
2015-10-06 15:49:37 -07:00
40ff4f4a86
Change default to bz1
2015-10-06 15:30:34 -07:00
41e3294d4a
Fix panic: assignment to entry in nil map
...
Closing the store did not properly return an error for in-flight
writes because the closing channel was set to nil when closed. A
nil channel is not selectable so writes continue on past the guard
checks and trigger panics.
2015-10-06 14:03:52 -06:00
be477b2aab
Fix cursor bug on index
2015-10-06 12:26:45 -07:00
d380ee37a2
tsdb.Iterator is no longer used. Removing
2015-10-06 10:34:07 -06:00
267f34b94e
Updates based on PR feedback
2015-10-05 20:09:56 -04:00
26a93ec23e
Fix deletes not kept if shutdown before flush on tsm1
2015-10-05 20:09:56 -04:00
bb398daf75
Updates based on @otoolp's PR comments
2015-10-05 20:09:56 -04:00
c6f2f9cec2
Avoid duplicating values slice when encoding
2015-10-05 20:09:56 -04:00
cb28dabf62
Make DecodeBlock panic if block size is too small
...
Should never get a block size 9 bytes since Encode always returns the min
timestampe and a 1 byte header. If we get this, the engine is confused.
2015-10-05 20:09:56 -04:00
b0449702e5
Fix comment typos
2015-10-05 20:09:56 -04:00
d9f94bdeeb
Add db crash recovery
2015-10-05 20:09:56 -04:00
1d754db00b
Propogate all encoding errors to engine
...
Avoid panicing in lower level code and allow the engine to decide what
it should do.
2015-10-05 20:09:56 -04:00
4c54c78009
Move compression encoding constants to encoders
...
Will make it less error-prone to add new encodings int the future
since each encoder has it's set of constants. There are some placeholder
contants for uncompressed encodings which are not in all encoder currently.
2015-10-05 20:09:56 -04:00
b1a57e1628
Fix go vet errors
2015-10-05 20:09:56 -04:00
ab791ba913
Fix TestStoreOpenShardCreateDelete
...
Shard path can be a directory.
2015-10-05 20:09:56 -04:00
d47ddb5454
Cleanup after pd1 -> tsm1 name change.
2015-10-05 20:09:55 -04:00
594253cbba
Rename storage engine to tsm1, for Time Structured Merge Tree!
2015-10-05 20:09:55 -04:00
0a11a2fdbc
Add deletes to new storage engine
2015-10-05 20:09:55 -04:00
4beca1a245
Implement reverse cursor direction on pd1
2015-10-05 20:09:55 -04:00
dbf6228817
Fix go vet
2015-10-05 20:09:55 -04:00
d9499f0598
Remove zig zag encoding from timestamp encoder
...
Not needed since all timestamps will be sorted in ascending order. Negatives
are not possible.
2015-10-05 20:09:55 -04:00
a2b139e006
Fix compaction and multi-write bugs.
...
* Fix bug with locking when the interval completely covers or is totally inside another one.
* Fix bug with full compactions running when the index is actively being written to.
2015-10-05 20:09:55 -04:00
2366baaf0b
Handle partial reads when loading WAL
...
If reading into fixed sized buffer using io.ReadFull, the func can
return io.ErrUnexpectedEOF if the read was short. This was slipping
through the error handling causing the shard to fail to load.
2015-10-05 20:09:55 -04:00
3332236527
Fix bugs with writing old data and compaction.
2015-10-05 20:09:55 -04:00
5d938d0a8b
Add test with duplicate timestamps
...
Should not happen but makes sure that the same values are encoded
and decoded correctly.
2015-10-05 20:09:55 -04:00
c47d14540d
Add compressed string encoding
...
Uses snappy to compress multiple strings into a block
2015-10-05 20:09:55 -04:00
861a15b3e6
Fix panic when data file has small index
2015-10-05 20:09:55 -04:00
be011b8da9
Add logging to pd1
2015-10-05 20:09:54 -04:00
c1213ba367
Update WAL to deduplicate values on Cursor query.
...
Added test and have failing section for single value encoding.
2015-10-05 20:09:54 -04:00
9f9692acdf
Rename float encoding tests
2015-10-05 20:09:54 -04:00
a4d92162ef
Add documentation about compression
2015-10-05 20:09:54 -04:00
2da52ec4fe
Fix deadlock in pd1_test.go
...
The defer tx.Rollback() tries to free the queryLock but the defer e.Cleanup() runs
before it and tries to take a write lock on the query lock (which blocks) and prevents
tx.Rollback() from acquring the read lock.
2015-10-05 20:09:54 -04:00
7e0df18e1a
Update simple8b api usage
2015-10-05 20:09:54 -04:00
cb23f5ac53
Add a compressed boolean encoding
...
Packs booleans into bytes using 1 bit per value.
2015-10-05 20:09:54 -04:00
1196587dc4
Keep track of the type of the block encoded
...
Allowes decode to decode an arbitrary block correctly.
2015-10-05 20:09:54 -04:00
731ae27123
Remove unnecessary allocations from int64 decoder
...
The decoder was creating a large slice and decoding all values when
instead, it could decode one packed value as needed.
2015-10-05 20:09:54 -04:00
95046c1e37
Add test assertions for time encoding type
2015-10-05 20:09:54 -04:00
e42d8660d0
Fix run length encoding check
...
Values were run length encoded even when they should not have been
2015-10-05 20:09:54 -04:00
092689c131
Reduce memory allocations
...
Converting between different encoders is wasting a lot of memory allocating different
typed slices.
2015-10-05 20:09:54 -04:00
ce1d45ecda
Use zigzag encoding for timestamp deltas
...
Previously were using a frame of reference approach where we would
transform the (possibly negative) deltas into positive values from
the minimum. That required an extra pass over the values as well
as a large slice allocation so we could encode the originals in uncompressed
form if they were too large.
This switches the encoding to use zigzag encoding for the deltas which
removes the extra slice allocation as well as the extra loops.
Improves encoding performane by ~4x.
2015-10-05 20:09:54 -04:00
4a37ba868d
Add int64 compression
...
This is using zig zag encoding to convert int64 to uint64s and then using simple8b
to compress them, falling back to uncompressed if the value exceeds 1 << 60. A
patched encoding scheme would likely be better in general but this provides decent
compression for integers that are not at the ends of the int64 range.
2015-10-05 20:09:53 -04:00
42e1babe7f
Add time and float compression
...
Time compression uses an adaptive approach using delta-encoding,
frame-of-reference, run length encoding as well as compressed integer
encoding.
Float compression uses an implementation of the Gorilla paper encoding
for timestamps based on XOR deltas and leading and trailing null suppression.
2015-10-05 20:09:53 -04:00
112a03f24c
Fix go vet errors
2015-10-05 20:08:58 -04:00
88248f3f81
Ensure we have files when iterating in cursor
...
Prevents index out of bounds panic
2015-10-05 20:08:58 -04:00
db4ad33f3c
Update tests to use transactions. Add test for single series 10k points.
2015-10-05 20:06:22 -04:00
0b33a71bb7
Add recover to maintenance. Change snapshot writer to not use bolt on shard.
2015-10-05 20:06:22 -04:00
0fd116d1f2
Ensure data files can't be deleted while query is running.
...
Also ensure that queries don't try to use files that have been deleted.
2015-10-05 20:06:22 -04:00
b1bdb4f15a
Make compaction run at most at set duration.
2015-10-05 20:06:22 -04:00
1c8eac1523
Add PerformMaintenance to store for flushes and compactions.
...
Also fixed shard to work again with b1 and bz1 engines.
2015-10-05 20:06:22 -04:00
d694454f47
Fix wal flushing, compacting, and write lock
2015-10-05 20:06:22 -04:00
6c94e738a0
Add support for multiple fields
2015-10-05 20:06:22 -04:00
667b3e6c08
Handle hash collisions on keys
2015-10-05 20:06:22 -04:00
48069e782c
Add compaction and time range based write locks.
2015-10-05 20:06:22 -04:00
2eb2a647d6
Add multicursor to combine wal and index
2015-10-05 20:06:22 -04:00
7baba84a21
Ensure we don't have duplicate values. Fix panic in compaction.
2015-10-05 20:06:22 -04:00
0770ccc87d
Make writes to historical areas possible
2015-10-05 20:06:21 -04:00
982c28b947
Update to work with new cursor definitiono and Point in models
2015-10-05 20:06:21 -04:00
365a631b53
Update wal to only open new segment file on flush if its not an idle flush
2015-10-05 20:06:21 -04:00
7c8ab4f1d8
Add test for close and restart of engine and fix errors.
2015-10-05 20:06:21 -04:00
c5f6c57d7f
Update engine to put index at the end of data files
2015-10-05 20:06:21 -04:00
fe1f9a51e5
Add memory settings and WAL backpressure
2015-10-05 20:06:21 -04:00
5e59cb9393
Update encoding test to work with new interface.
2015-10-05 20:06:21 -04:00
2100e66437
Add full durability to WAL and flush on startup
2015-10-05 20:06:21 -04:00
82e1be7527
WIP: more WAL work
2015-10-05 20:06:21 -04:00
2ba032b7a8
WIP: finish basics of PD1. IT WORKS! (kind of)
2015-10-05 20:06:21 -04:00
7555ccbd70
WIP: engine work
2015-10-05 20:06:21 -04:00
12ea1cb26f
Add comment about encoding float
2015-10-05 20:06:21 -04:00
fb2a1cb2f3
WIP: skeleton for encoding for new engine
2015-10-05 20:06:20 -04:00
4375545064
fix #4276 : walk DropSeriesStatement
2015-10-05 19:56:30 -04:00
2ac0357406
Support dropping non-Raft nodes
2015-10-04 00:19:52 -07:00
d74e0690c7
Revert "Merge pull request #4233 from influxdb/drop-server"
...
This reverts commit 0bdb36f6dc3085fbc138
2015-10-02 08:39:57 -07:00
f50813460e
protobuf update.. :-(
2015-10-01 15:39:15 -05:00
06c143c2dd
Handle missing values in aggregate derivative queries better
...
If an aggregate derivative query did not have a value in the first
time bucket, it would abort early and return a single row with value
of 0. Similarly, if either the current or previous value was nil,
it would skip the row and not append any values causing gaps and
no data to show up.
Instead, this will append a nil value if either the current or previous
valis is nil. This essentially allows nil values to carry through the
results as well as gives a more sensible value for rows where we cannot
compute a difference (instead of dropping the row as before).
Fixes #4237 #4263
2015-10-01 13:05:19 -06:00
343dd23ee7
refactor map functions to use list of values
...
This commit changes `tsdb.mapFunc` to use `tsdb.MapInput` instead
of an iterator. This will make it easier and faster to pass blocks
of values from the new storage engine into the engine.
2015-09-29 14:00:33 -06:00
842b36f30f
godoc
2015-09-23 15:52:44 -05:00
715e3d8d3c
add comment to clarify
2015-09-23 15:49:17 -05:00
d6ee542083
SelectFilterFields -> SelectWhereFields
2015-09-23 15:47:12 -05:00
b2474c9abf
reduce arguments to MapTopBottom
2015-09-23 15:44:36 -05:00
104c049a53
fix typo
2015-09-23 15:26:13 -05:00
81ad1f87a4
refactor MapTopBottom signature
2015-09-23 10:53:21 -05:00
7c3a542e13
revert signatures for mean/sum map funcs
2015-09-23 10:32:00 -05:00
6b19e3dd9c
minor refactorings
2015-09-23 09:47:24 -05:00
b1becfbcfc
more tests, simplify mapFunc signatures
2015-09-23 09:27:19 -05:00
78bc740434
full support for min/max/first/last
2015-09-23 09:03:57 -05:00
8e27cf1fd8
Merge remote-tracking branch 'upstream/master' into refactor-select-mapper
...
Conflicts:
tsdb/store.go
2015-09-22 13:58:24 -06:00
410eb4ece4
Merge pull request #3852 from influxdb/dmq-show-tag-keys
...
convert SHOW TAG KEYS to distributed query
2015-09-22 15:28:08 -04:00
96715d7d90
rename Cursor.Seek() to Cursor.SeekTo()
2015-09-22 13:23:16 -06:00
56cb2fae5d
fix integration tests
2015-09-22 13:10:13 -06:00
649663ca15
fix tests
2015-09-22 13:10:13 -06:00
b213ddad78
refactor cursor
2015-09-22 13:10:12 -06:00
a5269e9cc7
rename direction to ascending.
2015-09-22 13:09:26 -06:00
1b8b625787
refactor SelectMapper
2015-09-22 13:09:26 -06:00
0883182798
simplify select mapper stmt
2015-09-22 12:27:14 -06:00
79e6e3e07a
Merge pull request #4196 from influxdb/export_iterator
...
export tsdb.Iterator
2015-09-22 11:09:08 -06:00
d96119cc01
Correct MapperValueAsJSON comment
...
[ci skip]
2015-09-22 09:49:57 -07:00
007508bde0
export tsdb.Iterator
2015-09-22 10:29:49 -06:00
e43ad09f44
Perform custom unmarshal during unit tests
2015-09-22 00:24:43 -07:00
56193be05c
Simple tests work
2015-09-21 22:50:29 -07:00
07c9d40b87
Merge pull request #4179 from mark-rushakoff/sorted-measurements-by-tag-filters
...
Sort DatabaseIndex.measurementsByTagFilters result
2015-09-21 11:19:37 -07:00
e176d2c0dd
make SHOW TAG KEYS support (S)LIMIT & (S)OFFSET
2015-09-21 13:05:53 -04:00
8e236532e8
delete unused executeShowTagKeysStatement func
2015-09-21 11:32:13 -04:00
8bd2408320
convert SHOW TAG KEYS to distributed query
2015-09-21 11:30:51 -04:00
57b211a538
Merge branch 'master' into bottom
...
resolve changelog conflict
2015-09-21 12:31:29 +00:00
85275e7d59
Sort DatabaseIndex.measurementsByTagFilters result
...
Fixes #4118
2015-09-20 14:37:27 -07:00
5b24841973
rename topReduceOut
...
Also, gofmt
2015-09-18 15:04:09 +00:00
b07c36288d
Merge pull request #4130 from influxdb/topopt
...
optimize top queries
2015-09-18 14:40:57 +00:00
b5fc2a96b1
Implement bottom
...
After the rework of the ordering for top, this was just a matter
of adding a flag telling the minheap which way the values should
be sorted.
2015-09-18 14:39:46 +00:00
ce1dc840ef
Better documentation for typeCompare.
2015-09-18 13:11:46 +00:00
f9bfb2fcc5
Merge pull request #4142 from influxdb/nil_partition
...
If partition is nil return on Close immediately
2015-09-17 16:37:37 -07:00
72f6f7d268
Merge pull request #4134 from influxdb/issue-3447
...
Refactor Points and Rows to dedicated packages
2015-09-17 15:27:48 -05:00
d9e4b3b851
split comparison into 2 stages, one for type and one for value
...
Also, fix it so that heap inserts sort on tags as well.
This change makes it easier to implement bottom and also fixes the test that the previous change broke.
2015-09-17 14:59:02 +00:00
f73bc6ff19
Merge pull request #4140 from influxdb/engine_config
...
Make engine configurable
2015-09-16 21:17:43 -07:00
5e991f1703
If partition is nil return on Close immediately
2015-09-16 19:38:02 -07:00
7b1a4e6700
Control whether each query should be logged
...
Fixes issue #4138
2015-09-16 19:26:23 -07:00
e4fde993f1
Make engine configurable
2015-09-16 19:09:25 -07:00
ba830be3b9
actually move influxql.Row* -> models.Row*
2015-09-16 16:32:50 -05:00
d19a510ad2
refactor Points and Rows to dedicated packages
2015-09-16 15:33:08 -05:00
6d4319d244
Add function to tsdb.point to get line-protocol string in the correct units
2015-09-16 10:53:59 -07:00
59307b8b78
optimize top queries
...
Instead of rounding up the points, sorting and then slicing, keep a
heap that allows us to quickly see if the point needs to be in the
set. This cuts a top query on a dataset of 8 million points from 35
seconds to 11 seconds.
2015-09-16 17:02:44 +00:00
Paul Dix
Cory LaNou
Nathaniel Cook
Ben Johnson
Jason Wilder
Jason Wilder
Nathaniel Cook
Daniel Morsing
Sean Beckett
Daniel Morsing
Philip O'Toole
David Norton
Jason Wilder
Philip O'Toole
Mark Rushakoff
Cameron Sparr