Commit Graph

694 Commits (67ea0b757b3729764b9e772eab47fd4239bc6b2e)

Author SHA1 Message Date
Ben Johnson f2d23b070b add tsm1 wal quickcheck
This commit adds quickcheck testing for the tsm1 WAL.
2015-10-14 09:38:38 -06:00
Daniel Morsing 6d188d9703 Merge pull request #4409 from influxdb/intoq
wire up INTO queries.
2015-10-14 15:29:54 +01:00
Philip O'Toole 6d38559f24 RLock access to store's shards
Fix issue #4442.
2015-10-13 20:28:19 -07:00
Sean Beckett 1d83c8c427 Update meta.go 2015-10-13 16:46:59 -07:00
Daniel Morsing 62dff895e2 wire up INTO queries.
Since INTO queries need to have absolute information about the database
to work, we need to create a loopback interface back to the cluster
in order to perform them.
2015-10-13 15:00:36 +00:00
Cory LaNou 6787525912 fixes multiple selectors overwriting each other. fixes #4360 2015-10-12 21:40:57 -05:00
Jason Wilder 16b3084ca9 Merge pull request #4397 from influxdb/jw-tsmdump
tsm1 file dump
2015-10-12 08:12:23 -06:00
Paul Dix a99c9ec5af Ensure rewrite index doesn't write corrupt file.
Fixes #4401
2015-10-10 16:58:25 -04:00
Jason Wilder 629219951a Fix timestamp encoding not using run-length encoding when possible
influx_inpsect uncovered some scenarios where timestamps could be stored using
run-length encoding but were being stored using simple8 which uses more space.
2015-10-09 22:38:17 -06:00
David Norton 9627f5ab84 fix #4280: add comments based on code review 2015-10-09 18:38:33 -04:00
David Norton 512d6ac050 fix #4280: only drop points matching WHERE clause 2015-10-09 18:34:32 -04:00
Jason Wilder 758359accc Prevent panic in DecodeSameTypeBlock
If DecodeSameTypeBlock is called on on an empty Values slice, it would
panic with an index out of bounds error.  This func can actually be removed
because DecodeBlock can determine what type of values are encoded already.

This will still panic if the block cannot be decoded due to other reasons.

Fixes #4365
2015-10-09 12:52:23 -06:00
Ben Johnson 2b3bb5336d add tsm1 quickcheck tests 2015-10-08 11:59:57 -06:00
Jason Wilder 79185fc1dc Fix index out of bounds panic in int64Decoder
Code was missing a check for when we did not have anymore bytes to decode
so it panic when we tried to decode the empty slice.
2015-10-08 11:21:19 -06:00
Jason Wilder b3343a6d2a Fix similar float values encoding overflow
If similar float values were encoded, the number of leading bits would
overflow the 5 available bits to store them (e.g. store 33 in 5 bits).  When
decoding, the values after the overflowed value would spike to very large and
small values.

To prevent the overflow, we clamp the value to 31 which is the maximum
number of leading zero bits we can encoded.

Fixes #4357
2015-10-07 15:05:56 -06:00
Nathaniel Cook 6f1a44bd07 Merge pull request #4345 from influxdb/remove_iterator
tsdb.Iterator is no longer used. Removing
2015-10-06 17:00:50 -06:00
Paul Dix f041939a1c Merge pull request #4308 from influxdb/pd-storage-engine
The TSM storage engine
2015-10-06 15:54:56 -07:00
Paul Dix b11308133a Only limit field count for non-tsm engines 2015-10-06 15:49:37 -07:00
Paul Dix 40ff4f4a86 Change default to bz1 2015-10-06 15:30:34 -07:00
Jason Wilder 41e3294d4a Fix panic: assignment to entry in nil map
Closing the store did not properly return an error for in-flight
writes because the closing channel was set to nil when closed.  A
nil channel is not selectable so writes continue on past the guard
checks and trigger panics.
2015-10-06 14:03:52 -06:00
Paul Dix be477b2aab Fix cursor bug on index 2015-10-06 12:26:45 -07:00
Nathaniel Cook d380ee37a2 tsdb.Iterator is no longer used. Removing 2015-10-06 10:34:07 -06:00
Paul Dix 267f34b94e Updates based on PR feedback 2015-10-05 20:09:56 -04:00
Paul Dix 26a93ec23e Fix deletes not kept if shutdown before flush on tsm1 2015-10-05 20:09:56 -04:00
Paul Dix bb398daf75 Updates based on @otoolp's PR comments 2015-10-05 20:09:56 -04:00
Jason Wilder c6f2f9cec2 Avoid duplicating values slice when encoding 2015-10-05 20:09:56 -04:00
Jason Wilder cb28dabf62 Make DecodeBlock panic if block size is too small
Should never get a block size 9 bytes since Encode always returns the min
timestampe and a 1 byte header.  If we get this, the engine is confused.
2015-10-05 20:09:56 -04:00
Jason Wilder b0449702e5 Fix comment typos 2015-10-05 20:09:56 -04:00
Paul Dix d9f94bdeeb Add db crash recovery 2015-10-05 20:09:56 -04:00
Jason Wilder 1d754db00b Propogate all encoding errors to engine
Avoid panicing in lower level code and allow the engine to decide what
it should do.
2015-10-05 20:09:56 -04:00
Jason Wilder 4c54c78009 Move compression encoding constants to encoders
Will make it less error-prone to add new encodings int the future
since each encoder has it's set of constants.  There are some placeholder
contants for uncompressed encodings which are not in all encoder currently.
2015-10-05 20:09:56 -04:00
Jason Wilder b1a57e1628 Fix go vet errors 2015-10-05 20:09:56 -04:00
Jason Wilder ab791ba913 Fix TestStoreOpenShardCreateDelete
Shard path can be a directory.
2015-10-05 20:09:56 -04:00
Paul Dix d47ddb5454 Cleanup after pd1 -> tsm1 name change. 2015-10-05 20:09:55 -04:00
Paul Dix 594253cbba Rename storage engine to tsm1, for Time Structured Merge Tree! 2015-10-05 20:09:55 -04:00
Paul Dix 0a11a2fdbc Add deletes to new storage engine 2015-10-05 20:09:55 -04:00
Paul Dix 4beca1a245 Implement reverse cursor direction on pd1 2015-10-05 20:09:55 -04:00
Jason Wilder dbf6228817 Fix go vet 2015-10-05 20:09:55 -04:00
Jason Wilder d9499f0598 Remove zig zag encoding from timestamp encoder
Not needed since all timestamps will be sorted in ascending order.  Negatives
are not possible.
2015-10-05 20:09:55 -04:00
Paul Dix a2b139e006 Fix compaction and multi-write bugs.
* Fix bug with locking when the interval completely covers or is totally inside another one.
* Fix bug with full compactions running when the index is actively being written to.
2015-10-05 20:09:55 -04:00
Jason Wilder 2366baaf0b Handle partial reads when loading WAL
If reading into fixed sized buffer using io.ReadFull, the func can
return io.ErrUnexpectedEOF if the read was short.  This was slipping
through the error handling causing the shard to fail to load.
2015-10-05 20:09:55 -04:00
Paul Dix 3332236527 Fix bugs with writing old data and compaction. 2015-10-05 20:09:55 -04:00
Jason Wilder 5d938d0a8b Add test with duplicate timestamps
Should not happen but makes sure that the same values are encoded
and decoded correctly.
2015-10-05 20:09:55 -04:00
Jason Wilder c47d14540d Add compressed string encoding
Uses snappy to compress multiple strings into a block
2015-10-05 20:09:55 -04:00
Paul Dix 861a15b3e6 Fix panic when data file has small index 2015-10-05 20:09:55 -04:00
Paul Dix be011b8da9 Add logging to pd1 2015-10-05 20:09:54 -04:00
Paul Dix c1213ba367 Update WAL to deduplicate values on Cursor query.
Added test and have failing section for single value encoding.
2015-10-05 20:09:54 -04:00
Jason Wilder 9f9692acdf Rename float encoding tests 2015-10-05 20:09:54 -04:00
Jason Wilder a4d92162ef Add documentation about compression 2015-10-05 20:09:54 -04:00
Jason Wilder 2da52ec4fe Fix deadlock in pd1_test.go
The defer tx.Rollback() tries to free the queryLock but the defer e.Cleanup() runs
before it and tries to take a write lock on the query lock (which blocks) and prevents
tx.Rollback() from acquring the read lock.
2015-10-05 20:09:54 -04:00
Jason Wilder 7e0df18e1a Update simple8b api usage 2015-10-05 20:09:54 -04:00
Jason Wilder cb23f5ac53 Add a compressed boolean encoding
Packs booleans into bytes using 1 bit per value.
2015-10-05 20:09:54 -04:00
Jason Wilder 1196587dc4 Keep track of the type of the block encoded
Allowes decode to decode an arbitrary block correctly.
2015-10-05 20:09:54 -04:00
Jason Wilder 731ae27123 Remove unnecessary allocations from int64 decoder
The decoder was creating a large slice and decoding all values when
instead, it could decode one packed value as needed.
2015-10-05 20:09:54 -04:00
Jason Wilder 95046c1e37 Add test assertions for time encoding type 2015-10-05 20:09:54 -04:00
Jason Wilder e42d8660d0 Fix run length encoding check
Values were run length encoded even when they should not have been
2015-10-05 20:09:54 -04:00
Jason Wilder 092689c131 Reduce memory allocations
Converting between different encoders is wasting a lot of memory allocating different
typed slices.
2015-10-05 20:09:54 -04:00
Jason Wilder ce1d45ecda Use zigzag encoding for timestamp deltas
Previously were using a frame of reference approach where we would
transform the (possibly negative) deltas into positive values from
the minimum.  That required an extra pass over the values as well
as a large slice allocation so we could encode the originals in uncompressed
form if they were too large.

This switches the encoding to use zigzag encoding for the deltas which
removes the extra slice allocation as well as the extra loops.

Improves encoding performane by ~4x.
2015-10-05 20:09:54 -04:00
Jason Wilder 4a37ba868d Add int64 compression
This is using zig zag encoding to convert int64 to uint64s and then using simple8b
to compress them, falling back to uncompressed if the value exceeds 1 << 60.  A
patched encoding scheme would likely be better in general but this provides decent
compression for integers that are not at the ends of the int64 range.
2015-10-05 20:09:53 -04:00
Jason Wilder 42e1babe7f Add time and float compression
Time compression uses an adaptive approach using delta-encoding,
frame-of-reference, run length encoding as well as compressed integer
encoding.

Float compression uses an implementation of the Gorilla paper encoding
for timestamps based on XOR deltas and leading and trailing null suppression.
2015-10-05 20:09:53 -04:00
Jason Wilder 112a03f24c Fix go vet errors 2015-10-05 20:08:58 -04:00
Jason Wilder 88248f3f81 Ensure we have files when iterating in cursor
Prevents index out of bounds panic
2015-10-05 20:08:58 -04:00
Paul Dix db4ad33f3c Update tests to use transactions. Add test for single series 10k points. 2015-10-05 20:06:22 -04:00
Paul Dix 0b33a71bb7 Add recover to maintenance. Change snapshot writer to not use bolt on shard. 2015-10-05 20:06:22 -04:00
Paul Dix 0fd116d1f2 Ensure data files can't be deleted while query is running.
Also ensure that queries don't try to use files that have been deleted.
2015-10-05 20:06:22 -04:00
Paul Dix b1bdb4f15a Make compaction run at most at set duration. 2015-10-05 20:06:22 -04:00
Paul Dix 1c8eac1523 Add PerformMaintenance to store for flushes and compactions.
Also fixed shard to work again with b1 and bz1 engines.
2015-10-05 20:06:22 -04:00
Paul Dix d694454f47 Fix wal flushing, compacting, and write lock 2015-10-05 20:06:22 -04:00
Paul Dix 6c94e738a0 Add support for multiple fields 2015-10-05 20:06:22 -04:00
Paul Dix 667b3e6c08 Handle hash collisions on keys 2015-10-05 20:06:22 -04:00
Paul Dix 48069e782c Add compaction and time range based write locks. 2015-10-05 20:06:22 -04:00
Paul Dix 2eb2a647d6 Add multicursor to combine wal and index 2015-10-05 20:06:22 -04:00
Paul Dix 7baba84a21 Ensure we don't have duplicate values. Fix panic in compaction. 2015-10-05 20:06:22 -04:00
Paul Dix 0770ccc87d Make writes to historical areas possible 2015-10-05 20:06:21 -04:00
Paul Dix 982c28b947 Update to work with new cursor definitiono and Point in models 2015-10-05 20:06:21 -04:00
Paul Dix 365a631b53 Update wal to only open new segment file on flush if its not an idle flush 2015-10-05 20:06:21 -04:00
Paul Dix 7c8ab4f1d8 Add test for close and restart of engine and fix errors. 2015-10-05 20:06:21 -04:00
Paul Dix c5f6c57d7f Update engine to put index at the end of data files 2015-10-05 20:06:21 -04:00
Paul Dix fe1f9a51e5 Add memory settings and WAL backpressure 2015-10-05 20:06:21 -04:00
Paul Dix 5e59cb9393 Update encoding test to work with new interface. 2015-10-05 20:06:21 -04:00
Paul Dix 2100e66437 Add full durability to WAL and flush on startup 2015-10-05 20:06:21 -04:00
Paul Dix 82e1be7527 WIP: more WAL work 2015-10-05 20:06:21 -04:00
Paul Dix 2ba032b7a8 WIP: finish basics of PD1. IT WORKS! (kind of) 2015-10-05 20:06:21 -04:00
Paul Dix 7555ccbd70 WIP: engine work 2015-10-05 20:06:21 -04:00
Paul Dix 12ea1cb26f Add comment about encoding float 2015-10-05 20:06:21 -04:00
Paul Dix fb2a1cb2f3 WIP: skeleton for encoding for new engine 2015-10-05 20:06:20 -04:00
David Norton 4375545064 fix #4276: walk DropSeriesStatement 2015-10-05 19:56:30 -04:00
Philip O'Toole 2ac0357406 Support dropping non-Raft nodes 2015-10-04 00:19:52 -07:00
Philip O'Toole d74e0690c7 Revert "Merge pull request #4233 from influxdb/drop-server"
This reverts commit 0bdb36f6dc, reversing
changes made to 3085fbc138.
2015-10-02 08:39:57 -07:00
Cory LaNou f50813460e protobuf update.. :-( 2015-10-01 15:39:15 -05:00
Jason Wilder 06c143c2dd Handle missing values in aggregate derivative queries better
If an aggregate derivative query did not have a value in the first
time bucket, it would abort early and return a single row with value
of 0.  Similarly, if either the current or previous value was nil,
it would skip the row and not append any values causing gaps and
no data to show up.

Instead, this will append a nil value if either the current or previous
valis is nil.  This essentially allows nil values to carry through the
results as well as gives a more sensible value for rows where we cannot
compute a difference (instead of dropping the row as before).

Fixes #4237 #4263
2015-10-01 13:05:19 -06:00
Ben Johnson 343dd23ee7 refactor map functions to use list of values
This commit changes `tsdb.mapFunc` to use `tsdb.MapInput` instead
of an iterator. This will make it easier and faster to pass blocks
of values from the new storage engine into the engine.
2015-09-29 14:00:33 -06:00
Cory LaNou 842b36f30f godoc 2015-09-23 15:52:44 -05:00
Cory LaNou 715e3d8d3c add comment to clarify 2015-09-23 15:49:17 -05:00
Cory LaNou d6ee542083 SelectFilterFields -> SelectWhereFields 2015-09-23 15:47:12 -05:00
Cory LaNou b2474c9abf reduce arguments to MapTopBottom 2015-09-23 15:44:36 -05:00
Cory LaNou 104c049a53 fix typo 2015-09-23 15:26:13 -05:00
Cory LaNou 81ad1f87a4 refactor MapTopBottom signature 2015-09-23 10:53:21 -05:00
Cory LaNou 7c3a542e13 revert signatures for mean/sum map funcs 2015-09-23 10:32:00 -05:00
Cory LaNou 6b19e3dd9c minor refactorings 2015-09-23 09:47:24 -05:00
Cory LaNou b1becfbcfc more tests, simplify mapFunc signatures 2015-09-23 09:27:19 -05:00
Cory LaNou 78bc740434 full support for min/max/first/last 2015-09-23 09:03:57 -05:00
Ben Johnson 8e27cf1fd8 Merge remote-tracking branch 'upstream/master' into refactor-select-mapper
Conflicts:
	tsdb/store.go
2015-09-22 13:58:24 -06:00
dgnorton 410eb4ece4 Merge pull request #3852 from influxdb/dmq-show-tag-keys
convert SHOW TAG KEYS to distributed query
2015-09-22 15:28:08 -04:00
Ben Johnson 96715d7d90 rename Cursor.Seek() to Cursor.SeekTo() 2015-09-22 13:23:16 -06:00
Ben Johnson 56cb2fae5d fix integration tests 2015-09-22 13:10:13 -06:00
Ben Johnson 649663ca15 fix tests 2015-09-22 13:10:13 -06:00
Ben Johnson b213ddad78 refactor cursor 2015-09-22 13:10:12 -06:00
Ben Johnson a5269e9cc7 rename direction to ascending. 2015-09-22 13:09:26 -06:00
Ben Johnson 1b8b625787 refactor SelectMapper 2015-09-22 13:09:26 -06:00
Ben Johnson 0883182798 simplify select mapper stmt 2015-09-22 12:27:14 -06:00
Nathaniel Cook 79e6e3e07a Merge pull request #4196 from influxdb/export_iterator
export tsdb.Iterator
2015-09-22 11:09:08 -06:00
Philip O'Toole d96119cc01 Correct MapperValueAsJSON comment
[ci skip]
2015-09-22 09:49:57 -07:00
Nathaniel Cook 007508bde0 export tsdb.Iterator 2015-09-22 10:29:49 -06:00
Philip O'Toole e43ad09f44 Perform custom unmarshal during unit tests 2015-09-22 00:24:43 -07:00
Philip O'Toole 56193be05c Simple tests work 2015-09-21 22:50:29 -07:00
Philip O'Toole 07c9d40b87 Merge pull request #4179 from mark-rushakoff/sorted-measurements-by-tag-filters
Sort DatabaseIndex.measurementsByTagFilters result
2015-09-21 11:19:37 -07:00
David Norton e176d2c0dd make SHOW TAG KEYS support (S)LIMIT & (S)OFFSET 2015-09-21 13:05:53 -04:00
David Norton 8e236532e8 delete unused executeShowTagKeysStatement func 2015-09-21 11:32:13 -04:00
David Norton 8bd2408320 convert SHOW TAG KEYS to distributed query 2015-09-21 11:30:51 -04:00
Daniel Morsing 57b211a538 Merge branch 'master' into bottom
resolve changelog conflict
2015-09-21 12:31:29 +00:00
Mark Rushakoff 85275e7d59 Sort DatabaseIndex.measurementsByTagFilters result
Fixes #4118
2015-09-20 14:37:27 -07:00
Daniel Morsing 5b24841973 rename topReduceOut
Also, gofmt
2015-09-18 15:04:09 +00:00
Daniel Morsing b07c36288d Merge pull request #4130 from influxdb/topopt
optimize top queries
2015-09-18 14:40:57 +00:00
Daniel Morsing b5fc2a96b1 Implement bottom
After the rework of the ordering for top, this was just a matter
of adding a flag telling the minheap which way the values should
be sorted.
2015-09-18 14:39:46 +00:00
Daniel Morsing ce1dc840ef Better documentation for typeCompare. 2015-09-18 13:11:46 +00:00
Philip O'Toole f9bfb2fcc5 Merge pull request #4142 from influxdb/nil_partition
If partition is nil return on Close immediately
2015-09-17 16:37:37 -07:00
Cory LaNou 72f6f7d268 Merge pull request #4134 from influxdb/issue-3447
Refactor Points and Rows to dedicated packages
2015-09-17 15:27:48 -05:00
Daniel Morsing d9e4b3b851 split comparison into 2 stages, one for type and one for value
Also, fix it so that heap inserts sort on tags as well.

This change makes it easier to implement bottom and also fixes the test that the previous change broke.
2015-09-17 14:59:02 +00:00
Philip O'Toole f73bc6ff19 Merge pull request #4140 from influxdb/engine_config
Make engine configurable
2015-09-16 21:17:43 -07:00
Philip O'Toole 5e991f1703 If partition is nil return on Close immediately 2015-09-16 19:38:02 -07:00
Philip O'Toole 7b1a4e6700 Control whether each query should be logged
Fixes issue #4138
2015-09-16 19:26:23 -07:00
Philip O'Toole e4fde993f1 Make engine configurable 2015-09-16 19:09:25 -07:00
Cory LaNou ba830be3b9 actually move influxql.Row* -> models.Row* 2015-09-16 16:32:50 -05:00
Cory LaNou d19a510ad2 refactor Points and Rows to dedicated packages 2015-09-16 15:33:08 -05:00
Cameron Sparr 6d4319d244 Add function to tsdb.point to get line-protocol string in the correct units 2015-09-16 10:53:59 -07:00
Daniel Morsing 59307b8b78 optimize top queries
Instead of rounding up the points, sorting and then slicing, keep a
heap that allows us to quickly see if the point needs to be in the
set. This cuts a top query on a dataset of 8 million points from 35
seconds to 11 seconds.
2015-09-16 17:02:44 +00:00
Cory LaNou 5395ac7634 Period in field name that matches measurement fails. Fixes #3457 2015-09-15 16:26:39 -05:00
Daniel Morsing 51b22f180f simplify the sorting for top/distinct.
Instead of using closures and 2 different sort routines, have an
interface compare method that makes it easy to switch directions
for comparisons.

Note that this changes the sort order of distinct to match that of
top. While it is a change, I don't think it will break any code. The
important thing for distinct is just that the ordering is absolute,
not what the order is.
2015-09-15 15:03:31 +00:00
Philip O'Toole 93f1fcb38c Merge pull request #4073 from influxdb/wal_stat_fixes
WAL statistics fixes
2015-09-14 07:10:26 -07:00
Daniel Morsing eb65f50593 gofmt 2015-09-11 14:41:59 +00:00
Daniel Morsing ab9ae468ca unexport some functions
For good measure basically. Not needed for correctness, but it keeps
people from using these.
2015-09-11 14:39:09 +00:00
Daniel Morsing 66fc270d1e Move aggregate funcs into tsdb
Pure move, No functional changes.
2015-09-11 14:27:47 +00:00
Cory LaNou 5c1ba44c9b Merge pull request #4071 from influxdb/issue-3902
Issue 3902
2015-09-11 06:58:27 -05:00
Philip O'Toole 40b1068c81 Use unified statMap for WAL
Don't declare distinct stat map for partitions. It's more useful to see
the stats collated together per-WAL. This may need further change in the
future.
2015-09-10 14:23:40 -07:00
Philip O'Toole 13a302e533 WAL tag keys are "path" not "bind". 2015-09-10 14:10:45 -07:00
Philip O'Toole bf55f61edd Add stats for the WAL 2015-09-10 12:30:47 -07:00
Philip O'Toole 5086ea42fa Update WAL comments
[ci skip]
2015-09-10 11:29:43 -07:00
Philip O'Toole 101a4d2a55 Merge pull request #4066 from influxdb/pd-fail-writes-on-memory-pressure
Update WAL to fail writes if pressure too high.
2015-09-10 11:27:32 -07:00
Cory LaNou 05f69b3d6c Validate Dimension for proper time scenarios. Fixes #3902 2015-09-10 13:10:21 -05:00
Philip O'Toole ed917aa7f8 Log each query before execution
If a query causes a server to panic, the query is nowhere in the logs,
which makes debug very difficult.
2015-09-10 10:52:51 -07:00
Ben Johnson 733fa0a109 disable bz1 recompression
This commit only appends new blocks of points and disables checks for
recompressing small blocks at the end of a series.
2015-09-10 11:26:29 -06:00
Paul Dix 2d67a9ea22 Update WAL to fail writes if pressure too high.
If the memory gets 5x above the partition size threshold, the WAL will start returning write failures to the clients. This will allow them to backoff their write volume.

Also updated the stress script to track failed requests and output messages on failure and when it returns to success.
2015-09-09 22:41:32 -07:00
Paul Dix 482e00d3e3 Merge pull request #4011 from influxdb/pd-simplify-wal
Simplify WAL to not compact
2015-09-08 22:32:53 -07:00
Philip O'Toole 3db9cc9b76 Basic instrumentation for shards 2015-09-08 19:56:42 -07:00
Philip O'Toole 2b0a40e262 Merge pull request #4007 from influxdb/instrument_bz1
Basic instrumentation for bz1 engine
2015-09-08 19:56:06 -07:00
Paul Dix ecbc79e7e3 Fix disksize to work with new WAL 2015-09-08 19:37:33 -07:00
Philip O'Toole 5373f263a3 Add pending control to batcher
With this change, the generic batcher used by many inputs can now be
buffered. Testing shows that this performance of the Graphite input by
10-100%, with the biggest improvements at lower numbers of connections.
2015-09-08 19:32:00 -07:00
Paul Dix dfd6b11dda Fix memory compaction logic.
* Only fire a go routine to flush and compact if it isn't already running
* Have a sleep backoff time that scales up as the percentage of memory used goes up
2015-09-08 19:28:29 -07:00
Paul Dix a1fb77198b Simplify WAL to not compact since it doesn't really help the engine anyway 2015-09-08 19:28:29 -07:00
Philip O'Toole 76903f7440 Instrument bz1 engine 2015-09-08 19:09:39 -07:00
Cory LaNou 62e9c24b25 fixes #3926 2015-09-08 14:15:48 -05:00
Philip O'Toole 255fb3364d Don't close nil channel
Obviously a TSDB store may be closed before it's opened.
2015-09-08 10:49:45 -07:00
Jason Wilder 6b2b29625d Ensure the tsdb.Store is not closed before creating a shard
Fixes panic: assignment to entry in nil map

Fixes #3848
2015-09-08 11:04:00 -06:00
Ben Johnson fd9be63b4e rollback bolt tx on mapper open error
This commit fixes `SelectMapper.Open()` so that it properly rolls back
transactions. Previously, this caused transactions to stay open
indefinitely which caused mmap resizes to hang indefinitely.
2015-09-08 10:28:51 -06:00
Jason Wilder 13dbc8f0ba Merge pull request #3841 from influxdb/jw-file-utils
Add inspect tool
2015-09-04 14:12:05 -06:00
Cory LaNou fa4415b3a4 refactor processing top/bottom results. clarify some comments 2015-09-04 13:30:43 -05:00
Cory LaNou b62d8c0515 expand variable names for clarity 2015-09-04 13:30:43 -05:00
Cory LaNou 65e652850a btf -> tmin 2015-09-04 13:30:42 -05:00
Cory LaNou 08295c578f refactor processTopBottom 2015-09-04 13:30:42 -05:00
Cory LaNou 9ab3d89c06 bucketTime* -> tMin* 2015-09-04 13:30:42 -05:00
Cory LaNou 3ca93594c3 BucketTime -> TMin 2015-09-04 13:30:42 -05:00
Cory LaNou 347ffc70b4 wire up advanced top sorting/slicing 2015-09-04 13:30:41 -05:00
Cory LaNou 8c4595b345 top is coming together. filling out fields properly 2015-09-04 13:30:41 -05:00
Cory LaNou ba79007960 wip 2015-09-04 13:30:41 -05:00
Cory LaNou 35b9215aa9 refactor processTopBottom - wip 2015-09-04 13:30:40 -05:00
Cory LaNou 046282249a wip remapping top output 2015-09-04 13:30:40 -05:00
Cory LaNou 97f2dc830f comment/type fixes 2015-09-04 13:30:39 -05:00
Cory LaNou 72fd115dc2 exposing tags on cursors, top/bottom are valid funcs now 2015-09-04 13:30:39 -05:00
Jason Wilder 6b4926257a Add inspect tool
Start of a lower-level file inspection tool.  This currently dumps
summary statistics for the shards, index and WAL that can be used to
understand the shape of the data is in the local shards.  This util
operates on the shards itself and not through the server and is intended
more for debugging/troubleshooting.
2015-09-04 10:38:59 -06:00
Jason Wilder 6f41c0fa87 Merge pull request #3986 from influxdb/jw-order-by
Support sorting by time desc
2015-09-04 09:42:58 -06:00
Jason Wilder df70a1c8ce Update tests to use Direction enum 2015-09-04 09:00:11 -06:00
Jason Wilder e767feb8d9 Fix order by desc with aggregate function not return any values 2015-09-03 22:31:58 -06:00
Jason Wilder 7fa3d445f7 Support reverse iteration for b1 engine 2015-09-03 22:31:58 -06:00
Jason Wilder 2725757dba Simplify WAL cursor seek movement logic 2015-09-03 22:31:58 -06:00
Jason Wilder 5a6b0afc4b Replace cursor direction with a type 2015-09-03 22:31:48 -06:00
Jason Wilder 7c67e60c4f Add bz1 reverse cursor test 2015-09-03 22:28:36 -06:00
Jason Wilder 5e481181bc Add WAL reverse cursor test 2015-09-03 22:28:36 -06:00
Jason Wilder e206021e13 Add reverse multi-cursor tests 2015-09-03 22:28:36 -06:00
Jason Wilder 266bdc1c2b Support sort by time DESC in wal and bz1 engines 2015-09-03 22:28:36 -06:00
Philip O'Toole e07432c59f Implement diagnostics support
This change adds support for diagnostics by decomposing the existing
interface into two interfaces -- one for stats, and the other for
diags. It also adds some basic monitor of system, network, and the Go
runtime.
2015-09-03 20:50:54 -07:00
Cory LaNou 6592dcc699 EnableLogging -> LoggingEnabled 2015-09-03 16:56:07 -05:00
Ben Johnson deff06f850 add copier service
This commit adds the copier service which allows one server to
copy shards from another server. This will be used for moving
shards in the cluster.
2015-09-03 13:07:35 -06:00
David Norton 816c5f5368 fix #2555: don't normalize target names 2015-09-03 07:12:15 -04:00
David Norton 99a22c174b fix #2555: add backreference in CQs
Add new query syntax to allow the following in CQs:

INTO "1hPolicy".:MEASUREMENT
2015-09-03 07:12:15 -04:00
Ben Johnson b63ebb72a5 limit bz1 quickcheck tests to 10 iterations on CI
This commit checks the `CI` environment variable in the bz1
test suite and limits the quickcheck runs if the value is `true`.
2015-09-02 11:27:11 -06:00
Philip O'Toole e15bc6df11 Remove obsolete TSDB monitor file
This functionality will be superseded by the new monitor service.
2015-09-01 23:15:57 -07:00
Philip O'Toole 14c04eb4d6 Merge pull request #3916 from influxdb/new_stats_diags
Statistics and Diagnostics service
2015-09-01 18:30:53 -07:00
Philip O'Toole f05dc20b58 Hook new monitor service to server
u
2015-09-01 15:03:52 -07:00
Jason Wilder 898ee8c399 Fix write fails for multiple points when tag starts with quote
Fixes #3928
2015-09-01 11:20:34 -06:00