Commit Graph

435 Commits (090f681737896feb99870ee534c9ccc4203b9a6d)

Author SHA1 Message Date
Edd Robinson 1e891b5fbc Change logging level 2017-11-30 14:08:44 +00:00
Jason Wilder 5cf7d52694 Ensure series keys are sorted
The Measurement added series keys from a map where the iteration
order is non-deterministic.  The keys should be returned in sorted
order.
2017-11-29 11:24:10 -07:00
Ben Johnson ca09f18e65
intermediate: tsdb compile 2017-11-29 11:20:18 -07:00
Edd Robinson 81976bca59 Refactor based on new design 2017-11-28 17:54:29 +00:00
Edd Robinson 38e0dd695f Allow concurrent access to Engine Index 2017-11-28 15:57:03 +00:00
Edd Robinson abae36f992 Ensure all index fields set 2017-11-28 15:57:02 +00:00
Edd Robinson 368420c670 Fix test due to index changes 2017-11-28 15:57:02 +00:00
Edd Robinson 67c67aeb34 Update test for Windows 2017-11-28 15:57:02 +00:00
Edd Robinson 12a2ff7fac Add support for TSI shard streaming and shard size
This commit firstly ensures that a shard's size on disk is accurately
reported when using the tsi1 index, by including the on-disk size of the
tsi1 index in the calculation.

Secondly, this commit add support for shard streaming/copying when using
the tsi1 index. Prior to this, a tsi1 index would not be correctly
restored when streaming shards.
2017-11-28 15:57:02 +00:00
Ben Johnson cc22134d8f
Merge branch 'er-tsi-index-part' of https://github.com/influxdata/influxdb into er-tsi-index-part 2017-11-27 07:52:39 -07:00
Ben Johnson 01491ca4f4
intermediate 2017-11-27 07:52:18 -07:00
Edd Robinson 4831545830 Add PR typo/doc changes 2017-11-27 14:05:30 +00:00
Ben Johnson fc966a1b67
Add series file backup/restore. 2017-11-22 08:55:54 -07:00
Edd Robinson 68dd5e27c8 Improve performance of TagKeys 2017-11-21 17:16:47 +00:00
Edd Robinson 5ff96d9193
Merge pull request #9127 from influxdata/er-fga
Implement FGA on SHOW COMMANDS
2017-11-20 14:51:58 +00:00
Jason Wilder 0551849298 Reduce calls to time.Now()
These were showing up in profiles during heavy write load.
2017-11-17 14:23:02 -07:00
Edd Robinson a5af19fc06 Address PR feedback 2017-11-17 12:43:48 +00:00
Edd Robinson bff69f7a82 Refactor inmem implementation 2017-11-17 11:06:43 +00:00
Edd Robinson 25f0fedd6f Fix MeasurementNamesByExpr in tsi1 2017-11-17 11:06:43 +00:00
Edd Robinson 3967e78885 Consolidate tests to tsdb package 2017-11-17 11:06:43 +00:00
Edd Robinson b3407c5d46 Correct authorisation on inmem SHOW MEASUREMENTS 2017-11-17 11:06:43 +00:00
Edd Robinson d4cecd7cc7 Add index authorisation test coverage 2017-11-17 11:06:43 +00:00
Edd Robinson 6851db3fc9 Add FGA support to SHOW MEASUREMENTS 2017-11-17 11:06:43 +00:00
Edd Robinson aa17ef55f9 Implement FGA on SHOW SERIES 2017-11-17 11:06:43 +00:00
Edd Robinson 8acab9b5ac Fix existing bug where database was empty 2017-11-17 11:06:43 +00:00
Ben Johnson ede3fcf98e
intermediate 2017-11-15 16:09:25 -07:00
Ben Johnson ba4c9e0317
Merge remote-tracking branch 'upstream/master' into er-tsi-index-part 2017-11-14 16:14:13 -07:00
Jason Wilder 8b18cc4456 Optimize deletes in tsi
The DropSeries code path ended up creating a MeasurementSeriesIterator
for each dropped series, this was too expensive just to see if a
series exists.

This adds a HasSeries func and fixes and issue where TSI files were
compacted while an iterator was still in use causing a panic.
2017-11-13 12:35:38 -07:00
Jason Wilder 13692639cb Fix create/delete series race
This fixes a race where writes and deletes to the same series and
measurements could sometimes leave the index in an inconsistent state.
2017-11-13 09:02:10 -07:00
Jason Wilder 80cd5e63af Optimize DeleteSeriesRange
This removes more allocations and speeds up some critical sections.
2017-11-13 09:02:10 -07:00
Jason Wilder f893beb6d8 Use MeasurementSeriesKeysByExprIterator for deletes 2017-11-13 09:02:10 -07:00
Jason Wilder 16d1f4309b Extract MeasurementSeriesKeysByExprIterator 2017-11-13 09:02:10 -07:00
Ben Johnson 9756a29678
import fix 2017-11-13 08:54:32 -07:00
Jonathan A. Sternberg 0b7c56bcd8 Update the zap logger dependency
The previous sha was taken from a revision on a devel branch that I
thought would continue staying in the tree after it was merged. That
revision was rebased away and the API was changed for the logger.

This updates the usage of the logger and adds a simple package for
constructing the base logger.

The 1.0 version of zap changed the format of the default console logger
so this change moves over to this new logger instead of attempting to
retain backwards compatibility with the old format.
2017-11-10 16:27:16 -06:00
Ben Johnson e278af2b18
intermediate 2017-11-09 09:30:19 -07:00
Ben Johnson d3cd750509
Refactor series file tombstoning. 2017-11-09 09:30:19 -07:00
Ben Johnson 3034d3fb54
intermediate 2017-11-09 09:30:19 -07:00
Ben Johnson 919f99f34d
Fixing tests. 2017-11-09 09:30:19 -07:00
Ben Johnson 07a743cca7
Rebase fixes 2017-11-09 09:29:19 -07:00
Edd Robinson 4471341d7e
Ensure error channel has capacity for all partitions 2017-11-09 09:28:37 -07:00
Ben Johnson b24b08a23c
Fix partition loading. 2017-11-09 09:28:37 -07:00
Ben Johnson 1f6d4ed1d1
Add series map. 2017-11-09 09:28:37 -07:00
Edd Robinson 49218fd3bd
Fix issue with series being added to log file 2017-11-09 09:28:37 -07:00
Edd Robinson 87778f3c45
Open partitions in parallel 2017-11-09 09:28:37 -07:00
Edd Robinson 3ae799b3a5
WIP Fix build 2017-11-09 09:28:37 -07:00
Edd Robinson ebb23df1cf
Implement most merge based methods 2017-11-09 09:28:37 -07:00
Edd Robinson 6d87ff7fa2
WIP - series point iterator 2017-11-09 09:28:37 -07:00
Edd Robinson aec607bddf
Implement Measurement sketches 2017-11-09 09:28:37 -07:00
Edd Robinson b39aa858cf
Implement series creation 2017-11-09 09:28:37 -07:00
Edd Robinson 65c6fa747e
Implement methods that don't require merge 2017-11-09 09:28:37 -07:00
Edd Robinson bf132004a3
Implement basic partition layout 2017-11-09 09:28:37 -07:00
Edd Robinson 7aa9de508d
Initial refactor of tsi1.Index
This commit carries out the initial refactor of the tsi1.Index into
tsi1.Partition. We then create a new tsi1.Index that will be an
abstraction over a collection of Partitions.
2017-11-09 09:27:56 -07:00
Edd Robinson fb646549f4
Index files -> partition files 2017-11-09 09:26:06 -07:00
Ben Johnson 328bffd658
Convert series ids to 64-bits. 2017-11-09 09:26:06 -07:00
Ben Johnson 0ffd94a37a
Fix rebase 2017-11-09 09:25:10 -07:00
Ben Johnson 08e459357a
Fix tsi race conditions. 2017-11-09 09:18:33 -07:00
Ben Johnson c75f1127aa
intermediate 2017-11-09 09:18:33 -07:00
Ben Johnson f223153078
Initial working version of series file. 2017-11-09 09:18:33 -07:00
Ben Johnson e05d4fdeeb
intermediate 2017-11-09 09:18:33 -07:00
Ben Johnson 9ad2b53881
intermediate 2017-11-09 09:18:33 -07:00
Ben Johnson 7259589241
intermediate 2017-11-09 09:18:33 -07:00
Ben Johnson 48b48a8927
intermediate 2017-11-09 09:13:46 -07:00
Ben Johnson 156f25ac23
Improve SHOW TAG KEYS performance. 2017-11-07 10:59:19 -07:00
Edd Robinson 07c4fdc1ed Fix data race on SeriesPointIterator 2017-11-07 10:48:23 +00:00
Stuart Carnie f3d45ba301 influxdata/influxdb/influxql -> influxdata/influxql 2017-10-30 14:40:26 -07:00
Ben Johnson 49c1fca036
Handle nil MeasurementIterator. 2017-10-26 11:25:46 -06:00
Edd Robinson 47bd069315 Fix race in Measurement index
Fixes #8989 and #8633.

Previously when issuing commands involving a regex check, walking
through the tags keys/values on a measurement, using the measurement's
index, would be racy.

This commit adds a new `TagKeyValue` type that abstracts away the
multi-layer map we were using as an inverted index from tag keys and
values to series ids. With this abstraction we can also make concurrent
access to this inverted index goroutine safe.

Finally, this commit fixes a very old bug in the index which will affect
any query using a regex. Previously we would always check _every_ tag
against a regex for a measurement, even when we had found a match.
2017-10-25 13:34:21 +01:00
Ben Johnson 5a77238f30
Sort & validate TSI key value insertion. 2017-10-23 10:46:01 -06:00
Ben Johnson 62093d2641 Merge pull request #8975 from benbjohnson/tsi-copy-returned-bytes
Copy returned bytes from TSI meta functions.
2017-10-18 09:26:02 -06:00
Ben Johnson 8ad2048a6b
TSI byte copy usage comments. 2017-10-18 07:21:54 -06:00
Ben Johnson d17d0f18e0
Move copyBytes() and copyByteSlices() to bytesutil. 2017-10-18 07:19:46 -06:00
Jason Wilder a6f4069ca7 Fix max select series limit for tsi
TSI did not check that the max select series limit during planning
the same way that inmem did.  This means that the limit could be
set but the planning of a high cardinality query would still OOM
the server.  This fixes that limit as well as makes the query interruptible
during planning.
2017-10-17 15:24:41 -06:00
Ben Johnson dceb88eb30
Copy returned bytes from TSI meta functions. 2017-10-17 14:05:35 -06:00
Jason Wilder 5033783a33 Handle deleted series when rebuilding measurment index 2017-10-16 10:50:16 -06:00
Joe LeGasse 1443b22379 auth: add series auth to 'show tag values' 2017-09-27 20:01:18 -04:00
Edd Robinson ea104596f0 Implement TSI index versioning
This commit adds a basic TSI versioning scheme, by adding a Version field
to an index's MANIFEST file.

Existing TSI indexes will not have this field present in their MANIFEST
files, and thus will be deemed incomatible with the current version.

Users with existing TSI indexes will be able to remove them, and convert the
resulting inmem indexes to the current version of a TSI index using the
influx_inspect tooling.
2017-09-22 17:59:39 +01:00
Edd Robinson 44691847e9 Merge branch 'master' into er-8678-tsi1-where 2017-09-22 16:54:49 +01:00
Jason Wilder d5d9f9acfe Remove debug line 2017-09-11 15:31:28 -06:00
Ben Johnson ee4d3c7b3d Invalidate all bloom filters. 2017-09-11 15:29:26 -06:00
Ben Johnson 3c2487b97a Clean up tsi bloom filter invalidation. 2017-09-11 15:29:26 -06:00
Ben Johnson 6af936ee61 Fix bloom filter invalidation. 2017-09-11 15:29:26 -06:00
Ben Johnson a40b2bb210 Simplify bloom filter invalidation. 2017-09-11 15:29:26 -06:00
Edd Robinson 408a78d904 Increase size of SeriesBlock partition 2017-09-11 15:29:26 -06:00
Ben Johnson 0ec2736f23 Incrementally rebuild tsi bloom filters. 2017-09-11 15:29:25 -06:00
Jason Wilder a5a2957567 Reduce allocation in log_file 2017-09-11 15:29:25 -06:00
Jason Wilder a9e89ede75 Reduce lock contenton on Index
Stat and Size are read-only and can take an RLock.
2017-09-11 15:26:25 -06:00
Jason Wilder 94e229ff59 Merge branch 'master' into jw-drop-series 2017-09-08 15:34:32 -06:00
Joe LeGasse 4fb35b373b auth: apply series auth to TSI 2017-09-08 09:09:53 -04:00
Jason Wilder a8d9eeef36 Reduce lock contention when deleting high cardinality series
Deleting high cardinality series could take a very long time, cause
write timeouts as well as dead lock the process.  This fixes these
issue to by changing the approach for cleaning up the indexes and
reducing lock contention.

The prior approach delete each series and updated every index (inmem)
during the delete.  This was very slow and cause the index to be locked
while it items in a slice were removed one by one.  This has been changed
to mark series as deleted and then rebuild the index asynchronously which
speeds up the process.

There was also a dead lock that could occur when deleing the field set.
Deleting the field set held a write lock and the function it invoked under
the lock could try to take a read lock on the field set.  This would then
deadlock.  This approach was also very slow and caused time out for writes.
It now uses faster approach that checks for the existing of the measurment
in the cache and filestore which does not take write locks.
2017-09-07 11:36:02 -06:00
Joe LeGasse 732a0c2eaa Merge pull request #8769 from influxdata/jl-map-cleanup
cleanup: remove poor usage of ',ok' with maps
2017-08-31 09:18:42 -04:00
Joe LeGasse a95647b720 cleanup: remove poor usage of ',ok' with maps
There are several places in the code where comma-ok map retrieval was
being used poorly. Some were benign, like checking existence before
issuing an unconditional delete with no cleanup. Others were potentially
far more serious: assuming that if 'ok' was true, then the resulting
pointer retrieved from the map would be non-nil. `nil` is a perfectly
valid value to store in a map of pointers, and the comma-ok syntax is
meant for when membership is distinct from having a non-zero value.
There was only one or two cases that I saw that being used correctly for
maps of pointers.
2017-08-30 09:49:31 -04:00
Stuart Carnie 51eb85193c release lock to avoid dead lock when calling WalkWhereForSeriesIDs
* WalkWhereForSeriesIDs may call SeriesIDs, which may attempt to
  upgrade from a `RLock` to a `Lock`, causing the dead lock
2017-08-29 16:12:51 -07:00
Stuart Carnie 0ced270197 fix race condition reading map 2017-08-28 13:36:49 -07:00
Jonathan A. Sternberg 697759613c Remove time comparisons from the inner sections of the storage engine 2017-08-16 16:51:13 -05:00
Jonathan A. Sternberg 9a2357c2c0 Separate the query engine into a separate package
This change provides a clear separation between the query engine
mechanics and the query language so that the language can be parsed and
dealt with separate from the query engine itself.
2017-08-16 13:38:43 -05:00
Ben Johnson 06bc3b6fbf
TSI Index Migration 2017-08-15 11:40:24 -06:00
Edd Robinson 45969ef3c6 Allow tag filtering when using DELETE with tsi1 2017-08-14 19:09:36 +01:00
Joe LeGasse 1121b69a9e auth: apply FGA to SHOW SERIES 2017-08-09 14:56:53 -04:00
Edd Robinson 0f648e5170 Remove unsafe shenanigans 2017-08-03 16:38:05 +01:00
Edd Robinson 2d57b599e9 Remove debugging statement 2017-08-02 17:24:00 +01:00
Edd Robinson da676a79ae Implement TSI iterator 2017-08-02 16:29:14 +01:00
Edd Robinson aa7095be5a Use a merge-based approach for TagValues 2017-08-02 14:10:52 +01:00
Jason Wilder 94a48774b7 Pull in new index filter 2017-08-02 14:10:52 +01:00
Jason Wilder c25f7b8b3f Fix duplicate points returned after delete
The sortedSeriesIds slice was not getting reset to 0 which caused
the same series ids to exist in the slice more than once.  Since
the size of the slice never matched the size of the seriesID map,
it kept appendending to the slice and sorting it which cause multiple
cursor to get created for the same series.

Fixes #8531
2017-07-10 10:37:01 -06:00
Stuart Carnie 2ccdda72a1 Free RLock prior to returning 2017-07-08 07:14:50 -07:00
Ben Johnson 9e64813db8
Defer unlock all write locks in inmem index.
Currently two write locks in `inmem` are obtained and then
manually unlocked at function exit points. However, we have
reports that the `inmem` index is hanging on a write lock and
cannot track the issue down to anything else besides a lock
that could have been left unlocked because of a panic.

This commit changes the two locks to always defer their unlocks
to prevent these hangs.
2017-06-29 10:23:13 -06:00
Ben Johnson f9dc61928a
Fix TSI issue with spaces in tag values. 2017-06-28 11:39:48 -06:00
Jason Wilder 9bd703d597 Fix possible deadlocks in inmem index 2017-06-21 12:07:40 -06:00
Ben Johnson b51f604030
Fix TSI non-contiguous compaction panic.
This fixes the case where log files are compacted out of order
and cause non-contiguous sets of index files to be compacted.

Previously, the compaction planner would fetch a list of index files
for each level and compact them in order starting with the oldest
ones. This can be a problem for level 1 because level 0 (log files)
are compacted individually and in some cases a log file can finish
compacting before older log files are finished compacting. This
causes there to be a gap in the list of level 1 files that is
ignored when fetching a list of index files.

Now, the planner reads the list of index files starting from the
oldest but stops once it hits a log file. This prevents that gap
from being ignored.
2017-06-13 10:53:26 -06:00
Summer d17c205b54 fix typo 2017-06-12 11:20:08 +08:00
marchtea 6e6f92c99a fix index file fd leak 2017-06-12 10:58:05 +08:00
Ben Johnson bcc6ef769b
Check file count before attempting a TSI level compaction.
This check was previously in a different section of code which
was lost during a refactor to the new compaction strategy. The
compaction planning now makes a check to ensure at least two
files are available for compaction in a level.
2017-06-06 11:08:59 -06:00
Stuart Carnie 47f97ea134 use parsed measurement and models.Tags 2017-05-26 13:21:59 -07:00
Stuart Carnie 3ec9b401f7 fix benchmark test 2017-05-26 13:21:59 -07:00
Stuart Carnie 46796d932f add database to index, engine and shard; call AuthorizeSeriesRead 2017-05-26 13:21:50 -07:00
Joe LeGasse 815f740f4c initial fga work
wip

wip

fix tests / build
2017-05-26 13:16:27 -07:00
Stuart Carnie c30c33dbcb Merge remote-tracking branch 'origin/master' into sgc-tagsets 2017-05-26 09:10:18 -07:00
Stuart Carnie c89d98dc02 gofmt 2017-05-25 16:00:23 -07:00
Stuart Carnie 386720b2e7 improvements to inmem/Measurement.TagSets API
```
benchmark                                             old ns/op     new ns/op     delta
BenchmarkMeasurement_TagSetsNoDimensions_1000-8       234054        117315        -49.88%
BenchmarkMeasurement_TagSetsDimensions_1000-8         996838        313313        -68.57%
BenchmarkMeasurement_TagSetsNoDimensions_100000-8     58940464      39452117      -33.06%
BenchmarkMeasurement_TagSetsDimensions_100000-8       175612060     70195562      -60.03%

benchmark                                             old allocs     new allocs     delta
BenchmarkMeasurement_TagSetsNoDimensions_1000-8       1026           26             -97.47%
BenchmarkMeasurement_TagSetsDimensions_1000-8         8026           2029           -74.72%
BenchmarkMeasurement_TagSetsNoDimensions_100000-8     100064         64             -99.94%
BenchmarkMeasurement_TagSetsDimensions_100000-8       800064         200067         -74.99%

benchmark                                             old bytes     new bytes     delta
BenchmarkMeasurement_TagSetsNoDimensions_1000-8       117080        69080         -41.00%
BenchmarkMeasurement_TagSetsDimensions_1000-8         549081        117176        -78.66%
BenchmarkMeasurement_TagSetsNoDimensions_100000-8     23298264      18498265      -20.60%
BenchmarkMeasurement_TagSetsDimensions_100000-8       66498276      23298360      -64.96%
```
2017-05-25 15:52:27 -07:00
Jason Wilder 14b54e08cb Fix compile error 2017-05-25 15:18:35 -06:00
Ben Johnson 24446a0297
Implement zap logging in TSI. 2017-05-25 08:57:50 -06:00
Ben Johnson 547db32d01
Fix tsi go vet issues. 2017-05-23 13:42:38 -06:00
Ben Johnson 48456d80ad
Remove tsi commented code. 2017-05-23 10:24:37 -06:00
Ben Johnson 2524df3405
Convert tsi1 series keys to uint32. 2017-05-23 09:48:13 -06:00
Ben Johnson c744e2f562
TSI pull request fixes. 2017-05-23 09:01:05 -06:00
Ben Johnson 57eeae03fc
Add note about SeriesIDs() limitation. 2017-05-23 08:42:25 -06:00
Ben Johnson e7f39c06ab
Refactor TSI1 compaction. 2017-05-23 08:42:25 -06:00
Ben Johnson 1975940f76
intermediate compaction commit 2017-05-23 08:42:25 -06:00
Ben Johnson 79edc0979c
Add temporary debugging stats for offset lookups. 2017-05-23 08:41:31 -06:00
Ben Johnson 48a06432df
Add tsi1 bloom filter. 2017-05-23 08:41:31 -06:00
Ben Johnson f3e08c5871
Delta encode tag and measurement block series data. 2017-05-23 08:41:31 -06:00
Ben Johnson 6f58149052
Increase tsi compaction factor. 2017-05-23 08:40:26 -06:00
Stuart Carnie 5c5bea2baa move Measurement and Series to inmem package 2017-05-19 08:17:09 -07:00
Jason Wilder 2cac46ebbc Convert usage of strings to []byte
Measurement name and field were converted between []byte and string
repetively causing lots of garbage.  This switches the code to use
[]byte in the write path.
2017-05-12 14:05:19 -06:00
Ben Johnson 489c89bea4
Add tsi support tooling. 2017-05-08 11:00:15 -06:00
Jason Wilder 73ddd4787b Fix race in SeriesN and CreateSeriesIfNotExists 2017-05-04 14:40:50 -06:00
Jason Wilder 7371f1067b Fix deadlock in Index.ForEachMeasurementTagKey
Index.ForEachMeasurementTagKey held an RLock while call the fn,
if the fn made another call into the index which acquired an RLock
and after another goroutine tried to acquire a Lock, it would deadlock.
2017-05-03 22:48:10 -06:00
Ben Johnson ba7108f94e
Add TSI documentation. 2017-04-21 14:45:03 -06:00
Jason Wilder 02b663b651 Fix lock contention in Index.CreateSeriesListIfNotExists
There was contention on the write lock which only needs to be acquired
when checking to see if the log file should be rolled over.
2017-04-20 12:28:42 -06:00
Jason Wilder 40ec85aacd Fix lock contention in LogFile.SeriesWithBuffer
Under high write load, the check for each series was done sequentially
which caused a lot of CPU time to acquire/release the RLock on LogFile.

This switches the code to check multiple series at once under an RLock
similar to the chang for inmem.
2017-04-20 12:28:42 -06:00
Jason Wilder 3c2825a851 Reduce lock thrashing when checking series
The inmem index would call CreateSeriesIfNotExist for each series
which takes and releases and RLock to see if a series exists. Under
high write load, the lock shows up in profiles quite a bit.  This
adds a filtering step that obtains a single RLock and checks all the
series and returns the non-existent series to contine though the slow
path.
2017-04-20 12:28:41 -06:00
Jason Wilder a19ce9c10f Reduce index lock contention
Series and Measurment have their own locks and we do not need to
hold locks on the index while using those types.
2017-04-18 16:32:33 -06:00
Ben Johnson 9c97cd8601
Merge remote-tracking branch 'upstream/master' into tsi 2017-04-04 12:46:09 -06:00
Ben Johnson 0d74497abe
Reset rhh map elements to reuse allocations. 2017-04-04 11:57:37 -06:00
Ben Johnson 6ff27c95e5
Fix tsi assertions. 2017-04-04 11:29:21 -06:00
Ben Johnson 95d4016ff2
Merge branch 'tsi' of https://github.com/influxdata/influxdb into tsi-series-block-partitioning 2017-04-04 10:14:03 -06:00
Ben Johnson bf49b176f5
Partition tsi1 series index. 2017-04-04 09:46:04 -06:00
Edd Robinson fddaff2cc8 Merge master in 2017-03-29 18:00:28 +01:00
Edd Robinson 116230b427 Use varint for tag count 2017-03-29 16:31:13 +01:00
Ben Johnson 2edfb1c92d
Ignore series limit on database load. 2017-03-24 16:27:16 -06:00
Jason Wilder ee03fbb164 Fix series tombstone sketch not updated when dropping measurment 2017-03-24 15:49:00 -06:00
Ben Johnson 9fb8f1ec1d
Fix database and tag limits. 2017-03-24 09:48:10 -06:00
Ben Johnson 85c1ae4dd4
Remove sort.Slice 2017-03-21 12:33:05 -06:00
Ben Johnson 1e9fa7bc2c
Fix 32-bit rhh implementation. 2017-03-21 11:44:13 -06:00
Jason Wilder 58c8736ebc Merge pull request #8172 from influxdata/jw-dropped-points
Fix series not getting created
2017-03-21 09:44:31 -06:00
Jason Wilder 92c722913d Unlock index if dropping non-existent series 2017-03-21 09:19:44 -06:00
Ben Johnson 5cf41adcb8
Optimize tsi1 write path.
Removes sorted series list in log, uses a buffer for HasSeries(),
and prepends a length for series key encoding.
2017-03-21 08:44:35 -06:00
Jason Wilder 7f78c6ad8e Fix series not getting created 2017-03-20 17:19:22 -06:00
Ben Johnson 5d67c424bf
Refactor tsi1 write locking. 2017-03-17 11:20:50 -06:00
Ben Johnson 70efc70abe
Reduce lock contention, fix rhh lookup. 2017-03-17 09:44:11 -06:00
Ben Johnson 1807772388
Fix tsi tests. 2017-03-15 11:23:58 -06:00
Ben Johnson ee2e046853
Merge remote-tracking branch 'upstream/tsi-log-compact' into tsi 2017-03-15 10:22:32 -06:00
Ben Johnson cf7ba96377
Merge branch 'tsi-log-compact' into tsi 2017-03-15 10:18:40 -06:00
Ben Johnson 358b1e0b05
Merge remote-tracking branch 'upstream/master' into tsi 2017-03-15 10:13:32 -06:00
Edd Robinson ddcea1c322 WHY YOU SMITE ME BEN. B. JOHNSON? 2017-03-15 12:50:03 +00:00
Ben Johnson d23f2971c3
Refactor TagBlockEncoder. 2017-03-10 10:08:16 -07:00
Ben Johnson 7ee0b9bb28
Rewrite SeriesBlockEncoder. 2017-02-16 09:39:51 -07:00
Ben Johnson 7e6f33ef48
TSI1 Index file compaction. 2017-02-15 08:44:22 -07:00
Ben Johnson 71d13fed6d
Immediate, parallel tsl compactions. 2017-02-15 08:44:21 -07:00
Edd Robinson 7749bc1c67 Store sorted series in logFile 2017-02-13 17:03:42 +00:00
Edd Robinson a6a2f9d5f0 Don't load meta data for tsi 2017-02-09 17:59:14 +00:00
Edd Robinson 2d59fb788c Modify arguments to reduce allocations 2017-02-09 15:01:32 +00:00
Edd Robinson c2bbc18e4b Reduce repeated string allocations 2017-02-09 11:23:23 +00:00
Edd Robinson 21e821e5f9 Reduce memory footprint by pre-allocating 2017-02-08 18:49:03 +00:00
Edd Robinson a9d58cc879 Add log file benchmarks 2017-02-08 16:50:46 +00:00
Ben Johnson d6fedcacd5
Refactor tsi.LogFile compaction. 2017-02-08 09:00:08 -07:00
Ben Johnson c4e9430de3
Recover after partial tsi log file write. 2017-02-07 10:02:31 -07:00
Ben Johnson d91e6eabac
Add max-values-per-tag to inmem index. 2017-02-06 11:14:13 -07:00
Edd Robinson 908959a78a Fix vet 2017-02-03 11:57:35 +00:00
Ben Johnson c246f3d9b0
Use inmem index on existing shards. 2017-02-02 10:04:25 -07:00
Edd Robinson 5a37d51f02 Refactor DropSeries 2017-02-01 22:07:44 +00:00
Ben Johnson 57f44d5f0c
Include index in snapshot. 2017-02-01 14:19:42 -07:00
Edd Robinson 0a9fc910f6 Fix potential race on inmem 2017-02-01 16:29:52 +00:00
Edd Robinson ffced9b538 Move sketch merging into File 2017-02-01 15:33:30 +00:00
Edd Robinson ab238251f4 Refactor measurement sketches 2017-02-01 14:56:35 +00:00
Edd Robinson d8a73a1954 Go vet 2017-02-01 14:51:15 +00:00
Ben Johnson 05bbda2c0a
Fix measurement block sketches. 2017-02-01 07:40:00 -07:00
Ben Johnson 86e908c384
Remove debug println(). 2017-01-31 09:24:58 -07:00
Ben Johnson f4a4d1e196
Fix series sketches. 2017-01-31 08:47:18 -07:00
Edd Robinson ab94c1b743 Fixes #7882 2017-01-30 19:12:24 +00:00
Ben Johnson 42959ead1f
Fix CreateSeriesListIfNotExists() data race. 2017-01-30 10:07:07 -07:00
Jason Wilder 2ae9406c1d Reincorporate memory leak fix
7832 got lost during a rebase/merge.  This reincorporate the fix
for the in-memory index as well as tsi.
2017-01-24 20:13:48 -07:00
Jason Wilder 173003bc38 Fix go vet 2017-01-24 20:13:37 -07:00
Ben Johnson 047c21f4d9
Merge remote-tracking branch 'upstream/master' into tsi 2017-01-24 09:28:58 -07:00
Ben Johnson c459d24a60
Test coverage. 2017-01-23 09:38:27 -07:00
Ben Johnson f43b0f7ec9
Fix series & measurement deletion. 2017-01-12 09:29:40 -07:00
Ben Johnson 352817e8c4
Convert 32-bit offsets to 64-bit. 2017-01-11 08:58:10 -07:00
Ben Johnson 64c7715243
Rebase fixes. 2017-01-09 10:10:12 -07:00
Ben Johnson d1f1e19591
Fixing rebase. 2017-01-06 09:31:25 -07:00
Ben Johnson 1003db0067
Add active log file tracking, time-based compaction. 2017-01-05 10:17:12 -07:00
Ben Johnson c1c98223ec
Fix and optimize tsi1 FileSet. 2017-01-05 10:17:12 -07:00
Ben Johnson 31e74d809b
Add tsi FileSet. 2017-01-05 10:17:11 -07:00
Ben Johnson dcd2a771b0
Optimizing tsi compaction. 2017-01-05 10:17:11 -07:00
Ben Johnson 1ce99e797f
Use series map in tsi1.LogFile. 2017-01-05 10:17:11 -07:00
Ben Johnson 9b1e8215e0
Remove dictionary encoding, add bulk series insertion. 2017-01-05 10:17:11 -07:00
Ben Johnson 5f7654173e
Add locking to sketch merge. 2017-01-05 10:17:11 -07:00
Ben Johnson 9bd19cdc69
Fix inmem DELETE SERIES. 2017-01-05 10:17:11 -07:00
Ben Johnson f9efcb3365
Re-add shared in-memory index. 2017-01-05 10:17:09 -07:00
Edd Robinson 0f9b2bfe6a
Fix tests 2017-01-05 10:16:15 -07:00
Edd Robinson 4ccb8dbab1
Move series count check to shard 2017-01-05 10:16:13 -07:00
Edd Robinson 0cb74eedbf
Add log file (WAL) sketches 2017-01-05 10:15:38 -07:00
Edd Robinson 190c78c644
Add series sketches 2017-01-05 10:15:37 -07:00
Edd Robinson 695adafc00
Add measurement sketches 2017-01-05 10:15:37 -07:00
Ben Johnson 745b1973a8
tsi compaction 2017-01-05 10:15:37 -07:00
Ben Johnson 83e80f6d0b
Fix in-mem index integration tests. 2017-01-05 10:15:37 -07:00
Ben Johnson 183418dcbd
Fix tsi TAG KEYS iterator. 2017-01-05 10:15:36 -07:00
Ben Johnson 759ff4ab80
Add tsi1 term hash index. 2017-01-05 10:15:35 -07:00
Ben Johnson 75cfe244c4
Add series hash index. 2017-01-05 10:15:35 -07:00
Ben Johnson 9f8b206b51
Fix measurement system queries. 2017-01-05 10:15:34 -07:00
Ben Johnson 4aa78383d1
Fix tsi1 series deletion. 2017-01-05 10:14:48 -07:00
Ben Johnson 5965610de6
Refactoring tsi tombstoning. 2017-01-05 10:14:02 -07:00
Ben Johnson e7940cc556
Add tsi1 series system iterator. 2017-01-05 10:14:00 -07:00
Ben Johnson 87f4e0ec0a
Add regex support in tsi1. 2017-01-05 10:12:29 -07:00
Ben Johnson d13afa8f47
Iterator refactoring 2017-01-05 10:11:49 -07:00
Jason Wilder f0427d180e
Fix tsi index panics
Hardcoded panics cause the server to crash in 10s due to stats collection.
2017-01-05 10:11:12 -07:00
Jason Wilder 4bf7b2bb19
Allow tsi to be enabled via config option 2017-01-05 10:11:12 -07:00
Jason Wilder 2b96c5d4d0
Set Tags on entry
These were lost when reloading from the index.  Fixes queries not
returning any data.
2017-01-05 10:11:12 -07:00
Jason Wilder a6490920fd
Fix reslicing indices
The slicing was backwards causing the buffer to grow indefinitely
and filling the disks on writes.
2017-01-05 10:11:12 -07:00
Jason Wilder 59864226b7
Add RWMutex to LogFile
Fixes concurrent map access panic
2017-01-05 10:11:12 -07:00
Ben Johnson fbe7f464ee
Improve insert performance. 2017-01-05 10:11:12 -07:00
Ben Johnson 33412782ed
Fix go vet issue. 2017-01-05 10:11:10 -07:00
Ben Johnson 2b864c72c5
Refactor MeasurementBlockTrailer read/write. 2017-01-05 10:11:10 -07:00
Ben Johnson cb93f10120
Remove per-shard in-memory index. 2017-01-05 10:11:09 -07:00
Ben Johnson 409b0165f5
shared in-memory index 2017-01-05 10:09:57 -07:00
Ben Johnson a812502ea3
reintegrating in-memory index 2017-01-05 10:07:35 -07:00