Commit Graph

297 Commits (077cbba0e8d248f1e85b46973e7ecc7332b38e5b)

Author SHA1 Message Date
Edd Robinson 7d13bf3262 merge master 2017-12-08 17:21:58 +00:00
Edd Robinson f6835632e7 Merge master into branch 2017-12-08 17:11:07 +00:00
Adam a0b2195d6b
Pulled in backup-relevant code for review (#9193)
for issue #8879
2017-12-07 11:35:20 -05:00
Ben Johnson 493c1ed0d1
inmem tests passing. 2017-12-05 10:49:58 -07:00
Ben Johnson ca09f18e65
intermediate: tsdb compile 2017-11-29 11:20:18 -07:00
Edd Robinson e6b7140d65
Merge pull request #9143 from influxdata/er-show-tag-key-perf
SHOW TAG KEYS with high cardinality and many shards
2017-11-27 15:04:15 +00:00
Jason Wilder 279f82a72e Remove dead code 2017-11-22 11:17:34 -07:00
Jason Wilder cacb55fac4 Fix typos 2017-11-22 11:17:34 -07:00
Jason Wilder b674311830 Add magic number to fields index file 2017-11-22 11:17:34 -07:00
Jason Wilder dd1c030815 Remove limit count param on fields
It's not used anymore.
2017-11-22 11:17:34 -07:00
Jason Wilder c14b0e81b7 Save field types to speed up startup
This persists the field types in a shard to avoid having to scan
all the TSM files at startup.
2017-11-22 11:17:34 -07:00
Edd Robinson 68dd5e27c8 Improve performance of TagKeys 2017-11-21 17:16:47 +00:00
Edd Robinson 6851db3fc9 Add FGA support to SHOW MEASUREMENTS 2017-11-17 11:06:43 +00:00
Ben Johnson ede3fcf98e
intermediate 2017-11-15 16:09:25 -07:00
Jason Wilder 97e0d496a6 Add capability to force a full compaction
This adds the capability to the engine to force a full compaction
to be scheduled.  When called, it snapshots any data in the cache,
aborts running compactions and prevents level plans from returning
level plans.
2017-11-15 07:14:27 -07:00
Ben Johnson ba4c9e0317
Merge remote-tracking branch 'upstream/master' into er-tsi-index-part 2017-11-14 16:14:13 -07:00
Jason Wilder aee395d3bd Make DeleteSeriesRange take SeriesIterator 2017-11-13 09:02:10 -07:00
Jason Wilder f893beb6d8 Use MeasurementSeriesKeysByExprIterator for deletes 2017-11-13 09:02:10 -07:00
Jonathan A. Sternberg 0b7c56bcd8 Update the zap logger dependency
The previous sha was taken from a revision on a devel branch that I
thought would continue staying in the tree after it was merged. That
revision was rebased away and the API was changed for the logger.

This updates the usage of the logger and adds a simple package for
constructing the base logger.

The 1.0 version of zap changed the format of the default console logger
so this change moves over to this new logger instead of attempting to
retain backwards compatibility with the old format.
2017-11-10 16:27:16 -06:00
Ben Johnson 0ffd94a37a
Fix rebase 2017-11-09 09:25:10 -07:00
Ben Johnson 9ad2b53881
intermediate 2017-11-09 09:18:33 -07:00
Edd Robinson 98d584b63f Use index for SHOW X meta queries
When a meta query does not include a time component then it can be
answered exclusively by the index. This should result in a much faster
query execution that if the TSM engine was engaged.

This commit rewrites the following queries such that they make use
of the index where no time component is present:

  - SHOW MEASUREMENTS
  - SHOW SERIES
  - SHOW TAG KEYS
  - SHOW FIELD KEYS
2017-11-06 19:15:00 +00:00
Stuart Carnie f3d45ba301 influxdata/influxdb/influxql -> influxdata/influxql 2017-10-30 14:40:26 -07:00
Stuart Carnie c39f1ad748 Add batch cursor support to tsdb and tsm1
* batch cursors return slices of timestamps and values to reduce call
  overhead. Significantly improved iteration.
* added CreateCursor API to Shard, Engine
* moved build*Cursor to code gen
2017-10-25 13:38:07 -07:00
Stuart Carnie b7579340fe return query.ErrQueryInterrupted for read on InterruptCh 2017-10-24 14:10:28 -07:00
Stuart Carnie e9313876ab EXPLAIN ANALYZE
* Introduces EXPLAIN ANALYZE command, which
  produces a detailed tree of operations used to
  execute the query.

introduce context.Context to APIs

metrics package

* create groups of named measurements
* safe for concurrent access

tracing package

EXPLAIN ANALYZE implementation for OSS

Serialize EXPLAIN ANALYZE traces from remote nodes

use context.Background for tests

group with other stdlib packages

additional documentation and remove unused API

use influxdb/pkg/testing/assert

remove testify reference
2017-10-20 08:01:37 -07:00
Joe LeGasse 1443b22379 auth: add series auth to 'show tag values' 2017-09-27 20:01:18 -04:00
Edd Robinson 2def219f09 Refactor Shard to further protect Engine 2017-09-25 17:43:30 +01:00
Edd Robinson 4a67f92acc Prevent store from directly accessing Shard's engine 2017-09-25 17:43:01 +01:00
Edd Robinson 8e9cabbb9c Fix race in TagValues when reaching into engine 2017-09-25 17:43:01 +01:00
Edd Robinson 7739ff749a Ensure engine protected by shard mutex 2017-09-25 17:42:30 +01:00
Jason Wilder 940da04a34 Merge pull request #8829 from influxdata/jw-mmap
Release mmap pages when shard is cold
2017-09-18 12:08:37 -06:00
Jason Wilder 31646aae3a Release mmap pages when shard is cold
This instructs the kernel that it can release memory used by mmap'd
TSM files when they are not actively being used.  It the mappings are
use, the kernel will fault the pages back in.  On linux, this causes
RES memory to drop immediately when run.
2017-09-18 11:51:51 -06:00
Edd Robinson e39de3e427 Merge pull request #8782 from oiooj/pr-shard-fix
Correctly check if the Shard is ready for queries or writes
2017-09-18 18:17:19 +01:00
Jonathan A. Sternberg 2228b91b0d Unsigned data type parsing and prioritization 2017-09-14 12:28:13 -05:00
Stuart Carnie 4a6114028c exported UnloadIndex checks for ready state 2017-09-05 11:22:13 -07:00
kun 8a283e248c Correctly check if the Shard is ready for queries or writes 2017-09-03 15:14:58 +08:00
Jonathan A. Sternberg 091ea5f9a5 Merge pull request #8776 from influxdata/js-explain-plan
Initial implementation of explain plan
2017-09-01 16:19:37 -05:00
Jonathan A. Sternberg 50d404e690 Initial implementation of explain plan
It prints the statistics of each iterator that will access the storage
engine. For each access of the storage engine, it will print the number
of shards that will potentially be accessed, the number of files that
may be accessed, the number of series that will be created, the number
of blocks, and the size of those blocks.
2017-09-01 09:01:10 -05:00
kun 5d5225e77d Fix panic when engine closed in a shard 2017-08-29 17:22:45 +08:00
Jonathan A. Sternberg 9a2357c2c0 Separate the query engine into a separate package
This change provides a clear separation between the query engine
mechanics and the query language so that the language can be parsed and
dealt with separate from the query engine itself.
2017-08-16 13:38:43 -05:00
Ben Johnson 60ab1282ea
Refactor system iterators.
Previously pseudo iterators could be created for meta data such
as series, measurement, and tag data. These iterators were created
at a higher level and lacked a lot of the power of the query engine.

This commit moves system iterators down to the series level and
supports the following:

	- _name
	- _seriesKey
	- _tagKey
	- _tagValue
	- _fieldKey

These can be used as normal fields such as:

	SELECT _seriesKey FROM cpu

This will return all the series keys for `cpu`.
2017-08-16 09:27:29 -06:00
Ben Johnson c9b5d60753
Parse SHOW CARDINALITY. 2017-08-16 09:27:15 -06:00
Ben Johnson 06bc3b6fbf
TSI Index Migration 2017-08-15 11:40:24 -06:00
Stuart Carnie eec80692c4 Taught tsm1 storage engine how to read and write uint64 values
* introduced UnsignedValue type
  * leveraged existing int64 compression algorithms (RLE, Simple 8B)
* tsm and WAL can read and write UnsignedValue
* compaction is aware of UnsignedValue
* unsigned support to model, cursors and write points

NOTE: there is no support to create unsigned points, as the line
protocol has not been modified.
2017-07-24 09:03:22 -07:00
Jason Wilder 77afe50f7e Fix panic in ForEachMeasurementTagKey
If a shard was closed, ForEachMeasurementTagKey and TagKeyCardinality
would panic because the engine was nil.
2017-06-13 12:04:32 -06:00
Stuart Carnie 46796d932f add database to index, engine and shard; call AuthorizeSeriesRead 2017-05-26 13:21:50 -07:00
Ben Johnson 24446a0297
Implement zap logging in TSI. 2017-05-25 08:57:50 -06:00
Stuart Carnie 5c5bea2baa move Measurement and Series to inmem package 2017-05-19 08:17:09 -07:00
Jason Wilder 9445ccbad3 Expose shard meta info on Shard 2017-05-16 11:18:02 -06:00