Commit Graph

207 Commits (64c7715243eb1b48f20ea596cc617cb8bbe29a0f)

Author SHA1 Message Date
Ben Johnson d1f1e19591
Fixing rebase. 2017-01-06 09:31:25 -07:00
Ben Johnson 9b1e8215e0
Remove dictionary encoding, add bulk series insertion. 2017-01-05 10:17:11 -07:00
Ben Johnson f9efcb3365
Re-add shared in-memory index. 2017-01-05 10:17:09 -07:00
Edd Robinson 0f9b2bfe6a
Fix tests 2017-01-05 10:16:15 -07:00
Edd Robinson 4ccb8dbab1
Move series count check to shard 2017-01-05 10:16:13 -07:00
Ben Johnson 9f8b206b51
Fix measurement system queries. 2017-01-05 10:15:34 -07:00
Ben Johnson e7940cc556
Add tsi1 series system iterator. 2017-01-05 10:14:00 -07:00
Ben Johnson 87f4e0ec0a
Add regex support in tsi1. 2017-01-05 10:12:29 -07:00
Jason Wilder 1ba64f3610
Disable max-value-per-tag option temporarily
This is too slow currently and causes all writes to timeout.
2017-01-05 10:11:47 -07:00
Jason Wilder 4bf7b2bb19
Allow tsi to be enabled via config option 2017-01-05 10:11:12 -07:00
Ben Johnson fbe7f464ee
Improve insert performance. 2017-01-05 10:11:12 -07:00
Ben Johnson bf89b94d17
Fix WalkTagKeys(). 2017-01-05 10:11:11 -07:00
Ben Johnson cb93f10120
Remove per-shard in-memory index. 2017-01-05 10:11:09 -07:00
Ben Johnson 409b0165f5
shared in-memory index 2017-01-05 10:09:57 -07:00
Ben Johnson a812502ea3
reintegrating in-memory index 2017-01-05 10:07:35 -07:00
Ben Johnson 62d2b3ebe9
Series filtering. 2017-01-05 10:02:42 -07:00
Ben Johnson 62269c3cea
intermediate 2017-01-05 10:02:41 -07:00
Ben Johnson 5f5b02e052
intermediate 2017-01-05 10:01:49 -07:00
Ben Johnson 2a81351992
Implement tsdb.Index interface on tsi1.Index. 2017-01-05 10:00:43 -07:00
Edd Robinson da63b349a4
Fix bad rebase 2017-01-05 09:59:44 -07:00
Edd Robinson 2d9bd09784
Use []byte where possible in Index 2017-01-05 09:57:34 -07:00
Edd Robinson 3187cd4432
Cleanup series created stat 2017-01-05 09:56:49 -07:00
Edd Robinson d19fbf5ab4
Wire in HLL estimator 2017-01-05 09:54:03 -07:00
Edd Robinson 05bc4dec00
Refactor 2017-01-05 09:50:23 -07:00
Edd Robinson c535e3899a
Remove in-memory index from Shard and Store 2017-01-05 09:47:09 -07:00
Edd Robinson 2171d9471b
Initialise index in shards 2017-01-05 09:42:48 -07:00
Mark Rushakoff 4a774eb600 Update godoc for the tsdb package 2016-12-30 21:12:37 -08:00
Jonathan A. Sternberg ec57108520 Use proper uber-go/zap import path
It looks like the real import path to the project is go.uber.org/zap
instead of github.com/uber-go/zap since the example in the project
references that path.
2016-12-15 08:54:14 -06:00
Jonathan A. Sternberg 21502a39e8 Switch logging to use structured logging everywhere
The logging library has been switched to use uber-go/zap. While the
logging has been changed to use structured logging, this commit does not
change any of the logging statements to take advantage of the new
structured log or new log levels. Those changes will come in future
commits.
2016-12-14 10:45:15 -06:00
Jason Wilder 27d157763a Merge pull request #7651 from influxdata/jw-shard-last-modified
Expose Shard.LastModified
2016-11-23 10:19:26 -07:00
Jason Wilder e8a28cfbab Expose Shard.LastModified
This returns the LastModified time of the shard.  The LastModified
time is the wall time when a change to the shards state occurred.
It uses the WAL or FileStore to determine the max mod time.
2016-11-23 10:04:07 -07:00
Edd Robinson b83b8df32f Merge pull request #7635 from influxdata/er-msg
Fix incorrect error message
2016-11-23 13:58:33 +00:00
Edd Robinson 9e9719749f Sprinkle some golint 2016-11-17 16:31:38 +00:00
Edd Robinson 28ba8ced74 Fixes #7625 2016-11-17 16:31:36 +00:00
Jason Wilder bf17074f58 Avoid allocation when counting tag keys
A new sorted slice was called by the monitor func every 10s.  The
tag keys don't need to be sorted so this avoid the allocation of the
slice and one during sorting.
2016-11-15 16:13:55 -07:00
Jason Wilder 0b6f5441b9 Add config option to messages when limits exceeded
When a limit is exceeded, we return errors and sometimes log (if appropriate)
that a limit was exceeded.  The messages don't always provide an indication
as to where or how they are configured.

Instead, return the config option (easily searchable for) as well as the limit
currently set and the value that exceeded it when possible.
2016-10-28 14:54:45 -06:00
Jonathan A. Sternberg 3681bc8a43 Filter out series within shards that do not have data for that series
Previously, we would return a full tag set for every shard and the tag
set would include all series that existed in the database index
including series that didn't physically exist within that shard. This
led to the tag sets returned being incredibly huge when we had high
cardinality but sparse data. Since the data was sparse, it was
unexpected that it would cause such a large strain on the system by most
people.

Now we filter out the series ids that are not assigned to the current
shard when computing a tag set for that shard. This lowers the memory
usage for high cardinality sparse data drastically and allows queries on
those to complete successfully.

This does not resolve issues for high cardinality data in every shard
that is also spread out over a long series of time. That situation isn't
nearly as common as the above situation though.
2016-10-20 14:15:34 -05:00
Jason Wilder a5f871d62c Rework monitoring to avoid allocations 2016-10-10 11:42:15 -06:00
Jason Wilder bbecb3f03d Drop points that would execeed limits
This changes the behavior of the max-series-per-database and
max-values-per-tag limits to drop points that would exceed the limits
and allow the remaining points to be written.  Previously, the whole
batch would fail and return and 500 error to the client.

This now will write the allow points and return a `partial write`
error indicating some of the points were dropped, how many were
dropped and one of the problem measureent and tags.
2016-10-10 11:42:15 -06:00
Jason Wilder 8fce6bba48 Add tag value cardinality limit 2016-10-10 11:42:15 -06:00
Joe LeGasse 743946fafb models: Add FieldIterator type
The FieldIterator is used to scan over the fields of a point, providing
information, and delaying parsing/decoding the value until it is needed.
This change uses this new type to avoid the allocation of a map for the
fields which is then thrown away as soon as the points get converted
into columns within the datastore.
2016-10-03 16:30:21 -06:00
Mark Rushakoff 97c2f6f5c1 Add walPath tag to shard stats
Without the WAL path as a tag, the diskBytes field looked like it was
reporting the size of the data directory incorrectly.

Fixes #7382.
2016-09-29 10:19:11 -07:00
Jonathan A. Sternberg 3afdf3cd94 Merge tag 'v1.0.1' 2016-09-27 17:53:33 -05:00
rw 3e0d3be461 Use pre-existing function. 2016-09-26 13:12:10 -07:00
rw bea010b5f3 Fix data race in *tsdb.Shard write path.
Ensure that the Shard's Index is read-locked before calculating the
count of its constituent series.
2016-09-26 12:42:35 -07:00
rw 68c2212aac Shorten name of static-lifetime string var. 2016-09-26 11:26:24 -07:00
rw 02c86ea9db Remove unnecessary string constant. 2016-09-26 11:25:04 -07:00
rw b86885c5cd Remove a few short-lived string allocs.
(*tsdb.Shard).validateSeriesAndFields uses fewer string allocs in some
hot spots.
2016-09-22 17:55:57 -07:00
Jason Wilder 39ade11944 Unload index before closing shard
When deleting a shard, the shard is locked and then removed from the
index.  Removal from the index can be slow if there are a lot of
series.  During this time, the shard is still expected to exist by
the meta store and tsdb store so stats collections, queries and writes
could all be run on this shard while it's locked.  This can cause everything
to lock up until the unindexing completes and the shard can be unlocked.

Fixes #7226
2016-09-22 11:16:45 -06:00
Jason Wilder d06b28992d Unload index before closing shard
When deleting a shard, the shard is locked and then removed from the
index.  Removal from the index can be slow if there are a lot of
series.  During this time, the shard is still expected to exist by
the meta store and tsdb store so stats collections, queries and writes
could all be run on this shard while it's locked.  This can cause everything
to lock up until the unindexing completes and the shard can be unlocked.

Fixes #7226
2016-09-16 12:01:50 -06:00