Commit Graph

12783 Commits (3034d3fb5481ed58fa0306693be46a07420a66bd)

Author SHA1 Message Date
Jason Wilder a1d0b52897 Allow lower priority compactions to use excess capacity
If there is a backlog of level 3 and 4 compacitons, but few level 1
and 2 compactions, allow them to use some excess capacity.
2017-10-04 08:11:44 -06:00
Mark Rushakoff bb6cb80d45 Update changelog 2017-10-03 16:43:26 -07:00
Mark Rushakoff f8e85e6df5 Include man pages in releng packaging
The *.1.gz files are included into the source tarball, and the .deb and
.rpm files will now install the man pages when the package is installed.

Fixes #8908.
2017-10-03 16:39:30 -07:00
Mark Rushakoff 6160e369b1 Merge pull request #8909 from influxdata/mr-service-var
Use correct variable style in influxdb.service
2017-10-03 12:59:17 -07:00
Jonathan A. Sternberg 920b2bee2e Merge pull request #7949 from lfarkas/master
always force the file inside the tarball owned by root
2017-10-03 14:37:06 -05:00
Mark Rushakoff 0aaede8421 Update CHANGELOG 2017-10-03 11:28:16 -07:00
Mark Rushakoff c2841c8342 Use correct variable style in influxdb.service
From the systemd.service docs:

Use "${FOO}" as part of a word, or as a word of its own, on the command
line, in which case it will be replaced by the value of the environment
variable including all whitespace it contains, resulting in a single
argument. Use "$FOO" as a separate word on the command line, in which
case it will be replaced by the value of the environment variable split
at whitespace, resulting in zero or more arguments.
2017-10-03 11:15:22 -07:00
Edd Robinson c80c3636dd Merge pull request #8904 from influxdata/er-changelog
Update 1.3.3 release date
2017-10-03 18:43:56 +01:00
Mark Rushakoff 276a26656e Merge pull request #8903 from influxdata/mr-systemd-detection
Use `command` instead of `which`
2017-10-03 10:11:12 -07:00
Mark Rushakoff 7525d331d1 Merge branch 'master' into mr-systemd-detection 2017-10-03 09:54:45 -07:00
Jason Wilder a43e652985 Update changelog 2017-10-03 10:49:46 -06:00
Jason Wilder f2a681c4cf Unconditionally remove file when calling Remove 2017-10-03 10:49:17 -06:00
Jason Wilder 0c0505881f Remove multiple file skipping for full compaction planning
This check doesn't make sense for high cardinality data as the files
typically get big and sparse very quickly.  This causes a lot of extra
disk space to be used which is taken up by large indexes and sparse
data.
2017-10-03 10:48:14 -06:00
Jason Wilder 90df803802 Prevent infinite scheduling loop
One shard might be able to run a compaction, but could fail to
limits being hit.  This loop would continue indefinitely as the
same task would continue to be rescheduled.
2017-10-03 10:48:14 -06:00
Jason Wilder 4ff4ba0841 Use first file in generation for level
With higher cardinality or larger series keys, the files can roll
over early which causes them to take longer to be compacted by higher
levels.  This causes larger disk usage and higher numbers of tsm files
at times.
2017-10-03 10:48:14 -06:00
Jason Wilder 71071ed67a Add compaction backlog stat
This gives an indication as to whether compactions are backed up
or not.
2017-10-03 10:48:14 -06:00
Jason Wilder 16ece490ef Reduce allocation in tsmKeyIterator.Next
The chunked slice is unnecessary and we can re-use k.blocks throughout
the compaction.
2017-10-03 10:48:14 -06:00
Jason Wilder 2c5006fccc Rework snapshotting concurrency
This switches the thresholds that are used for writing snapshots
concurrently.  This scales better than the prior model.
2017-10-03 10:48:14 -06:00
Jason Wilder 3af9c7df37 Remove a defer allocation
Shows up under high cardinality compactions.
2017-10-03 10:48:14 -06:00
Jason Wilder 70817350b7 Ensure temp index files are cleaned up on error 2017-10-03 10:48:14 -06:00
Jason Wilder a5afaf7499 Fix cache mem size not including key size 2017-10-03 10:48:14 -06:00
Jason Wilder ae821f4e2d Rework compaction scheduling
This changes the compaction scheduling to better utilize the available
cores that are free.  Previously, a level was planned in its own goroutine
and would kick off a number of compactions groups.  The problem with this
model was that if there were 4 groups, and 3 completed quickly, the planning
would be blocked for that level until the last group finished.  If the compactions
at the prior level are running more quickly, a large backlog could accumlate.

This now moves the planning to a single goroutine that plans each level in
succession and starts as many groups as it can.  When one group finishes,
the planning will start the next group for the level.
2017-10-03 10:48:13 -06:00
Jason Wilder f668b0cc3f Only use O_SYNC for tsm file writing
Doing this for the WAL reduces throughput quite a bit.
2017-10-03 10:48:13 -06:00
Jason Wilder 1610ae5727 Don't return tsm files part of a compaction plan 2017-10-03 10:48:13 -06:00
Joe LeGasse 1525069213 Merge pull request #8892 from influxdata/jl-tag-values
auth: add series auth to 'show tag values'
2017-10-03 08:47:39 -04:00
Edd Robinson defb81413e Merge branch 'master' into er-changelog 2017-10-03 12:20:49 +01:00
Jonathan A. Sternberg 1566deb320 Merge pull request #8902 from influxdata/js-uint-binary-expr
Support unsigned binary math in fields
2017-10-02 20:32:14 -05:00
Jonathan A. Sternberg 79092610c8 Support unsigned binary math in fields
Field math works similar to condition evaluation, but not the exact same
because we have more information to work with in field expressions than
we do in conditional math because fields retain the information about
their source while conditions do not.

The main difference is that you cannot add an unsigned literal to the
output of an integer iterator while you can inside of a condition. You
can perform math on a positive integer literal to an unsigned iterator.
Inside of the condition, we aren't sure if an integer is because of a
literal or because of an iterator so we can't make that distinction.
2017-10-02 17:06:49 -05:00
Jonathan A. Sternberg cd84509313 Merge pull request #8901 from influxdata/revert-8840-bugfix/issues/8813
Revert "Convert duration to string without dot and zeros"
2017-10-02 16:55:32 -05:00
Jonathan A. Sternberg 06994a6585 Revert "Convert duration to string without dot and zeros" 2017-10-02 16:31:03 -05:00
Mark Rushakoff 4d32c157a9 Update changelog 2017-10-02 11:49:16 -07:00
Mark Rushakoff e623b1b16b Use `command` instead of `which`
Some systems don't have `which` installed.

Also fix some whitespace issues.

Fixes #8900.
2017-10-02 11:49:08 -07:00
Lyon Hill 9cf517bcdf Merge pull request #8896 from influxdata/lh-batcher-leak
Fix a minor memory leak when batching points for some services.
2017-10-02 12:26:58 -06:00
Lyon Hill 7e5fd14e8a add in some optimization 2017-10-02 12:02:38 -06:00
Lyon Hill a6cbce0d3e fix issues brought up by joe 2017-10-02 11:41:03 -06:00
Lyon Hill 38dc837910 Fix a minor memory leak when batching points for some services.
fixes #8895
2017-10-02 11:26:25 -06:00
Jonathan A. Sternberg 1a920c5c42 Merge pull request #8897 from influxdata/js-msgpack-formatter
Add message pack formatter
2017-10-02 11:31:35 -05:00
Andrew Hare 8328bc1e04 Merge pull request #8828 from influxdata/jl-show-measurements
auth: Fix permissions for SHOW MEASUREMENTS
2017-09-29 14:13:26 -06:00
Jonathan A. Sternberg a0df2fe504 Add message pack formatter 2017-09-29 14:58:38 -05:00
Andrew Hare 4d6672fa66 Merge with master 2017-09-29 11:07:40 -06:00
Jonathan A. Sternberg 9db4439984 Merge pull request #8840 from liketic/bugfix/issues/8813
Convert duration to string without dot and zeros
2017-09-29 11:24:40 -05:00
liketic 986a0e18da Convert duration to string without dot and zeros 2017-09-28 11:56:25 -05:00
Jonathan A. Sternberg 7d1c32fb96 Merge pull request #8763 from influxdata/js-8762-document-subquery-ordering-change
Document that query ordering must be consistent in subqueries
2017-09-28 11:15:22 -05:00
Mark Rushakoff 8c70cfd74b Merge pull request #8808 from liketic/feature/issues/8590
Influx CLI: Make cli case insensitive
2017-09-28 09:13:19 -07:00
Jonathan A. Sternberg f0acb988d6 Merge pull request #8884 from influxdata/js-uint-eval
Implement uint64 logic into eval
2017-09-28 10:26:11 -05:00
Joe LeGasse 1443b22379 auth: add series auth to 'show tag values' 2017-09-27 20:01:18 -04:00
Jonathan A. Sternberg d474a0ed9c Handle UnsignedLiteral in the Reduce operations 2017-09-27 15:10:30 -05:00
Jonathan A. Sternberg f0fb0174b6 Implement uint64 logic into eval
Any operation between an int64 and uint64 results in the type becoming a
uint64. This is because the most common will usually be an unsigned
cursor being modified by an int literal. Even in cases where we were to
add an unsigned literal to an integer iterator, that would just result
in it being recast back and overflow.
2017-09-27 15:10:16 -05:00
Edd Robinson e0cba4477c Merge pull request #8885 from influxdata/er-entry-race
Fix race on Cache entry
2017-09-27 18:42:45 +01:00
Edd Robinson 152b97bfc3 Merge pull request #8743 from influxdata/er-8742-go-bits
Prepare for Go 1.9 math/bits package.
2017-09-27 18:13:25 +01:00