Commit Graph

12627 Commits (510d426821c5a8a7aaa3ae6c7e077f5e19d1fc83)

Author SHA1 Message Date
Mark Rushakoff d8dedc5644 ci: use -p=8 in GOFLAGS to reduce linker OOMs
The environment Go runs in thinks there are 32 CPUs, but according to
https://circleci.com/docs/2.0/configuration-reference/#resource_class
that's a known issue, and only 2 CPUs are actually available.

We're still overallocating with -p=8, so if we continue to get OOMs we
may need to reduce that further.
2018-11-02 17:21:33 -07:00
Michael Desa cff0634001
Merge pull request #1195 from influxdata/feat/bolt-cli
Add local flag to influx command
2018-11-02 14:49:28 -04:00
Michael Desa 459f2cbfbb feat(cmd/influx): add local flag for operating directly with boltdb
feat(internal/fs): move influxDir function to internal package in base of repo

feat(cmd/influx): add local flag that operates on local file system

feat(cmd/influx): add local support for org command

feat(cmd/influx): add local support for user command

eat(cmd/influx): add local support for bucket command

feat(cmd/influx): add error for local with query/task/setup cmds

fix(cmd/influx): unshadow error in delete bucket command

fix(cmd/influx): update copy for local flag commands
2018-11-02 14:34:48 -04:00
Michael Desa c70c884fb9
Merge pull request #1258 from influxdata/feat/operation-log
WIP: add operation log for resources
2018-11-02 14:34:12 -04:00
Chris Goller 9d507a220a
Merge pull request #1280 from influxdata/feature/query-plan
refactor(http): preallocate query plans
2018-11-02 13:25:42 -05:00
Michael Desa 7227bf8fa2 feat(platform): add operation log for resources
feat(platform): add dashboard operation log interface

feat(bolt): add dashboard operation log to bolt client

feat(platform): add links to dashboard load response

fix(http): update links in json response of dashboards

feat(bolt): add operation logs for each resource

feat(http): add routes for operation logs

feat(cmd/influxd): set operation logs on http api backend
2018-11-02 14:21:14 -04:00
Adam 612005cad1
add output to to function (#1282)
* added (unused) stats struct, and piped written data to output table

* updated to tests since to returns data now

* changes based on review
2018-11-02 13:40:25 -04:00
kelwang 3fed14dabc
Merge pull request #943 from influxdata/feat/inmem_onboarding_basic_auth
add inmem service for basic auth, http onboarding testing
2018-11-02 13:22:00 -04:00
Chris Goller 84af710e42 refactor(http): preallocate query plans 2018-11-02 12:03:45 -05:00
Kelvin Wang 63da5d1e9f (feat/testing) add onboarding and basic auth service 2018-11-02 12:20:45 -04:00
kelwang 3dec916178
Merge pull request #1277 from influxdata/fix/400_for_unknown_error_code
fix the error code for 400
2018-11-02 12:16:17 -04:00
Kelvin Wang cb4e095463 fix the error code for 400 2018-11-02 11:28:39 -04:00
Chris Goller 3e54ef9f53
Merge pull request #1275 from influxdata/feature/query-plan
feat(http): add plan endpoint to analyze queries
2018-11-02 09:25:51 -05:00
Chris Goller 0d7b77d3dd feat(http): add query plan validation 2018-11-02 09:11:57 -05:00
kelwang 8f52f9e1dc
Merge pull request #1273 from influxdata/fix/http_error
(fix/http): make error compatible with new and old lib
2018-11-02 10:03:21 -04:00
Chris Goller 7ca5187dc6 feat(http): add plan endpoint to analyze queries 2018-11-01 21:00:37 -05:00
kelwang d022a26250
Merge pull request #1252 from influxdata/link_reset_user_password
fix user response
2018-11-01 21:53:15 -04:00
Kelvin Wang 5378fe877a (fix/http): make error compatible with new and old lib 2018-11-01 20:28:40 -04:00
Kelvin Wang 5b7b123d1e fix user response 2018-11-01 20:12:04 -04:00
Jorge Landivar 34ab06634f
Merge pull request #1266 from influxdata/opentracing
open tracing context handling for tasks and for HTTP endpoints for tasks and query
2018-11-01 16:33:19 -05:00
docmerlin (j. Emrys Landivar) bcbd7b5c24 context handling for tasks and for HTTP endpoints for tasks and query 2018-11-01 16:29:52 -05:00
Stuart Carnie 56510ba0dc
Merge pull request #1262 from influxdata/sgc-fix-race
Fix race condition as `Close` was called by multiple goroutines
2018-11-01 14:22:32 -07:00
Nathaniel Cook 95b2bfb15b Merged pull request #1269 from influxdata/nc-flux-0.7.0-rc1
chore: Update to Flux 0.7.0-rc1
2018-11-01 14:57:32 -06:00
Nathaniel Cook 89e6c6adbd chore: Update to Flux 0.7.0-rc1 2018-11-01 14:32:56 -06:00
Mark Rushakoff 1d42012c8d
Merge pull request #1267 from influxdata/chore/megacheck
Re-enable megacheck
2018-11-01 13:29:04 -07:00
Mark Rushakoff 68c0ad29e2 ci: add megacheck to tools.go 2018-11-01 13:25:22 -07:00
Mark Rushakoff b37279c50e ci: remove 2/3 disabled exit-non-zero flags for megacheck
The one remaining failing static check is in a pigeon-generated promql
file. It looks like megacheck is supposed to ignore generated files by
default, so it's unclear at this time if this is a bug in megacheck, or
if fixing pigeon is the only path forward.
2018-11-01 12:54:46 -07:00
Mark Rushakoff ff2998dcf8 chore(chronograf): fix megacheck errors 2018-11-01 12:54:46 -07:00
Mark Rushakoff dae3dc8a7e chore(bolt,http,mock,query,testing): fix megacheck errors 2018-11-01 12:54:46 -07:00
Mark Rushakoff 4c129e0044 chore(context,gather,inmem,kit): fix megacheck errors 2018-11-01 12:54:46 -07:00
Mark Rushakoff 89813b8aa4 chore(cmd): fix megacheck errors 2018-11-01 12:54:46 -07:00
Mark Rushakoff 985c260af7 chore(storage,tsdb): fix megacheck errors 2018-11-01 12:54:46 -07:00
Mark Rushakoff fe84ff7357 chore(task): fix megacheck errors 2018-11-01 12:46:33 -07:00
Mark Rushakoff 9acc251ab0 ci: re-enable megacheck 2018-11-01 12:46:33 -07:00
Edd Robinson f998c3369d
Merge pull request #1104 from influxdata/er-batch-decoders
Batch oriented encoders
2018-11-01 19:20:12 +00:00
Stuart Carnie a0300064df feat(tsm1): Improve performance of Gorilla float block decoding
```
name                        old time/op   new time/op    delta
FloatArrayDecodeAll/1-8      45.9ns ± 1%    13.8ns ± 1%   -70.00%  (p=0.000 n=9+9)
FloatArrayDecodeAll/55-8      686ns ± 0%     232ns ± 1%   -66.10%  (p=0.000 n=9+8)
FloatArrayDecodeAll/550-8    5.78µs ± 0%    2.22µs ± 1%   -61.61%  (p=0.000 n=9+9)
FloatArrayDecodeAll/1000-8   10.2µs ± 2%     4.0µs ± 5%   -60.47%  (p=0.000 n=10+10)

name                        old speed     new speed      delta
FloatArrayDecodeAll/1-8     414MB/s ± 1%  1383MB/s ± 1%  +233.76%  (p=0.000 n=9+9)
FloatArrayDecodeAll/55-8    144MB/s ± 0%   424MB/s ± 1%  +194.19%  (p=0.000 n=9+9)
FloatArrayDecodeAll/550-8   133MB/s ± 0%   346MB/s ± 1%  +160.09%  (p=0.000 n=9+10)
FloatArrayDecodeAll/1000-8  135MB/s ± 2%   340MB/s ± 5%  +153.03%  (p=0.000 n=10+10)
```
2018-11-01 18:59:20 +00:00
Edd Robinson 353df7edca Fix imports 2018-11-01 18:59:20 +00:00
Edd Robinson e282d012c8 Address PR feedback 2018-11-01 18:59:20 +00:00
Stuart Carnie c21336af0a fix(encoding): Improve array string encoding perf a little more
Encode the compressed data at the start internal buffer. This ensures
the returned slice maintains the entire capacity and is available for
subsequent use.

When we pool / reuse string buffers, this will help considerably.

Improvements over previous commit:

```
name                        old time/op    new time/op    delta
EncodeStrings/10/batch-8       542ns ± 1%     355ns ± 2%   -34.53%  (p=0.008 n=5+5)
EncodeStrings/100/batch-8     5.29µs ± 1%    3.58µs ± 2%   -32.20%  (p=0.008 n=5+5)
EncodeStrings/1000/batch-8    48.6µs ± 0%    36.2µs ± 2%   -25.40%  (p=0.008 n=5+5)

name                        old alloc/op   new alloc/op   delta
EncodeStrings/10/batch-8        704B ± 0%        0B       -100.00%  (p=0.008 n=5+5)
EncodeStrings/100/batch-8     9.47kB ± 0%    0.00kB       -100.00%  (p=0.008 n=5+5)
EncodeStrings/1000/batch-8    90.1kB ± 0%     0.0kB       -100.00%  (p=0.008 n=5+5)

name                        old allocs/op  new allocs/op  delta
EncodeStrings/10/batch-8        0.00           0.00           ~     (all equal)
EncodeStrings/100/batch-8       1.00 ± 0%      0.00       -100.00%  (p=0.008 n=5+5)
EncodeStrings/1000/batch-8      1.00 ± 0%      0.00       -100.00%  (p=0.008 n=5+5)
```
2018-11-01 18:59:20 +00:00
Stuart Carnie 296d39059a fix(encoding): Improve simple8b another 6%; fix inconsequential bug
simple8b encodes deltas[1:], thus deltas[0] >= simple8b.MaxValue is
invalid.

Also changed loop calculating deltas, RLE and max to be similar to
batch timestamp, for greater consistency.

Improvements over previous commit:

```
name                             old time/op    new time/op    delta
name                             old time/op    new time/op    delta
EncodeIntegers/1000_seq/batch-8    1.50µs ± 1%    1.48µs ± 1%  -1.40%  (p=0.008 n=5+5)
EncodeIntegers/1000_ran/batch-8    6.10µs ± 0%    5.69µs ± 2%  -6.58%  (p=0.008 n=5+5)
EncodeIntegers/1000_dup/batch-8    1.50µs ± 1%    1.49µs ± 0%  -1.21%  (p=0.008 n=5+5)
```

Improvements overall:

```
name                             old time/op    new time/op    delta
EncodeIntegers/1000_seq/batch-8    2.04µs ± 0%    1.48µs ± 1%  -27.25%  (p=0.008 n=5+5)
EncodeIntegers/1000_ran/batch-8    8.80µs ± 2%    5.69µs ± 2%  -35.29%  (p=0.008 n=5+5)
EncodeIntegers/1000_dup/batch-8    2.03µs ± 1%    1.49µs ± 0%  -26.93%  (p=0.008 n=5+5)
```
2018-11-01 18:59:20 +00:00
Stuart Carnie 9fa01f7115 feat(encoding): Improve timestamp encoding
Timestamp improvements prior to any improvements to simple8b

```
name                               old time/op    new time/op    delta
name                               old time/op    new time/op    delta
EncodeTimestamps/1000_seq/batch-8    2.64µs ± 1%    1.36µs ± 1%  -48.25%  (p=0.008 n=5+5)
EncodeTimestamps/1000_ran/batch-8    64.0µs ± 1%    32.2µs ± 1%  -49.64%  (p=0.008 n=5+5)
EncodeTimestamps/1000_dup/batch-8    9.32µs ± 0%    1.30µs ± 1%  -86.06%  (p=0.008 n=5+5)
```
2018-11-01 18:59:20 +00:00
Stuart Carnie a339f8f620 feat(encoding): Improve integer and simple8b encoding performance
simple8b EncodeAll improvements should

```
name                     old time/op  new time/op  delta
EncodeAll/1_bit-8        28.5µs ± 1%  28.6µs ± 1%     ~     (p=0.133 n=9+10)
EncodeAll/2_bits-8       28.9µs ± 2%  28.7µs ± 0%     ~     (p=0.068 n=10+8)
EncodeAll/3_bits-8       29.3µs ± 1%  28.8µs ± 0%   -1.70%  (p=0.000 n=10+10)
EncodeAll/4_bits-8       29.6µs ± 1%  29.1µs ± 1%   -1.85%  (p=0.000 n=10+10)
EncodeAll/5_bits-8       30.6µs ± 1%  29.8µs ± 2%   -2.70%  (p=0.000 n=10+10)
EncodeAll/6_bits-8       31.3µs ± 1%  30.0µs ± 1%   -4.08%  (p=0.000 n=9+9)
EncodeAll/7_bits-8       32.6µs ± 1%  30.8µs ± 0%   -5.49%  (p=0.000 n=9+9)
EncodeAll/8_bits-8       33.6µs ± 2%  31.0µs ± 1%   -7.77%  (p=0.000 n=10+9)
EncodeAll/10_bits-8      34.9µs ± 0%  31.9µs ± 2%   -8.55%  (p=0.000 n=9+10)
EncodeAll/12_bits-8      36.8µs ± 1%  32.6µs ± 1%  -11.35%  (p=0.000 n=9+10)
EncodeAll/15_bits-8      39.8µs ± 1%  34.1µs ± 2%  -14.40%  (p=0.000 n=10+10)
EncodeAll/20_bits-8      45.2µs ± 3%  36.2µs ± 1%  -19.97%  (p=0.000 n=10+9)
EncodeAll/30_bits-8      55.0µs ± 0%  40.9µs ± 1%  -25.62%  (p=0.000 n=9+9)
EncodeAll/60_bits-8      86.2µs ± 1%  55.2µs ± 1%  -35.92%  (p=0.000 n=10+10)
EncodeAll/combination-8   582µs ± 2%   502µs ± 1%  -13.80%  (p=0.000 n=9+9)
```

EncodeIntegers:

```
name                             old time/op    new time/op    delta
EncodeIntegers/1000_seq/batch-8    2.04µs ± 0%    1.50µs ± 1%  -26.22%  (p=0.008 n=5+5)
EncodeIntegers/1000_ran/batch-8    8.80µs ± 2%    6.10µs ± 0%  -30.73%  (p=0.008 n=5+5)
EncodeIntegers/1000_dup/batch-8    2.03µs ± 1%    1.50µs ± 1%  -26.04%  (p=0.008 n=5+5)
```

EncodeTimestamps (ran is improved due to simple8b improvements)

```
name                               old time/op    new time/op    delta
EncodeTimestamps/1000_seq/batch-8    2.64µs ± 1%    2.65µs ± 2%     ~     (p=0.310 n=5+5)
EncodeTimestamps/1000_ran/batch-8    64.0µs ± 1%    33.8µs ± 1%  -47.23%  (p=0.008 n=5+5)
EncodeTimestamps/1000_dup/batch-8    9.32µs ± 0%    9.28µs ± 1%     ~     (p=0.087 n=5+5)
```
2018-11-01 18:59:20 +00:00
Stuart Carnie dd83bd230f chore(simple8b): Add more unit tests and benchmarks 2018-11-01 18:59:20 +00:00
Edd Robinson 5e7b2cb273 Fix index bug in float encoder 2018-11-01 18:59:20 +00:00
Edd Robinson 80c953b774 Add TSM batch key iterator
The batch focussed TSM key iterator iterates TSM blocks, decoding and
merging blocks where appropriate using the the batch focussed
approaches.
2018-11-01 18:59:20 +00:00
Edd Robinson 5074b834cd Add batch block encoders 2018-11-01 18:59:19 +00:00
Edd Robinson ab68204683 Batch oriented unsigned encoder 2018-11-01 18:59:19 +00:00
Edd Robinson aeeef803c0 Batch oriented boolean encoders
This commit adds a tsm1 function for encoding a batch of booleans into a
provided buffer.

The following benchmarks compare the performance of the existing
iterator based encoders, and the new batch oriented encoders using
randomly generated sets of booleans.
2018-11-01 18:59:19 +00:00
Jeff Wendling 5376530392 Improvements to batch float encoder
- Inlined the closure to avoid a function call.
- Changed append(b, make([]byte, 8)...) to inline the make call.
- Check for NaN once at the end assuming NaN is infrequent.

New performance delta comparing the current iterators to the new batch
function:

name                   old time/op    new time/op    delta
EncodeFloats/10_seq      1.32µs ± 2%    0.17µs ± 2%  -87.39%  (p=0.000 n=10+10)
EncodeFloats/10_ran      2.09µs ± 1%    0.15µs ± 0%  -92.97%  (p=0.000 n=10+9)
EncodeFloats/100_seq     8.37µs ± 2%    1.28µs ± 2%  -84.74%  (p=0.000 n=10+10)
EncodeFloats/100_ran     19.1µs ± 1%     1.3µs ± 1%  -93.08%  (p=0.000 n=9+9)
EncodeFloats/1000_seq    60.4µs ± 1%    12.6µs ± 0%  -79.13%  (p=0.000 n=9+7)
EncodeFloats/1000_ran     212µs ± 1%      12µs ± 1%  -94.53%  (p=0.000 n=9+8)

name                   old alloc/op   new alloc/op   delta
EncodeFloats/10_seq       0.00B          0.00B          ~     (all equal)
EncodeFloats/10_ran       0.00B          0.00B          ~     (all equal)
EncodeFloats/100_seq      0.00B          0.00B          ~     (all equal)
EncodeFloats/100_ran      0.00B          0.00B          ~     (all equal)
EncodeFloats/1000_seq     0.00B          0.00B          ~     (all equal)
EncodeFloats/1000_ran     0.00B          0.00B          ~     (all equal)

name                   old allocs/op  new allocs/op  delta
EncodeFloats/10_seq        0.00           0.00          ~     (all equal)
EncodeFloats/10_ran        0.00           0.00          ~     (all equal)
EncodeFloats/100_seq       0.00           0.00          ~     (all equal)
EncodeFloats/100_ran       0.00           0.00          ~     (all equal)
EncodeFloats/1000_seq      0.00           0.00          ~     (all equal)
EncodeFloats/1000_ran      0.00           0.00          ~     (all equal)
2018-11-01 18:59:19 +00:00
Edd Robinson d8b5f9d432 Batch oriented string encoders
This commit adds a tsm1 function for encoding a batch of strings into a
provided buffer. The new function also shares the buffer between the
input data and the snappy encoded output, reducing allocations.

The following benchmarks compare the performance of the existing
iterator based encoders, and the new batch oriented encoders using
randomly generated strings.

name                old time/op    new time/op    delta
EncodeStrings/10      2.14µs ± 4%    1.42µs ± 4%   -33.56%  (p=0.000 n=10+10)
EncodeStrings/100     12.7µs ± 3%    10.9µs ± 2%   -14.46%  (p=0.000 n=10+10)
EncodeStrings/1000     132µs ± 2%     114µs ± 2%   -13.88%  (p=0.000 n=10+9)

name                old alloc/op   new alloc/op   delta
EncodeStrings/10        657B ± 0%      704B ± 0%    +7.15%  (p=0.000 n=10+10)
EncodeStrings/100     6.14kB ± 0%    9.47kB ± 0%   +54.14%  (p=0.000 n=10+10)
EncodeStrings/1000    61.4kB ± 0%    90.1kB ± 0%   +46.66%  (p=0.000 n=10+10)

name                old allocs/op  new allocs/op  delta
EncodeStrings/10        3.00 ± 0%      0.00       -100.00%  (p=0.000 n=10+10)
EncodeStrings/100       3.00 ± 0%      1.00 ± 0%   -66.67%  (p=0.000 n=10+10)
EncodeStrings/1000      3.00 ± 0%      1.00 ± 0%   -66.67%  (p=0.000 n=10+10)
2018-11-01 18:59:19 +00:00