Commit Graph

129 Commits (5a9a28151be12d2650c00c6ce881ca215de2e176)

Author SHA1 Message Date
Johnny Steenbergen fd289883fe feat(influx): add User-Agent header to the CLI http calls
closes: #18336
2020-06-02 16:09:11 -07:00
Pavel Zavora 7af307d76a chore(pkg/csv2lp): repair doc 2020-05-12 21:32:29 +02:00
Pavel Zavora 9699352e1a chore(pkg/csv2lp): clarify README.md 2020-05-12 21:32:29 +02:00
Pavel Zavora d12d6deae9 chore(pkg/csv2lp): rename CsvToProtocolLines to CsvToLineProtocol 2020-05-12 21:32:29 +02:00
Pavel Zavora 010f23f82f feat(pkg/csv2lp): support quoted new lines in SkipHeaderLinesReader 2020-05-12 21:32:29 +02:00
Pavel Zavora 584ade526d feat(pkg/csv2lp): improve README.md 2020-05-12 21:32:29 +02:00
Pavel Zavora 8dcbd15df2 feat(pkg/csv2lp): add SkipHeaderLinesReader 2020-05-12 21:32:29 +02:00
Pavel Zavora bfba3480fb feat(pkg/csv2lp): add MultiCloser 2020-05-12 21:32:29 +02:00
Pavel Zavora 94da00598a chore: improve doc 2020-05-12 21:32:29 +02:00
Pavel Zavora 8cdde7ac17 chore(pkg/csv2lp): apply code-review comments 2020-05-12 21:32:29 +02:00
Pavel Zavora 6e5aca1d8c chore(pkg/csv2lp): document tests 2020-05-12 21:32:29 +02:00
Pavel Zavora 476acc496a chore(pkg/csv2lp): use snake_case 2020-05-12 21:32:29 +02:00
Pavel Zavora 786f06b453 chore(pkg/csv2lp): add README.md 2020-05-12 21:32:29 +02:00
Pavel Zavora 234eac85eb feat(pkg/csv2lp): add csv to line protocol conversion library 2020-05-12 21:32:29 +02:00
Johnny Steenbergen 35ed5734bb chore(httpc): refactor inputs to eliminate required path
the base address should be enough to make a request. All requests
are now valid without a path provided. This will be true as long
as the address is valid.
2020-05-01 17:47:37 -07:00
Johnny Steenbergen 9495ec1c4e chore(testtp): extend testtp request with SetFormValue 2020-04-16 14:39:07 -07:00
Jonathan A. Sternberg 6e4cf7ffef
refactor: fix imports from go template files (#17615) 2020-04-03 17:40:36 -05:00
Jonathan A. Sternberg 0ae8bebd75
refactor: rewrite imports to include the /v2 suffix for version 2 2020-04-03 12:39:20 -05:00
Edd Robinson 4e9fdf862c feat: buffered io.WriteCloser 2020-03-13 11:00:28 +00:00
Jacob Marble 26ca766459
refactor(tsdb): move series file to its own package (#17224)
* refactor(storage): move type ByTagKey to the only package that uses it

* refactor(tsdb): use types in tsdb/cursors

* refactor(tsdb): remove unused type SeriesIDElems

* refactor(tsdb): inline only use of tsdb.ReadAllSeriesIDIterator

* refactor(tsdb): move series file to its own package

* refactor(storage): remove platform->influxdb aliases
2020-03-12 11:32:52 -07:00
Jacob Marble b91e3f36ab
refactor(hll): remove unused Sketch interface (#17218) 2020-03-12 08:59:05 -07:00
Johnny Steenbergen e650c941c7 fix: remove duplicate content types from httpc requests
2 issues from investigating this error. First is the status check func
did not identify it was a media unsupported issue adn tries to unmarshal
the empty response body. The 2nd, was the double content type headers were
causing an error. Locally this error does not surface, cannot repoduce on
macos, but in cloud it is persistent.

closes: #16819
2020-02-11 11:14:44 -08:00
Chris Goller 0746533b55
feat(pkg/httpc): add cookie based authorization (#16802)
This will begin work to mimic the browser via the client API.

Signed-off-by: Chris Goller <goller@gmail.com>
2020-02-10 12:37:13 -06:00
David McKay 8e51fe0aad fix(CORS): accept OPTIONS preflight request with Origin header 2020-02-06 20:28:29 +01:00
tmgordeeva 61adfe95fe
fix(storage): simple8b passes checkptr (#15512) (#16719)
Cherry-pick of 8f665ec from 1.8.

* fix(simple8b): disable checkptr on batch decodes

* perf: improve performance of simple8b encoding

```
name                      old time/op    new time/op    delta
EncodeAll/1_bit-24          35.8µs ± 2%    35.8µs ± 2%     ~     (p=0.887 n=10+7)
EncodeAll/2_bits-24         35.7µs ± 4%    35.6µs ± 3%     ~     (p=0.684 n=10+10)
EncodeAll/3_bits-24         36.5µs ± 3%    36.4µs ± 4%     ~     (p=0.780 n=10+9)
EncodeAll/4_bits-24         36.4µs ± 3%    37.8µs ± 2%   +3.71%  (p=0.000 n=10+10)
EncodeAll/5_bits-24         38.0µs ± 3%    38.3µs ± 2%     ~     (p=0.280 n=10+10)
EncodeAll/6_bits-24         39.4µs ± 1%    38.5µs ± 2%   -2.43%  (p=0.001 n=10+9)
EncodeAll/7_bits-24         39.6µs ± 3%    38.9µs ± 3%   -1.83%  (p=0.029 n=10+10)
EncodeAll/8_bits-24         40.3µs ± 3%    40.5µs ± 4%     ~     (p=0.278 n=10+9)
EncodeAll/10_bits-24        42.6µs ± 3%    41.9µs ± 2%     ~     (p=0.075 n=10+10)
EncodeAll/12_bits-24        44.2µs ± 2%    42.8µs ± 2%   -3.11%  (p=0.000 n=10+10)
EncodeAll/15_bits-24        46.0µs ± 1%    46.1µs ± 3%     ~     (p=0.447 n=9+10)
EncodeAll/20_bits-24        51.4µs ± 2%    50.0µs ± 2%   -2.86%  (p=0.000 n=10+10)
EncodeAll/30_bits-24        60.6µs ± 1%    58.1µs ± 2%   -4.02%  (p=0.000 n=10+10)
EncodeAll/60_bits-24        90.2µs ± 2%    85.2µs ± 1%   -5.50%  (p=0.000 n=10+10)
EncodeAll/combination-24     782µs ± 3%     762µs ± 5%     ~     (p=0.063 n=10+10)
Encode-24                   3.08µs ± 2%    3.09µs ± 1%     ~     (p=0.656 n=9+8)
Encoder-24                  4.58µs ± 0%    4.40µs ± 1%   -4.04%  (p=0.000 n=8+8)

```

Co-authored-by: Jacob Marble <jacobmarble@gmail.com>
2020-02-04 10:57:43 -08:00
Johnny Steenbergen 63fd1e2e9a feat(pkger): make pkger packages CRD compliant 2020-01-14 13:20:00 -08:00
Johnny Steenbergen fe12fd051e feat(pkger): add jsonnet support for package files 2020-01-13 09:55:08 -08:00
Johnny Steenbergen a6e768dc7c chore: refactor notication endpoints to make a service out of notification endpoints
this is a blocker for anyone who hits the endpoint services internally. They
had to know that they need to also know of hte secret service then do all that
put/delete alongside the operation. This makes that unified inside the store tx.

one other thing this does is make obvious the dependencies that
notification services has. In this case it is the secrets service it
depends on.
2019-12-17 10:49:06 -08:00
Johnny Steenbergen 88468822e2 feat(testttp): extend testttp with new JSON methods and refactors test to use it 2019-12-13 14:07:11 -08:00
Johnny Steenbergen bf86cce1f1 feat(httpc): extend httpc Client with JSON shorthand methods
making most common API calls super easy to get up and running
2019-12-12 07:44:31 -08:00
Johnny Steenbergen 5cc02de988 feat(httpc): add httpc pkg to resuse code for http client type 2019-12-09 12:17:22 -08:00
Stuart Carnie 36b0dc7da3
chore(vendor): Add jsonparser dependency for filter predicates
* Added jsonparser helper package to decode influxdb.ID
2019-11-22 10:34:20 -07:00
Johnny Steenbergen f6dbfec346 feat: import jsteenb2/testttp directly into pkg 2019-11-12 14:14:40 -08:00
Jacob Marble 7f1d2be486
fix(storage): guard against compaction burst throughput limit (#14985) (#15101) 2019-09-11 07:46:33 -07:00
Stuart Carnie 0054562014
feat(gen): Add Copy(tsdb.<type>Array) method to Values
This allows data generators to produce tsdb arrays, which are
useful for testing
2019-08-16 10:33:41 -07:00
Edd Robinson 94dbbdca7b perf(storage): memoize hashmap prom labels
Prior to this, each hashmap operation that was instrumented involved
initialising a map. Now these maps are pre-initialised.

```
⇒  benchstat old.txt new.txt
name                                                     old time/op    new time/op    delta
Index_CreateSeriesListIfNotExist/create_series-8            5.00s ± 3%     5.13s ± 2%   +2.50%  (p=0.033 n=10+7)
Index_CreateSeriesListIfNotExist/already_exist_series-8     557ms ± 3%     530ms ± 6%   -4.85%  (p=0.001 n=10+8)

name                                                     old alloc/op   new alloc/op   delta
Index_CreateSeriesListIfNotExist/create_series-8           2.57GB ± 0%    1.84GB ± 1%  -28.52%  (p=0.000 n=8+10)
Index_CreateSeriesListIfNotExist/already_exist_series-8     678MB ± 0%     308MB ± 0%  -54.55%  (p=0.000 n=10+8)

name                                                     old allocs/op  new allocs/op  delta
Index_CreateSeriesListIfNotExist/create_series-8            28.9M ± 0%     24.5M ± 0%  -15.22%  (p=0.000 n=9+10)
Index_CreateSeriesListIfNotExist/already_exist_series-8     2.23M ± 0%     0.03M ± 0%  -98.51%  (p=0.000 n=10+10)
```
2019-08-13 11:28:51 +01:00
Max U 6bf821f5f6 test(fs_test): update readability of fs API tests 2019-07-25 10:09:52 -07:00
maxunt ca5a599261
Merge branch 'master' into er-rename 2019-07-08 13:42:24 -04:00
Max U 39f51969e9 replaced os.Create calls w API calls to fs.CreateFile, includes unit test 2019-07-08 13:01:42 -04:00
Jeff Wendling 59279837e5 tsi1: partition close deadlock
When a tsi1 partition closes, it waits on the wait group for compactions
and then acquires the lock. Unfortunately, a compaction may start in the
mean time, holding on to some resources. Then, close will attempt to
close those resources while holding the lock. That will block until
the compaction has finished, but it also needs to acquire the lock
in order to finish, leading to deadlock.

One cannot just move the wait group wait into the lock because, once
again, the compaction must acquire the lock before finishing. Compaction
can't finish before acquiring the lock because then it might be operating
on an invalid resource.

This change splits the locks into two: one to protect just against
concurrent Open and Close calls, and one to protect all of the other
state. We then just close the partition, acquire the lock, then free
the resources. Starting a compaction requires acquiring a resource
to the partition itself, so that it can't start one after it has
started closing.

This change also introduces a cancellation channel into a reference
to a resource that is closed when the resource is being closed, allowing
processes that have acquired a reference to clean up quicker if someone
is trying to close the resource.
2019-04-22 09:06:32 -06:00
Todd Persen cd64ec8718 Fix typos in miscellaneous packages 2019-04-17 13:30:22 -07:00
Ben Johnson 307bb6af9c
Improve bulk series file writes. 2019-04-05 14:38:58 -06:00
docmerlin (j. Emrys Landivar) 492d8406aa fix(tasks): make durations visibly show up nicely 2019-03-29 13:50:26 -05:00
Christopher M. Wolff 56e688beee
fix(query): add Check method to query service interfaces (#12866)
Fixes influxdata/idpe#2756.
2019-03-25 20:05:44 -07:00
Stuart Carnie 5b2e4672ac
feedback(gen): Responded to PR feedback
* renamed files to snake_case
* unexport `Series` interface and rename for clarity
2019-03-20 10:44:30 -07:00
Stuart Carnie 8669e94904
feat(influxd): Add the generate subcommand 2019-03-19 21:52:26 -07:00
Stuart Carnie fe8b63c10b
chore(gen): Update the gen package to produce 2.0 series keys 2019-03-19 20:35:02 -07:00
Stuart Carnie 15c8151c52
feat(gen): Add data/gen package from OSS 1.8
This version of the data/gen package adds support for custom schema.
An example of the schema for this version can be found in #11770
2019-03-19 20:35:02 -07:00
docmerlin (j. Emrys Landivar) ea82893abd feat(tasks): remove removable options 2019-03-19 10:24:42 -05:00
Jeff Wendling 55baf9ac89 pkg/lifecycle: resource debug tracking at runtime
Set the INFLUXDB_EXP_RESOURCE_DEBUG environment variable to any
non-empty value to have the process track resource leaks and
acquisitions.
2019-03-05 18:41:37 -07:00
Jeff Wendling 0fae44e219 storage: fix problems with keeping resources alive
This commit adds the pkg/lifecycle.Resource to help manage opening,
closing, and leasing out references to some resource. A resource
cannot be closed until all acquired references have been released.
If the debug_ref tag is enabled, all resource acquisitions keep
track of the stack trace that created them and have a finalizer
associated with them to print on stderr if they are leaked. It also
registers a handler on SIGUSR2 to dump all of the currently live
resources.

Having resources tracked in a uniform way with a data type allows us
to do more sophisticated tracking with the debug_ref tag, as well.
For example, we could panic the process if a resource cannot be
closed within a certain time frame, or attempt to figure out the
DAG of resource ownership dynamically.

This commit also fixes many issues around resources, correctness
during error scenarios, reporting of errors, idempotency of
close, tracking of memory for some data structures, resource leaks
in tests, and out of order dependency closes in tests.
2019-02-28 10:22:01 -07:00
Edd Robinson 9fcf27a25b Add safe rename functions at fs level 2019-02-12 12:41:27 +00:00
Edd Robinson 0858b2570d Rename --> RenameFileWithReplacement for clarity 2019-02-12 12:41:10 +00:00
Edd Robinson bd8a167a3e Rename file package to fs 2019-02-12 11:24:11 +00:00
Mark Rushakoff 1d38d18d79 chore: remove pkg/deep
Looks like the last reference to it was deleted in March 2018
(df7a660fb3).

Prior to that the last use was switched to go-cmp, which we've more
or-less standardized on, at this point.
2019-01-25 08:19:40 -08:00
Mark Rushakoff d73d73c0d4 chore: rename imports from platform to influxdb
I did this with a dumb editor macro, so some comments changed too.

Also rename root package from platform to influxdb.

In interest of minimizing risk, anyone importing the root package has
now aliased it to "platform" so that no changes beyond imports were
necessary in those files.

Lastly, replace the old platform module to local path /dev/null so that
nobody can accidentally reintroduce a platform dependency while
migrating platform code to influxdb.
2019-01-09 20:51:47 -08:00
Stuart Carnie 8f8311a3ae
chore(gen): Add ingen generator data structures to platform for reuse 2018-12-11 17:45:57 -07:00
Edd Robinson 6b63a3def7 Add option to disable sfile metrics 2018-12-10 14:36:28 +00:00
Edd Robinson b015757c06 Ensure all tsi1 metrics support multiple instances 2018-12-07 14:32:34 +00:00
Edd Robinson bff655786f Ensure tsdb metrics properly registered 2018-12-07 14:32:34 +00:00
Edd Robinson 79b108d174 Fix bug with slice reuse 2018-12-07 14:32:34 +00:00
Edd Robinson f9a2f7a017 go fmt 2018-12-07 14:32:34 +00:00
Edd Robinson e0c10227d0 Fix metric issue in series file 2018-12-07 14:32:34 +00:00
Edd Robinson 7960ccc320 Add TSI index metrics 2018-12-07 14:32:34 +00:00
Edd Robinson 55caa0fe54 Add RHH metrics 2018-12-07 14:32:34 +00:00
Edd Robinson d1fe2bc188 Add series file metrics 2018-12-07 14:32:34 +00:00
Ben Johnson 0084d4d824
Remove influxdb dependency. 2018-11-29 11:44:22 -07:00
Stuart Carnie 79f06f6b97
feedback: Add nil comparison tests 2018-11-21 12:01:36 -07:00
Stuart Carnie 8920dab57a
feedback: Changes per PR feedback 2018-11-21 12:01:36 -07:00
Stuart Carnie 7fb015cc7f
chore: Add GroupBy benchmark 2018-11-21 12:01:36 -07:00
Stuart Carnie 26408dc1fa
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err
* add option to Group sort to specify precedence of `nil`
* improve unit tests for GroupResultSet implementation
2018-11-21 12:01:35 -07:00
Stuart Carnie bef0577206 fix(simple8b): Fix incorrect encoding for a run of 119 or 239 1s
If 120th or 240th value is not a 1, k still passes the check in the
switch, causing the last value to be lost. If this value occurs at
the boundary of a block, the max time will be incorrect, resulting in
compaction failing to make forward progress.
2018-11-14 12:14:45 +00:00
Stuart Carnie a339f8f620 feat(encoding): Improve integer and simple8b encoding performance
simple8b EncodeAll improvements should

```
name                     old time/op  new time/op  delta
EncodeAll/1_bit-8        28.5µs ± 1%  28.6µs ± 1%     ~     (p=0.133 n=9+10)
EncodeAll/2_bits-8       28.9µs ± 2%  28.7µs ± 0%     ~     (p=0.068 n=10+8)
EncodeAll/3_bits-8       29.3µs ± 1%  28.8µs ± 0%   -1.70%  (p=0.000 n=10+10)
EncodeAll/4_bits-8       29.6µs ± 1%  29.1µs ± 1%   -1.85%  (p=0.000 n=10+10)
EncodeAll/5_bits-8       30.6µs ± 1%  29.8µs ± 2%   -2.70%  (p=0.000 n=10+10)
EncodeAll/6_bits-8       31.3µs ± 1%  30.0µs ± 1%   -4.08%  (p=0.000 n=9+9)
EncodeAll/7_bits-8       32.6µs ± 1%  30.8µs ± 0%   -5.49%  (p=0.000 n=9+9)
EncodeAll/8_bits-8       33.6µs ± 2%  31.0µs ± 1%   -7.77%  (p=0.000 n=10+9)
EncodeAll/10_bits-8      34.9µs ± 0%  31.9µs ± 2%   -8.55%  (p=0.000 n=9+10)
EncodeAll/12_bits-8      36.8µs ± 1%  32.6µs ± 1%  -11.35%  (p=0.000 n=9+10)
EncodeAll/15_bits-8      39.8µs ± 1%  34.1µs ± 2%  -14.40%  (p=0.000 n=10+10)
EncodeAll/20_bits-8      45.2µs ± 3%  36.2µs ± 1%  -19.97%  (p=0.000 n=10+9)
EncodeAll/30_bits-8      55.0µs ± 0%  40.9µs ± 1%  -25.62%  (p=0.000 n=9+9)
EncodeAll/60_bits-8      86.2µs ± 1%  55.2µs ± 1%  -35.92%  (p=0.000 n=10+10)
EncodeAll/combination-8   582µs ± 2%   502µs ± 1%  -13.80%  (p=0.000 n=9+9)
```

EncodeIntegers:

```
name                             old time/op    new time/op    delta
EncodeIntegers/1000_seq/batch-8    2.04µs ± 0%    1.50µs ± 1%  -26.22%  (p=0.008 n=5+5)
EncodeIntegers/1000_ran/batch-8    8.80µs ± 2%    6.10µs ± 0%  -30.73%  (p=0.008 n=5+5)
EncodeIntegers/1000_dup/batch-8    2.03µs ± 1%    1.50µs ± 1%  -26.04%  (p=0.008 n=5+5)
```

EncodeTimestamps (ran is improved due to simple8b improvements)

```
name                               old time/op    new time/op    delta
EncodeTimestamps/1000_seq/batch-8    2.64µs ± 1%    2.65µs ± 2%     ~     (p=0.310 n=5+5)
EncodeTimestamps/1000_ran/batch-8    64.0µs ± 1%    33.8µs ± 1%  -47.23%  (p=0.008 n=5+5)
EncodeTimestamps/1000_dup/batch-8    9.32µs ± 0%    9.28µs ± 1%     ~     (p=0.087 n=5+5)
```
2018-11-01 18:59:20 +00:00
Stuart Carnie dd83bd230f chore(simple8b): Add more unit tests and benchmarks 2018-11-01 18:59:20 +00:00
Chris Goller 90494a5309 chore(fmt): fix and simplify go formatting 2018-10-08 17:23:54 -05:00
Jeff Wendling 917555e3cd Fix invalid import comment 2018-10-04 12:29:34 -06:00
Edd Robinson ea02772fc5 Pull in more pkg packages 2018-10-01 12:08:37 +01:00
Edd Robinson 074f263e08 Initial import of tsm1.Engine 2018-10-01 12:08:37 +01:00
Edd Robinson fb0db04bc1 Initial import pkg package 2018-10-01 12:03:20 +01:00