Commit Graph

972 Commits (d977c0ac2494a59d72f41dc277771a3d297b8e98)

Author SHA1 Message Date
Stuart Carnie d977c0ac24 fix(tsdb): Fix existing Prometheus tests based on batch cursors 2018-07-16 08:55:37 -07:00
Stuart Carnie 497fc42779 pr(tsdb): Feedback items from megacheck
* batch cursors and cursorIterator will be removed in a follow up
  PR using Arrow array data structures
2018-07-16 08:55:37 -07:00
Stuart Carnie 910d0fe5e6 feat(tsm1): ArrayCursor interfaces and implementations
Array cursors are enabled for storage RPC calls

tsm1:

* Implemented cursors that utilize Array decoders

storage:

* Abstractions to easily switch to Array cursors
2018-07-16 08:55:37 -07:00
Gunnar e9f0dc48ca
Merge pull request #10051 from influxdata/ga-config
Update example config with UDP precision option
2018-07-10 08:16:28 -07:00
Gunnar Aasen 8870512e6b Update example config with UDP precision option
Also add a test for precision in the UDP configuration.
2018-07-09 10:07:05 -07:00
Jeff Wendling 07e5465cb8 httpd: fix flaky test in timeout handler
there were two problems with this code:

1. the send on pending did not imply that the handler was running
2. there was a race starting the handler with timing out

1 is fixed by sending to a begin channel inside the handler. it is
then guaranteed that the timeout handler code has been entered.

2 is fixed by attempting to acquire the semaphore channel once before
checking the timeout channel. in this way, if there is capacity, which
in this test there is known to be, it is guaranteed to be taken. if
we check with the timer at the same time and the timer has already
fired, there is a pseudorandom chance the timer will be taken even
if there is capacity.
2018-07-05 12:09:09 -06:00
Jonathan A. Sternberg 88b81941ac Modify the storage service to expose a grpc interface instead of yarpc 2018-06-28 14:03:09 -05:00
Edd Robinson 8fd00853e7 Ensure read service regexes get optimised 2018-06-22 19:32:17 +01:00
Jonathan A. Sternberg 87d2469877
Merge pull request #9964 from influxdata/js-enable-storage-service
Enable the storage service by default
2018-06-13 16:28:27 -05:00
Edd Robinson 3cb9e13d58 Address PR feedback 2018-06-13 17:41:50 +01:00
Jonathan A. Sternberg 17ca220f33 Enable the storage service by default 2018-06-13 10:56:50 -05:00
Edd Robinson 8a78e64868 Make zero results work properly 2018-06-12 23:49:04 +01:00
Edd Robinson 28b6df7afb Ensure remote read can handle no data in time 2018-06-12 23:10:18 +01:00
Edd Robinson 522e509709 Add further tests 2018-06-12 15:54:18 +01:00
Edd Robinson 524f400836 Make testing of handler easier 2018-06-12 15:54:18 +01:00
Edd Robinson 806464d9e7 Add storage package mocks 2018-06-12 15:54:18 +01:00
Edd Robinson 40fa4eddc0 Fix overflow 2018-06-12 15:54:18 +01:00
Paul Dix 4f7b93342c Update Prometheus read/write to use new storage query layer.
* Update Prometheus remote write to use metric name as measurement name and value as the field name.
* Update Prometheus remote read to use the storage.Read method to bypass the InfluxQL query engine.
2018-06-12 15:54:18 +01:00
Jeff Wendling 5ec7b901bb
Merge pull request #9847 from influxdata/jmw-docker-on-windows
fix(tsdb): attempt to work on docker on windows
2018-06-01 15:16:51 -06:00
Jeff Wendling e6aec771b0 fix(tsdb): attempt to work on docker on windows
multiple users have attempted to run influxdb in a docker container
with a windows host and a volume mounted from windows. that causes
problems because it apparently uses samba/cifs which does not
support fsync on directories. this patchset will, if it receives an EINVAL
on directory fsync, as is what appears to happen on samba/cifs, then it
will ignore it. this should help.

fixes #9833.
fixes #9630.
2018-06-01 14:57:18 -06:00
Stuart Carnie e4b3204328 fix(services/storage): Don't serialize empty group frames 2018-06-01 13:28:34 -07:00
Stuart Carnie 4b7a4868a9 fix(services/storage): Add nil checks to prevent panics
Fixes #9925
2018-05-31 14:22:19 -07:00
Stuart Carnie d42abde836 refactor(services/storage): Enhanced group support for Read RPC 2018-05-30 11:32:20 -07:00
Ben Johnson 8b44e3142c
Add optional pprof http endpoint immediately on startup.
This commit adds `debug-pprof-enabled` which will start the default
`net/http/pprof` endpoint and bind against `localhost:6060`. This
will help to debug startup performance issues.
2018-05-23 14:32:15 -06:00
Ben Johnson 8a74c6759f
Add http write throttling.
This commit adds throttling to the HTTP write endpoints based on
queue depth and, optionally, timeout. Two queues exist: `enqueued`
and `current`. The `current` queue is the number of concurrent
requests that can be processed. The `enqueued` queue limits the
maximum number of requests that can be waiting to be processed.

If the timeout is exceeded or the `enqueued` queue is full then
a `"503 Service unavailable"` code is returned and the error is
logged.

By default these options are turned off.
2018-05-21 13:08:24 -06:00
Jeff Wendling e89438f7c2 fix flaky subscriber tests
Fixes #9554.
2018-04-24 12:34:45 -06:00
Jonathan A. Sternberg a7e1da5f86 Add suppress-write-log option to disable the write log when the log is enabled 2018-04-23 12:45:48 -05:00
Stuart Carnie a8692a9e24 services/meta: improve readability of Contains function, add unit tests 2018-04-19 18:05:55 -07:00
Jacob Marble 232be14aef respect rp parameter in /query 2018-04-19 08:31:43 -07:00
Jacob Marble 321ae4ff04
update CircleCI config to 2.0 syntax (#9711)
* enable flaky test, see if CircleCI fails

* Use CircleCI 2.0 with docker layer caching

* update CONTRIBUTING
2018-04-16 12:00:44 -07:00
Jonathan A. Sternberg 8334693b47 Properly track the response bytes written for queries in all format types
The number of bytes written for CSV responses and probably MessagePack
responses was incorrect.
2018-04-09 12:42:58 -05:00
Jonathan A. Sternberg 243ed2ea5e Avoid a panic when using show diagnostics with text/csv
If the columns change between series, it will now act as if it was a new
statement id and reprint the headers. This only happens with show
diagnostics at the moment and we shouldn't add this functionality
anywhere else anyway.
2018-04-09 09:09:42 -05:00
Jonathan A. Sternberg 1b738d3991 Allow customizing the unix socket group and permissions created by the server 2018-04-05 14:40:12 -05:00
Ben Johnson 1fe9abd66f
Delete deleted shards in retention service. 2018-03-28 10:44:14 -06:00
Stuart Carnie ef2ba80ce2 don't overwrite `_measurement` for multi-tenant reads 2018-03-23 13:59:42 -07:00
Stuart Carnie 813cb1a2f6 to var () or not to var (), that is the question 2018-03-23 12:26:55 -07:00
Stuart Carnie ee3e2ad67f rename Tenant -> OrgID 2018-03-23 12:26:55 -07:00
Stuart Carnie 2cc1f5137e support for tenant+bucket
NOTE: to match storage service, values for database and rp are
hard-coded to `db` and `rp` respectively
2018-03-23 12:26:55 -07:00
Stuart Carnie aa61359cc7 Storage RPC API improvements. See PR for details
* reduce # allocations (115M -> 22M)
* reduce size allocations (53GB -> 1.3GB)
* reduce RPC query time (45s -> 12.9s)
2018-03-21 13:46:09 -07:00
Edd Robinson bdd61298ea Skip flaky test 2018-03-12 16:18:35 +00:00
Jonathan A. Sternberg 733d842812 Turn the ExecutionContext into a context.Context
Along with modifying ExecutionContext to be a context and have the
TaskManager return the context itself, this also creates a Monitor
interface and exposes the Monitor through the Context. This way, we can
access the monitor from within the query.Select method and keep all of
the limits inside of the query package instead of leaking them into the
statement executor.

An eventual goal is to remove the InterruptCh from the IteratorOptions
and use the Context instead, but for now, we'll just assign the done
channel from the Context to the IteratorOptions so at least they refer
to the same channel.
2018-03-08 14:03:20 -06:00
Jonathan A. Sternberg de4390ae83 Rename some of the structs and interfaces in the query package
Remove the `Query` prefix from some structs and interfaces. They were
there so when the query engine was in the same package as influxql,
these would be differentiated. Now that the package name is query, the
extra prefix seems redundant.
2018-03-02 09:44:12 -06:00
Stuart Carnie a74d296200 use underscore vs period, fix doc comment, add database name to CQ 2018-02-26 10:08:43 -07:00
Stuart Carnie 0a5a07dc3a series keys are produced in ascending order 2018-02-22 13:08:36 -07:00
Stuart Carnie d135aecf02 Generate trace logs for a number of significant influx operations
* tsdb Store.Open traces all events related to opening files
    * op.name : tsdb.open
* retention policy shard deletions
    * op.name : retention.delete_check
* all TSM compaction strategies
    * op.name : tsm1.compact_group
* series file compactions
    * op.name : series_partition.compaction
* continuous query execution (if logging enabled)
    * op.name : continuous_querier.execute
* TSI log file compaction
    * op_name: index.tsi.compact_log_file
* TSI level compaction
    * op.name: index.tsi.compact_to_level
2018-02-21 15:08:49 -07:00
Jonathan A. Sternberg d38413a849
Merge pull request #9454 from influxdata/js-structured-logging
Update logging calls to take advantage of structured logging
2018-02-21 09:14:40 -06:00
Jonathan A. Sternberg 2bbd96768d Update logging calls to take advantage of structured logging
Includes a style guide that details the basics of how to log.
2018-02-20 10:04:19 -06:00
Stuart Carnie 584e7ac09a Added option to write HTTP request logs to separate file. 2018-02-14 23:11:01 -07:00
Mark Rushakoff f7fc6a6501 Remove IsAdmin() method from meta.User interface
The method was only called in tests.
2018-02-13 16:59:22 -08:00
Stuart Carnie 8f978068f9
Merge pull request #9415 from influxdata/sgc-storage
restore `MetaClient`, which is needed by store
2018-02-09 07:51:36 -07:00