Commit Graph

1119 Commits (aa53a85774a39b2f60a322628060bc8812b7fcde)

Author SHA1 Message Date
Joe LeGasse a95647b720 cleanup: remove poor usage of ',ok' with maps
There are several places in the code where comma-ok map retrieval was
being used poorly. Some were benign, like checking existence before
issuing an unconditional delete with no cleanup. Others were potentially
far more serious: assuming that if 'ok' was true, then the resulting
pointer retrieved from the map would be non-nil. `nil` is a perfectly
valid value to store in a map of pointers, and the comma-ok syntax is
meant for when membership is distinct from having a non-zero value.
There was only one or two cases that I saw that being used correctly for
maps of pointers.
2017-08-30 09:49:31 -04:00
Edd Robinson 64933fea17 Support pre-existing request ID headers
Other applications or services sometimes expose a header containing a
unique ID, which can then be included in logging or response information
to allow an operator to link inter-service requests. The most common
header name used by services in the wild appears to be `X-Request-ID`,
but `Request-Id` is also used.

This commit adds support for specifying either `X-Request-ID` or
`Request-Id` headers, which will then be used by InfluxDB when logging
request information, and also in the `X-Request-ID` and `Request-Id`
response headers.

We populate both `X-Request-ID` and `Request-Id` to maintain backwards
compatibility with previous version, and to support the more common
`X-Request-ID` header name.

If both `X-Request-ID` and `Request-Id` are specified, then
`X-Request-ID` is used.

If neither header is specified, then in line with previous behaviour, we
generate a v1 UUID.
2017-08-23 13:42:33 +01:00
Edd Robinson 6b749f459e Merge pull request #8572 from influxdata/er-log-response
Log server error response messages
2017-08-16 23:17:57 +01:00
Jonathan A. Sternberg 8bd04ebe39 Remove TimeRange function and replace with a more accurate ConditionExpr function
The ConditionExpr function is more accurate because it parses the
condition and ensures that time conditions are actually used correctly.
That means that attempting to combine conditions with OR will not result
in the query silently pretending it's an AND and nested conditions work
correctly so there is only one way to read the query.

It also extracts the non-time conditions into a separate condition so we
can stop attempting to parse around the time conditions in lower layers
of the storage engine. This change does not remove those hacks, but a
following commit should be able to sanitize the condition and remove
them.
2017-08-16 16:45:35 -05:00
Edd Robinson a1d5913329 Log server error response messages
This commit provides more insight into server errors by both setting
the error on a response header, and, in the case of server errors (5xx),
logging those error messages to the HTTPD log, if [http] log_enabled =
true.
2017-08-16 21:32:58 +01:00
Jonathan A. Sternberg 9a2357c2c0 Separate the query engine into a separate package
This change provides a clear separation between the query engine
mechanics and the query language so that the language can be parsed and
dealt with separate from the query engine itself.
2017-08-16 13:38:43 -05:00
Edd Robinson 8934aacebb Adds mutex profile
Mutex profiles are now available via the:

  - /debug/pprof/mutex
  - /debug/pprof/all

endpoints.
2017-08-04 14:36:39 +01:00
Jonathan A. Sternberg 950753d036 Parse time literals using the time zone in the select statement 2017-07-27 13:05:51 -05:00
Jonathan Heusser c38d0de99d Update udp port to be in valid range
`80891` is not a valid port, changed to `8189`.
2017-07-27 13:26:37 +01:00
Connor Peet 4459e1e8d8 Return errors in CSV responses 2017-07-25 09:44:02 -05:00
Jonathan A. Sternberg 8bb80321d9 Merge pull request #8561 from influxdata/js-8525-query-http-pipelining
Support http pipelining for /query endpoint
2017-07-21 11:15:28 -05:00
Jonathan A. Sternberg 506c06241b Fixed time boundaries for continuous queries with time zones
This calculates the start and end time along with any time zones shifts
so that continuous queries are run at the correct time when a time zone
is included in the query.
2017-07-20 14:30:52 -05:00
Jonathan A. Sternberg 39591e5375 Support http pipelining for /query endpoint
Removing the forced `Connection: close` header from the `/query`
endpoint. This was originally added because of golang/go#13165, but it
seems like it's possible to use pipelining with go 1.8 and http 1.1,
just not recommended.

After some testing, it appears that the channel returned by
`ResponseWriter.CloseNotify()` will not send a value if the connection
was not interrupted. We already account for this in /query by exiting
from the goroutine if the request has finished by signaling another
channel.

Since the handler already accounts for the possibility that the channel
will not signal and since `CloseNotify()` doesn't interfere with a
pipelined request, we can remove the forced `Connection: close` that was
added to force clients to establish a new connection.
2017-07-20 09:38:01 -05:00
Jonathan A. Sternberg f7d07910aa Fix the cq start and end times to use unix timestamps
The Go timestamp leads Truncate to start a week on Monday, but the query
engine uses unix timestamps which has the week start on a Thursday.

Updating the service so it uses a custom truncate method that uses the
unix timestamp instead of `time.Time`.

Fixes #8569.
2017-07-17 14:59:45 -05:00
Adam 2259ada8c3 adds a new header key/value X-Influxdb-Build that has value OSS if called from open-source build, and ENT if called from enterprise. This commit sets the value for the OSS case, and also creates the proper flag 2017-07-07 10:59:26 -04:00
Edd Robinson 12248b7233 Allow panic recovery to be disabled
This commit adds a new environment variable INFLUXDB_PANIC_CRASH, which
when set to a truthy value, e.g., true, TRUE, 1, will prevent the server
from recovering from a panic.

Recover currently occurs in two places: the HTTP handler and the
QueryExecutor. INFLUXDB_PANIC_CRASH will control both.

Further, this commit adds _internal stats that will monitor the
occurrence of panics all the time (regardless of if INFLUXDB_PANIC_CRASH
has been set to true or not).

The recovered panic frequency can be inspected with the following
queries:

SELECT "recoveredPanics" FROM "_internal"."monitor"."httpd";
SELECT "recoveredPanics" FROM "_internal"."monitor"."queryExecutor";
2017-06-29 19:44:25 +01:00
Edd Robinson e1a5ee4ede Ensure privileges can't be set on non-existent DB 2017-06-26 14:15:52 +01:00
Adam b12ee4e6ee 8426: updated code based on pull-request feedback 2017-06-23 14:59:01 -04:00
Adam 5352b5865c 8426 Updated README to describe the new configuration option 2017-06-23 09:16:39 -04:00
Adam 3ca8b90f6c 8426: Refactored config to a string to make it more intuitive to the user, and to make setting a default easier 2017-06-23 09:12:52 -04:00
Adam 766702f9bd CORE-8426: finished feature update, tests demonstrate behavior. If config is set, it will group collectd stats into a single measurement as they appear in types.db 2017-06-22 18:54:16 -04:00
Stuart Carnie aa762b96df update field keys 2017-06-13 13:32:18 +10:00
Stuart Carnie 2de52834f0 CQ statistics written to monitor database, addresses #8188
* off by default, enabled by `query-stats-enabled`
* writes to cq_query measurement of configured monitor database
* see CHANGELOG for schema of individual points
2017-06-10 09:20:38 +08:00
Stuart Carnie 099371f2e7 Add config test; add negative value test 2017-06-06 08:42:45 +08:00
Stuart Carnie cac5dbfa5a NEW max-body-size config; HTTP 413 if body exceeds max size; fixes #8299 2017-06-05 21:38:33 +08:00
Stuart Carnie 55d1ba6d79 rework gzip compressor so it is lazily created for 200 OK requests only
* fix issue when panicking (before Write) gzip writer is closed, causing
header to be written and default status of 200 OK being written.
* update recovery middleware to set 500 Internal Server Error
2017-06-05 21:38:32 +08:00
Ryan Betts 19ef39d947 Merge pull request #8437 from influxdata/jl-points-auth
Fine Grained Authorization
2017-05-31 10:23:49 -04:00
Jonathan A. Sternberg 6a78f1cf4a URL query parameter credentials take priority over Authentication header 2017-05-30 09:26:24 -05:00
Stuart Carnie 8d8a7a0bfe pass meta.User to avoid future search requests 2017-05-26 15:05:38 -07:00
Stuart Carnie 47f97ea134 use parsed measurement and models.Tags 2017-05-26 13:21:59 -07:00
Stuart Carnie cec0712141 Add authorization error behavior API 2017-05-26 13:21:59 -07:00
Joe LeGasse 815f740f4c initial fga work
wip

wip

fix tests / build
2017-05-26 13:16:27 -07:00
David Norton c345f5d700 remove unused NewShardOwner func 2017-05-25 11:37:08 -04:00
Edd Robinson b4a427f9a2 Address PR feedback 2017-05-15 14:41:51 +01:00
Edd Robinson 1cbbaa9317 Add support for shards, stats and diagnostics 2017-05-15 14:12:00 +01:00
Edd Robinson 8f8ff0ec61 Adds handler for returning a profile archive
Currently, when debugging issues with InfluxDB we often ask for the
following profiles:

  curl -o block.txt "http://localhost:8086/debug/pprof/block?debug=1"
  curl -o goroutine.txt
"http://localhost:8086/debug/pprof/goroutine?debug=1"
  curl -o heap.txt "http://localhost:8086/debug/pprof/heap?debug=1"
  curl -o cpu.txt "http://localhost:8086/debug/pprof/profile

This can be bothersome for users, or even difficult if they're
unfamiliar with cURL (or it's not on their system).

This commit adds a new endpoint: /debug/pprof/all which will return a
single compressed archive of all of the above profiles. The CPU profile
is optional, and not returned by default. To include a CPU profile the
URL to request should be: /debug/pprof/all?cpu=true. It's also possible
to vary the length of the CPU profile by adding a `seconds=x` parameter,
where x defaults to 30, if absent.

The new command for gathering profiles from users should now be:

  curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all"

Or, if we need to see a CPU profile:

  curl -o profiles.tar.gz
"http://localhost:8086/debug/pprof/all?cpu=true"

It's important to remember that a CPU profile is a blocking operation
and by default it will take 30 seconds for the response to be returned
to the user.

Finally, if the user is unfamiliar with cURL, they will now be able to
visit http://localhost:8086/debug/pprof/all in a web browser, and the
archive will be downloaded to their machine.
2017-05-15 14:11:38 +01:00
Jason Wilder 2cac46ebbc Convert usage of strings to []byte
Measurement name and field were converted between []byte and string
repetively causing lots of garbage.  This switches the code to use
[]byte in the write path.
2017-05-12 14:05:19 -06:00
Jonathan A. Sternberg 2780630a5f Track HTTP client requests for /write and /query with /debug/requests
After using `/debug/requests`, the client will wait for 30 seconds
(configurable by specifying `seconds=` in the query parameters) and the
HTTP handler will track every incoming query and write to the system.
After that time period has passed, it will output a JSON blob that looks
very similar to `/debug/vars` that shows every IP address and user
account (if authentication is used) that connected to the host during
that time.

In the future, we can add more metrics to track. This is an initial
start to aid with debugging machines that connect too often by looking
at a sample of time (like `/debug/pprof`).
2017-05-09 10:18:33 -05:00
Jonathan A. Sternberg 260bdef3d4 Set the CSV output to an empty string for null values 2017-05-04 20:51:58 -05:00
Stuart Carnie 8097e817f6 prefix partial write errors with `partial write:`
NOTE: parser errors (via http API) are also transformed into
PartialWriteError
2017-04-28 11:00:14 -07:00
Jonathan A. Sternberg 1300f4cc6c Remove the admin UI 2017-04-25 16:58:24 -05:00
Jonathan A. Sternberg 45895862b7 Merge pull request #8058 from karlding/service-golinting
Make services/{admin, httpd, subscriber, udp} golintable
2017-04-05 12:30:11 -05:00
Karl 10b7f86368 Make services/subscriber golintable 2017-04-03 18:32:36 -04:00
Karl 08b3dbcbc7 Make services/udp golintable 2017-04-03 18:32:36 -04:00
Karl 616224695a Make services/httpd golintable 2017-04-03 18:32:36 -04:00
Karl 4171d6a843 Make services/admin golintable 2017-04-03 18:32:36 -04:00
Edd Robinson fddaff2cc8 Merge master in 2017-03-29 18:00:28 +01:00
Jason Wilder 8f7b251afd Merge branch 'master' into jw-tsi 2017-03-20 17:17:26 -06:00
Edd Robinson 255992f5ec Reduce cost of admin user check
This commits adds a caching mechanism to the Data object, such that
when large numbers of users exist in the system, the cost of determining
if there is at least one admin user will be low.

To ensure that previously marshalled Data objects contain the correct
cached admin user value, we exhaustively determine if there is an admin
user present whenever we unmarshal a Data object.
2017-03-20 12:04:03 +00:00
Jonathan A. Sternberg 9cdfdd04e9 Do not increment the continuous query statistic if no query is run
Instead of incrementing the `queryOk` statistic with or without the
continuous query running, it will only increment when the query is
actually executed.
2017-03-16 10:36:00 -05:00
Ben Johnson 358b1e0b05
Merge remote-tracking branch 'upstream/master' into tsi 2017-03-15 10:13:32 -06:00
Jason Wilder e62c72d1f9 Merge branch '1.2' into jw-merge-12 2017-03-14 15:15:50 -06:00
Jason Wilder c9740f753b Disable max-row-limit by default
max-row-limit was set at 10000 since 1.0, but due to a bug it was
effectively 0 (disabled).  1.2 fixed this bug via #7368, but this
caused a breaking change w/ Grafana and any users upgrading from <1.2
who had not disabled the config manually.
2017-03-14 12:47:32 -06:00
Mark Rushakoff 535cf597f1 Report subset of config values in SHOW DIAGNOSTICS
This includes hand-selected config settings that are safe to expose and
not expected to include any kind of secrets.

Fixes #7821
2017-03-14 11:34:19 -07:00
Jason Wilder 675d7c9d65 Merge branch '1.2' into jw-merge12 2017-03-06 11:09:05 -07:00
David Norton c7fa58473f fix #8028: call api.NewTypesDB() instead of new
The code was calling new(api.TypesDB) which didn't initialize an
unexported map inside of the type. Call api.NewTypesDB() instead.
2017-02-23 18:04:21 -05:00
Mark Rushakoff 601cbcd084 Merge branch '1.2' into mr-merge-12 2017-02-17 16:14:22 -08:00
Jonathan A. Sternberg 2fe48d6781 Rename zap import back to github.com/uber-go/zap
They rebased a revision we were previously relying upon that allowed us
to use the vanity name so we are reverting back to an older version with
the old import path.
2017-02-17 17:17:22 -06:00
Mark Rushakoff 53699aa24f Allow non-admin users to execute SHOW DATABASES
This commit introduces a new interface type, influxql.Authorizer, that
is passed as part of a statement's execution context and determines
whether the context is permitted to access a given database. In the
future, the Authorizer interface may be expanded to other resources
besides databases. In this commit, the Authorizer interface is
specifically used to determine which databases are returned when
executing SHOW DATABASES.

When HTTP authentication is enabled, the existing meta.UserInfo struct
implements Authorizer, meaning admin users can SHOW every database, and
non-admin users can SHOW only databases for which they have read and/or
write permission.

When HTTP authentication is disabled, all databases are visible through
SHOW DATABASES.

This addresses a long-standing issue where Chronograf or Grafana would
be unable to list databases if the logged-in user did not have admin
privileges.

Fixes #4785.
2017-02-13 08:59:16 -08:00
Carlo Alberto Ferraris 005e480b55 [influxd] add utility functions to make it harder to misuse gzipWriterPool 2017-02-09 09:31:11 +09:00
Carlo Alberto Ferraris a6a7782e04 [influxd] Use a sync.Pool to reuse gzip.Writer across requests
This brings alloc_space down from ~20200M to ~10700M in a run of
go test ./cmd/influxd/run -bench=Server -memprofile=mem.out -run='^$'
2017-02-07 05:23:58 +09:00
Edd Robinson 366223140d Add system information to /debug/vars 2017-01-30 18:23:21 +00:00
Edd Robinson 91ee34b111 Merge pull request #7837 from influxdata/er-tidy
General tidy up and subtle bug fixes
2017-01-26 13:43:07 +00:00
Ben Johnson 047c21f4d9
Merge remote-tracking branch 'upstream/master' into tsi 2017-01-24 09:28:58 -07:00
Edd Robinson 63562d9fd5 Fixes #7860.
Fix a regression introduced in #7449.

This commit ensures that create database with retention policy will work
correctly.
2017-01-23 19:01:13 +00:00
Edd Robinson 4442cd7efa Fixes #7860.
Fix a regression introduced in #7449.

This commit ensures that create database with retention policy will work
correctly.
2017-01-23 17:57:32 +00:00
Edd Robinson feb7a2842c Use unbuffered error channels in tests 2017-01-17 10:53:15 -08:00
Edd Robinson 9d30ee0a6b Fix subtle bugs and remove dead code from services 2017-01-17 09:47:34 -08:00
Edd Robinson c47be5bb56 Ensure subscriber service respects config 2017-01-13 22:15:01 +00:00
Edd Robinson aa6b85a2dc Ensure subscriber service respects config 2017-01-13 20:09:40 +00:00
Joe LeGasse 2db0250b22 Add db/rp name validation
This change adds some very basic name validation with the following
plain-english description: names must be non-zero sequence of printable
characters that do not contain slashes ('/' or '\') and are not equal to
either "." or "..".

The intent is that, since we currently just use database and retention
policy names directly as path elements, these rules will hopefully leave
us with names that should be at least close to valid directory names.

Ideally, we would restrict names even further or not use them as path
elements directly, but this should be a step towards the former without
restricting names "too much"
2017-01-12 17:38:10 -05:00
Jason Wilder ccdb68841b Merge pull request #7820 from influxdata/cjl-7689-sgi-unix-epoch-zero-time
fix marshaling of zero time
2017-01-12 12:34:48 -07:00
Cory LaNou c6dc164537
fix marshaling of zero time 2017-01-11 12:00:18 -06:00
Mark Rushakoff ede3f3337d Merge pull request #7818 from influxdata/mr-config-validation
Add more config validation
2017-01-11 07:41:32 -08:00
Vladimir Lopes f05df2a263 Fix panic when pruning shard groups
* Fix #7812 - Panic when pruning shard groups

* Update CHANGELOG.md
2017-01-11 14:56:40 +00:00
Mark Rushakoff bbb43faad2 Add more config validation 2017-01-10 10:28:49 -08:00
Mark Rushakoff a135906b43 Merge pull request #7747 from influxdata/mr-lint-cleanup
Miscellaneous lint cleanup
2017-01-10 08:22:00 -08:00
Jonathan A. Sternberg d7c8c7ca4f Support subquery execution in the query language
This adds query syntax support for subqueries and adds support to the
query engine to execute queries on subqueries.

Subqueries act as a source for another query. It is the equivalent of
writing the results of a query to a temporary database, executing
a query on that temporary database, and then deleting the database
(except this is all performed in-memory).

The syntax is like this:

    SELECT sum(derivative) FROM (SELECT derivative(mean(value)) FROM cpu GROUP BY *)

This will execute derivative and then sum the result of those derivatives.
Another example:

    SELECT max(min) FROM (SELECT min(value) FROM cpu GROUP BY host)

This would let you find the maximum minimum value of each host.

There is complete freedom to mix subqueries with auxiliary fields. The only
caveat is that the following two queries:

    SELECT mean(value) FROM cpu
    SELECT mean(value) FROM (SELECT value FROM cpu)

Have different performance characteristics. The first will calculate
`mean(value)` at the shard level and will be faster, especially when it comes to
clustered setups. The second will process the mean at the top level and will not
include that optimization.
2017-01-07 13:00:48 -06:00
Ben Johnson 2b3cd415e2
Fixing rebase. 2017-01-06 09:52:16 -07:00
Edd Robinson 33623c1fa9
Revert back to original approach 2017-01-05 09:58:39 -07:00
Edd Robinson 05bc4dec00
Refactor 2017-01-05 09:50:23 -07:00
Mark Rushakoff 6a94d200c8 Merge remote-tracking branch 'influx/master' into mr-godoc 2017-01-04 13:27:36 -08:00
Cory LaNou 3c518f8927
panicing is bad -> error returns are good 2017-01-03 14:28:29 -06:00
Mark Rushakoff 07b87f2630 Miscellaneous lint cleanup 2017-01-03 09:47:32 -08:00
Mark Rushakoff 959c445a88 Fix broken return statements swallowing errors
There was no comment on either case specifying that the `return nil`
was deliberate instead of `return err`, so I'm assuming these were
typos. I added tests to conserve the error-returning behavior.
2017-01-03 08:50:34 -08:00
Mark Rushakoff 218fc3890d Update godoc for services
The admin service was deliberately skipped due to it being deprecated.
2016-12-30 18:03:01 -08:00
Gustav Westling 13efbc6ce0
Removed empty line 2016-12-30 12:22:09 +01:00
Gustav Westling a8343dee99
Fix broken test. Change Fatalf to Fatal. 2016-12-29 20:41:00 +01:00
Gustav Westling 56d98325da
Removed ineffective assignments, and added checks for errors that previsouly was not checked 2016-12-29 20:26:15 +01:00
Cory LaNou 572da8985c enforce minimum shard duration when creating retention policies 2016-12-20 09:11:43 -06:00
Jonathan A. Sternberg ec57108520 Use proper uber-go/zap import path
It looks like the real import path to the project is go.uber.org/zap
instead of github.com/uber-go/zap since the example in the project
references that path.
2016-12-15 08:54:14 -06:00
Mark Rushakoff eceecf4cfa Merge pull request #7721 from influxdata/mr-fix-flaky-port-tests
Bind to random port in service open/close tests
2016-12-14 10:03:26 -08:00
Jonathan A. Sternberg 21502a39e8 Switch logging to use structured logging everywhere
The logging library has been switched to use uber-go/zap. While the
logging has been changed to use structured logging, this commit does not
change any of the logging statements to take advantage of the new
structured log or new log levels. Those changes will come in future
commits.
2016-12-14 10:45:15 -06:00
Mark Rushakoff ede3144300 Bind to random port in service open/close tests
These tests have been flaky on Circle in the past due to those specific
ports being in use. Binding to port 0 will pick a random, available
port.
2016-12-14 08:29:29 -08:00
Allen Petersen 5b25a93b45 Clean error and log messages 2016-12-07 12:54:41 -08:00
Allen Petersen da9941b9a9 Reject invalid subscription urls
The url must have a scheme of udp,http,https and a port number.
CREATE SUBSCRIPTION will fail if there are invalid destinations.

Additionally Service.createSubscription fail invalid destinations are detected.

Fixes #7615
2016-12-07 12:54:41 -08:00
Edd Robinson aff2d7ce64 Ensure user privs dropped along with db (#7677) 2016-12-02 18:23:35 +00:00
Cory LaNou d54d32b9d4
prune shards in meta data 2016-12-01 11:22:16 -06:00
Jonathan A. Sternberg 825aa07877 Use X-Forwarded-For IP address in HTTP logger if present 2016-12-01 09:16:36 -06:00
Jonathan A. Sternberg b4db76cee2 Introduce syntax for marking a partial response with chunking
The `partial` tag has been added to the JSON response of a series and
the result so that a client knows when more of the series or result will
be sent in a future JSON chunk.

This helps interactive clients who don't want to wait for all of the
data to know if it is done processing the current series or the current
result. Previously, the client had to guess if the next chunk would
refer to the same result or a new result and it had to match the name
and tags of the two series to know if they were the same series. Now,
the client just needs to check the `partial` field included with the
response to know if it should expect more.

Fixed `max-row-limit` so it counts rows instead of results and it
truncates the response when the `max-row-limit` is reached.
2016-11-22 11:16:22 -06:00
Jason Wilder 61daedac03 Use time.Tick instead of time.After
Eliminates an allocation
2016-11-15 16:13:55 -07:00
Jason Wilder 77e2c80a4f Merge pull request #7435 from marcv81/feature-7066
Support for collectd signed and encrypted packets
2016-11-11 11:30:24 -07:00
Jonathan A. Sternberg 64c2d704da Avoid deadlock when max-row-limit is hit
When the `max-row-limit` was hit, the goroutine reading from the results
channel would stop reading from the channel, but it didn't signal to the
sender that it was no longer reading from the results. This caused the
sender to continue trying to send results even though nobody would ever
read it and this created a deadlock.

Include an `AbortCh` on the `ExecutionContext` that will signal when
results are no longer desired so the sender can abort instead of
deadlocking.
2016-11-08 13:12:28 -06:00
Marc 7d277d0def Add support for collectd security policy
There are 2 new keys in the configuration file.
- security-level: "none", "sign", or "encrypt".
- auth-file: The location of the user/password file.

Please see the collectd network doc for more details.
2016-11-08 08:42:09 +08:00
Marc 0260dfb575 Replace kimor79/gollectd with collectd.org
At this point this is a like-for-like swap.

collectd.org is more active and supports more features.
2016-11-08 08:24:57 +08:00
Jason Wilder b6863b04d1 Disable/deprecate admin interface 2016-11-07 12:04:51 -07:00
Cory LaNou e112de2ee0 Merge pull request #7554 from influxdata/cjl-godeps
update latest dependencies with Godeps
2016-11-07 09:10:30 -06:00
Cory LaNou 3774d5e734
update latest dependencies with Godeps 2016-11-07 08:49:29 -06:00
Cory LaNou 6e290040bb
remove SetDefaultRetentionPolicy method from meta.Client 2016-11-03 09:39:41 -05:00
Cory LaNou cd272ce6c3 fix retention policy creation inconsistencies 2016-11-03 09:09:43 -05:00
Jonathan A. Sternberg 3e29d3d9ca Truncate the version string when linking to the documentation
The admin console would dynamically discover the version from the
InfluxDB server, but for patch releases, it included the patch in the
link to the documentation and that wasn't a valid link.

Truncate the version so the documentation url is correct since we only
do documentation for `major.minor`.
2016-10-27 16:40:08 -05:00
Jason Wilder 6fa145943a Merge pull request #7483 from hvnsweeting/doc-fixes
Doc fixes
2016-10-24 11:38:59 -06:00
Cory LaNou 5b72b874d8
remove ProcessContinousQueries from httpd endpoint 2016-10-20 11:22:36 -05:00
Hung Nguyen Viet 2cb6c64787 Sections headers should be all capitals 2016-10-20 18:57:13 +07:00
Hung Nguyen Viet 1f26984181 Graphite 2016-10-20 18:57:13 +07:00
Hung Nguyen Viet cc7b61d385 fix grammar error 2016-10-20 18:57:13 +07:00
Hung Nguyen Viet 179f5b02cc TSM is the only engine now 2016-10-20 18:57:05 +07:00
Hung Nguyen Viet 1f31e8b707 clarify 2016-10-20 18:48:56 +07:00
Hung Nguyen Viet 4751ca1f43 use markdown 2016-10-20 18:47:54 +07:00
Hung Nguyen Viet dcb494ded5 use plural 2016-10-20 18:47:54 +07:00
Edd Robinson 168c91cc67 graphite service should retry database creation 2016-10-18 16:50:07 +01:00
Edd Robinson 73c267abc1 udp service should retry database creation 2016-10-18 16:50:07 +01:00
Edd Robinson 0470f38478 opentsdb should retry creating its database 2016-10-18 16:50:07 +01:00
Edd Robinson cc60647827 collectd should retry creating database 2016-10-18 16:50:07 +01:00
Edd Robinson 6c149b31ef Remove unused exported methods 2016-10-18 16:19:44 +01:00
Edd Robinson ad2d33f859 Ensure input services can be safely opened and closed 2016-10-18 10:35:38 +01:00
zhexuany 931a6c6d08 fixed two typo 2016-10-13 13:43:38 +08:00
Jason Wilder bbecb3f03d Drop points that would execeed limits
This changes the behavior of the max-series-per-database and
max-values-per-tag limits to drop points that would exceed the limits
and allow the remaining points to be written.  Previously, the whole
batch would fail and return and 500 error to the client.

This now will write the allow points and return a `partial write`
error indicating some of the points were dropped, how many were
dropped and one of the problem measureent and tags.
2016-10-10 11:42:15 -06:00
Jason Wilder ebf50d06ca Merge branch '1.0' into jw-merge-102 2016-10-06 09:51:35 -06:00
Jason Wilder ea21588b9e Fix subscriber service dropping writes under high write load
The subscriber write goroutine would drop points if the write load
was higher than it could process.  This could happen with a just
a few writers to the server.

Instead, process the channel with multiple writers to avoid dropping
writes so easily.  This also adds some config options to control how
large the channel buffer is as well as how many goroutines are started.

Fixes #7330
2016-10-04 14:47:36 -06:00
Nathaniel Cook 91645c05cf Merge pull request #7392 from mglazer/feature/https-subscriber
Fix Kapacitor Issue #942: HTTPS subscriptions don't work
2016-10-04 14:45:05 -06:00
Mike Glazer fa3609b251 Fix Kapacitor Issue #942: HTTPS subscriptions don't work 2016-10-04 20:58:31 +02:00
Edd Robinson 537316eb7b Ensure pprof-enabled config option is respected 2016-09-30 18:41:38 +01:00
Joe LeGasse 9d26cc0732 Stablize the sorting of meta.ShardGroupInfos 2016-09-30 13:58:03 +01:00
Jason Wilder 6e7da3d24a Merge pull request #7379 from influxdata/jw-meta-clone
Don't clone the cached meta store
2016-09-28 11:52:43 -06:00
Jonathan A. Sternberg bb8c4efa95 Do not automatically reset the shard duration when using ALTER RETENTION POLICY 2016-09-28 12:26:03 -05:00
Jason Wilder bc00490b92 Don't clone the cached meta store
The cacheData is read-only and does not need to be cloned.  Cloning
it for read operations can be expensive with lots of shards/database/RPs/etc.
2016-09-28 10:35:59 -06:00
Jonathan A. Sternberg 3afdf3cd94 Merge tag 'v1.0.1' 2016-09-27 17:53:33 -05:00
joelegasse 1eda0277a7 Merge pull request #7365 from influxdata/jl-shardgroup-sort
Stablize the sorting of meta.ShardGroupInfos
2016-09-27 11:02:47 -04:00
Jason Wilder 14b0f6e496 Fix panic in subscriber.Statistics
The tags passed into Statistics() calls are not supposed to be modified.
The balanceWriter in subscribers tried to modify them triggering a panic
because they can be nil.
2016-09-16 11:49:15 -06:00
Jonathan A. Sternberg 477d6231db Update source files to pass vet checks for go 1.7
The vet checks for some files did not pass for go 1.7. As part of a
preliminary start to making go 1.7 work with this software, go vet
should pass.

Also updated the gogo/protobuf dependency which fixed the code generator
to work with go 1.7 too. Ran `go generate` on the entire repository to
ensure every file was up to date.
2016-09-14 15:01:22 -05:00
Todd Persen b8192d079e Regenerate static assets and update CHANGELOG for PR #7286. 2016-09-13 23:38:36 -07:00
Dan Dascalescu 858060375e Proper password field 2016-09-09 22:30:17 -07:00
Jonathan A. Sternberg ab4bca8495 Report cmdline and memstats in /debug/vars
When we refactored expvar, the cmdline and memstats sections were not
readded to the output. This adds it back if they can be found inside of
`expvar`.

It also stops trying to sort the output of the statistics so they get
returned faster. JSON doesn't need them to be sorted and it causes
enough latency problems that sorting them hurts performance.
2016-09-09 14:32:43 -05:00
Jonathan A. Sternberg dc2527ce86 Merge branch '1.0' 2016-08-31 14:45:57 -05:00
Jonathan A. Sternberg 23f2d50ecb Use defaults from `meta` package for `CREATE DATABASE`
Instead of having the parser set the defaults, the command will set the
defaults so that the constants for that are actually used. This way we
can also identify which things the user provided and which ones we are
filling with default values.

This allows the meta client to be able to make smarter decisions when
determining if the user requested a conflict or if the requested
capabilities match with what is currently available. If you just say
`CREATE DATABASE WITH NAME myrp`, the user doesn't really care what the
duration of the retention policy is and just wants to use the default.
Now, we can use that information to determine if an existing retention
policy would conflict with what the user requested rather than returning
an error if a default value ever gets changed since the meta client
command can communicate intent more easily.
2016-08-30 13:23:49 -05:00
Jonathan A. Sternberg c05c7f6360 Revert "limit shard concurrency"
This reverts commit 6c7d56d4bc.
2016-08-29 12:39:52 -05:00
Edd Robinson 6cafdbc604 Ensure we don't mutate provided statistics tags 2016-08-24 11:40:13 +01:00
Edd Robinson 90ff713f21 Fix base64 encoding issue in stats
Fixes #7177.
2016-08-22 15:21:31 +01:00
Jonathan A. Sternberg 9b28f7704f Merge pull request #7171 from influxdata/js-1.0-merge-response-formatter-bugfix
Merge branch '1.0'
2016-08-17 19:13:09 -05:00
Jonathan A. Sternberg 5e468fac0b Merge branch '1.0' into master 2016-08-17 13:27:02 -05:00
Jonathan A. Sternberg d2746ee8f2 The number of bytes recorded when using chunking was off by one
Previously, we implicitly added a newline and had to add one to the
number of bytes transmitted because we added that byte. That was removed
at some point and the metric was not updated to record the correct
value.
2016-08-17 11:54:19 -05:00
Jonathan A. Sternberg 71b54768d0 Remove redundant code the from response formatter and fix CloseNotify
The query killing functionality depends on the ResponseWriter exposing a
CloseNotify method. Since we wrap the http.ResponseWriter, the new
struct does not have that method and the HTTP handler would skip past
calling that method.

Instead of duplicating `Flush()` and `CloseNotify()` for every response
formatter, we will unify all of that under a single struct and create
formatters instead.

Also, fixes a bug where the header information from a query would not be
returned until some other data was returned with it because of
buffering and another bug in the gzipResponseWriter that wouldn't flush
the actual underlying ResponseWriter.
2016-08-16 15:54:03 -05:00
Ben Johnson 8aa224b22d
reduce memory allocations in index
This commit changes the index to point to index data in the shards
instead of keeping it in-memory on the heap.
2016-08-16 14:09:00 -06:00
Edd Robinson cebeda817c Update jwt-go to v3 2016-08-12 17:35:57 +01:00
Jonathan A. Sternberg 87f7c66b8a Merge pull request #7119 from influxdata/js-create-database-use-defaults
Use defaults from `meta` package for `CREATE DATABASE`
2016-08-11 10:34:22 -05:00
Jonathan A. Sternberg f58a50c231 Allow queries to be uploaded as a file rather than forcing a text parameter
The query can be uploaded from a file using `multipart/form-data` and
setting the file name to `q`. An example of using curl to execute an
async query would be:

    curl -F "q=@database.iql" -F "async=true" http://localhost:8086/query
2016-08-10 15:34:04 -05:00
Jonathan A. Sternberg 760767b033 Support running a command in async mode
It will return a 204 No Content as long as the query is accepted
(immediate errors will be returned, but not individual errors with
specific queries). The only way to kill the query is by using the task
manager.
2016-08-10 15:34:04 -05:00
Jonathan A. Sternberg a4e49963f5 Implement text/csv content encoding for the response writer
CSV doesn't offer a way to separate different sheets from each other and
it doesn't really have a standard format. We separate sheets with a
newline so they can be imported into something like Excel or LibreOffice
more easily.

The number of columns for each sheet is inferred from the first returned
row in each statement since they should all be the same.
2016-08-10 15:15:33 -05:00
kun 6945655be2 add support for unix socket binding service 2016-08-11 02:20:54 +08:00
Jonathan A. Sternberg 530b00bd76 Use defaults from `meta` package for `CREATE DATABASE`
Instead of having the parser set the defaults, the command will set the
defaults so that the constants for that are actually used. This way we
can also identify which things the user provided and which ones we are
filling with default values.

This allows the meta client to be able to make smarter decisions when
determining if the user requested a conflict or if the requested
capabilities match with what is currently available. If you just say
`CREATE DATABASE WITH NAME myrp`, the user doesn't really care what the
duration of the retention policy is and just wants to use the default.
Now, we can use that information to determine if an existing retention
policy would conflict with what the user requested rather than returning
an error if a default value ever gets changed since the meta client
command can communicate intent more easily.
2016-08-09 12:00:06 -05:00
Jonathan A. Sternberg b98763a3d8 Merge pull request #7118 from influxdata/js-go-generate
go generate on every package to ensure they are generated with the correct dependency
2016-08-08 09:02:32 -05:00
Jonathan A. Sternberg 4cdfc3280d Move the CQ interval by the group by offset
This will make the period selected by the CQ system work correctly for a
query with an offset.
2016-08-05 14:39:52 -05:00
Jonathan A. Sternberg 0879d43b74 Do not run continuous queries that have no time span 2016-08-05 14:39:52 -05:00
Jonathan A. Sternberg ed2f81357f go generate on every package to ensure they are generated with the correct dependency 2016-08-05 14:35:07 -05:00
Ben Johnson 6c7d56d4bc
limit shard concurrency
This commit limits queries to only process one shard at a time.
However, within a shard, multiple series can still be processed in
parallel. Shard iterators are lazily instantiated during query
execution to limit the amount of memory a given query uses.
2016-08-05 09:45:57 -06:00
Jonathan A. Sternberg 73bbbf24b1 Use the Accept header for determining the Content-Type instead of Content-Type
The `Content-Type` header is meant to say what the content type of the
request is supposed to be and `Accept` is used to ask for a specific
content type. I messed this up and used `Content-Type` instead of
`Accept`. This works with `GET` requests accidentally, but `POST`
requests stop working.
2016-07-29 14:29:39 -05:00
Jonathan A. Sternberg d85072ada9 Setup an interface for writing http responses based on the Content-Type header
Only `application/json` is supported right now, but this opens up the
easier possibility of additional content types to be returned from the
server.
2016-07-29 13:11:51 -05:00
Edd Robinson 9a2efafa13 Ensure CQ id names can't clash 2016-07-28 16:26:05 +01:00
Cameron Sparr d688676266 Update UDP buffer setting doc to 25MB 2016-07-28 12:02:56 +01:00
Cameron Sparr 6afdcf9572 Fix incrementing pointsRx twice 2016-07-28 11:06:50 +01:00
David Norton e3328117a9 fix #7081: hardcode auto gen RP name 2016-07-27 16:49:46 -04:00
Joe LeGasse b3341e8f66 Stablize the sorting of meta.ShardGroupInfos 2016-07-27 16:17:52 -04:00
David Norton b1079c9080 revert previous DefaultRetentionPolicyName change 2016-07-27 11:38:42 -04:00
David Norton 8fbfdc5616 fix #7068: use DefaultRetentionPolicyName from cfg
Use DefaultRetentionPolicyName from the config instead of passing
through meta data.
2016-07-26 18:53:55 -04:00
Jonathan A. Sternberg 4bdf57534b Log the CQ execution time when continuous query logging is enabled 2016-07-21 12:16:53 -05:00
Jonathan A. Sternberg f6ff8a572d Run continuous query for multiple buckets rather than one per bucket 2016-07-14 11:23:05 -05:00
Jonathan A. Sternberg 83caeec3a4 Support loading a folder for collectd typesdb files 2016-07-07 20:57:57 -05:00
Jonathan A. Sternberg 7a3bd19926 Properly use the 401 and 403 HTTP status codes
According to the HTTP standard, a lack of authentication credentials or
incorrect authentication credentials should send back a 401
(Unauthorized) with a `WWW-Authenticate` header with a challenge that
can be used to authenticate. This is because a 401 status should be sent
when an authentication attempt can be retried by the browser.

The 403 (Forbidden) status code should be sent when authentication
succeeded, but the user does not have the necessary authorization.
Previously, the server would always send a 401 status code.
2016-07-07 20:30:04 -05:00
Jonathan A. Sternberg 837a9804cf Refactoring the monitor service to avoid expvar
Truncate the time interval output of the monitor service to be on even
time intervals rather than on every minute based on the start time. This
normalizes the output from the monitor service.
2016-07-07 11:13:58 -05:00
Edd Robinson 2c85d498ef Remove unused errors 2016-07-06 15:04:39 +01:00
Mikael Taina b9d3ae085c Don't force root path on getClientVersion
Similar to #1339 .

Fix getClientVersion() on setups where the admin interface resides on some
other path than root path due to some reverse proxy setup.
2016-07-05 22:47:48 +03:00
Cole Mackenzie 6d926722b5 added favicon.ico and regenerated statik.go 2016-07-01 08:53:56 +00:00
Jonathan A. Sternberg b8e52ce39a Merge pull request #6889 from influxdata/js-update-config-options
Update help and remove unused config options from the configuration file
2016-06-23 11:49:10 -05:00
kun 8aa2fe8ede fix ClusterID generation bug 2016-06-23 14:52:58 +08:00
Jonathan A. Sternberg 22173acb70 Update help and remove unused config options from the configuration file
Normalize the output for the various help options so they all follow the
same format and display all relevant options.

Removing some of the unused config options from the configuration file
and updating the help documentation. Removing some remaining references
to clustering within the open source version.
2016-06-21 14:06:05 -05:00
Jonathan A. Sternberg 497db2a6d3 Removing dead code from every package except influxql
The tsdb package had a substantial amount of dead code related to the
old query engine still in there. It is no longer used, so it was removed
since it was left unmaintained. There is likely still more code that is
the same, but wasn't found as part of this code cleanup.

influxql has dead code show up because of the code generation so it is
not included in this pruning.
2016-06-20 22:41:07 -05:00
Edd Robinson 9603bad14c Merge pull request #6868 from smunier01/continuous-queries-admin-interface
Add continuous queries templates in admin interface
2016-06-20 11:35:47 +01:00
Edd Robinson 3d8cde2346 Merge pull request #6774 from minkikim89/patch-1
Fixed admin page reloaded when navbar-brand clicked
2016-06-20 11:34:08 +01:00
simon f007fccb4a Add continuous queries templates in admin interface 2016-06-18 16:33:41 +02:00
Jonathan A. Sternberg 8e1b036b0a Modify the max nanosecond time to be one nanosecond less
The highest time represented by a nanosecond needs to be used for an
exclusive range, so the maximum time needs to be one less than the
possible maximum number of nanoseconds representable by an int64 so that
we don't lose a point at that one time.

Previously worked in the open source version because the timestamp used
for finding a shard would be truncated by the retention policy so the
lookup time didn't run into this edge case because it didn't rest on the
truncation boundary. Since that point didn't really belong in that shard
group and was placed there by mistake, it's best to fix this bug since
the timestamp used to create the shard group should be capable of
retrieving it.
2016-06-16 12:15:41 -05:00
Jack Zampolin a96f96c024 Merge pull request #6834 from influxdata/jz-graphite
Add `port` to `[graphite]` log output
2016-06-14 11:35:19 -07:00
Cameron Sparr b9a3f2e1e5 Make httpd logger closer to Common Log Format
changes the httpd log lines from this:

    [httpd] 2016/06/08 14:06:39 ::1 - - [08/Jun/2016:14:06:39 +0100] POST /write?consistency=any&db=telegraf&precision=s&rp= HTTP/1.1 204 0 - InfluxDBClient d6aa01fc-2d79-11e6-8024-000000000000 2.751391ms

to this:

    [httpd] ::1 - - [08/Jun/2016:14:06:39 +0100] "POST /write?consistency=any&db=telegraf&precision=s&rp= HTTP/1.1" 204 0 "-" "InfluxDBClient" d6aa01fc-2d79-11e6-8024-000000000000 2751

So it changes a few things:

1. Remove the logger timestamp at the beginning which isn't very relevant anyways
2. adds quotes around "METHOD URI PROTOCOL", because this is part of the
common log format.
3. adds quotes around "AGENT" and "REFERRER" because this is part of the
"combined" log format.
4. Puts the response time in integer microseconds, because this is
consistent with apache's %D config mod option.

Compared with CLF, our logs now look like this:

    [httpd] %{COMMON_LOG_FORMAT} "<agent>" "<referrer>" <request_uuid> <response_time_µs>

For reference, see:
https://en.wikipedia.org/wiki/Common_Log_Format
http://httpd.apache.org/docs/current/mod/mod_log_config.html
2016-06-14 14:48:24 +01:00
Jack Zampolin f1db2f2ca5 Add port to all graphite log output to help with debugging multiple endpoints 2016-06-13 11:59:56 -07:00
Jason Wilder fd262697b9 Initialize subscriber channel on open
If the subscriber service was closed and re-open, it would panic
with panic: send on closed channel
2016-06-10 18:00:44 -06:00
Ben Johnson 48f1a6d858 Merge pull request #6820 from benbjohnson/http-query-node-id
Add NodeID to execution options
2016-06-10 13:10:15 -06:00
Jonathan A. Sternberg 9db82e6bf0 Switch ExecutionContext to be passed by value 2016-06-10 12:31:51 -05:00
Jonathan A. Sternberg 55973d2815 Separate the task manager from the query executor
The task manager now acts as its own statement executor so that a custom
statement executor can perform custom actions for KillQueryStatement and
ShowQueriesStatement.
2016-06-10 12:30:51 -05:00
Ben Johnson 7d4bea7153
add node id to execution options
This commit changes the `ExecutionOptions` and `SelectOptions` to
allow a `NodeID` for specifying an exact node to query against.
2016-06-10 09:20:44 -06:00
Jonathan A. Sternberg 4f37bc5a40 Support specifying a retention policy for the graphite service
The graphite service will attempt to create the retention policy and use
it. If the retention policy doesn't exist, it will be created with the
default options.

Fixes #5655.
2016-06-08 17:01:44 -05:00
Joe LeGasse 647210c57a Add TruncatedAt field to meta.ShardGroupInfo 2016-06-07 18:24:24 -04:00
Jonathan A. Sternberg b972c220aa Merge pull request #6757 from influxdata/js-refactor-execute-query
Refactor ExecuteQuery to take options as a struct
2016-06-07 10:35:52 -05:00
Jonathan A. Sternberg 1e562408f1 Merge pull request #6779 from joe2far/patch-1
Fixed typo in docstring
2016-06-07 08:26:23 -05:00
Jonathan A. Sternberg b8e22d9d79 Merge pull request #6586 from influxdata/js-3733-rename-default-retention-policy
Modify the default retention policy name and make it configurable
2016-06-06 15:05:29 -05:00
Rubycut a849f42cc2 Lock auth properly, code suggested by @jwilder 2016-06-04 09:05:10 +02:00
Joe Farrell bb8b08e2ca Fixed typo in docstring 2016-06-03 18:01:04 +01:00
Jonathan A. Sternberg 5c7bcda82c Set X-Influxdb-Version header on every request (even 404 requests)
Fixes #6756.
2016-06-03 10:29:50 -05:00
minkikim89 0e1e5e28a0 navbar-brand page refresh 2016-06-03 12:10:36 +09:00
Jonathan A. Sternberg 71c8e9e567 Refactor ExecuteQuery to take options as a struct
This allows us to add additional options to ExecuteQuery without
creating parameter bloat.

Removing the unused Series structs. Their necessity was removed by a
previous commit, but the structs were not removed yet.

Add another type of interrupt iterator that monitors the interrupt
channel and calls `Close()` on the iterator when the interrupt happens.
It will primarily be used for asynchronously closing the ReaderIterator,
but it will only close the read side of the connection properly. More
work needs to be done to allow closing the write side efficiently.
2016-06-01 12:30:52 -05:00
Edd Robinson f4fc905fa9 Reject timestamps too far in future 2016-05-27 11:07:48 +01:00
Jonathan A. Sternberg 907c88d4b5 Truncate the shard group end time if it exceeds MaxNanoTime
Related to #6599.
2016-05-25 21:25:10 -04:00
Edd Robinson ef4b36c063 Merge pull request #6695 from influxdata/er-trace
Add more context to recovered panics
2016-05-24 15:16:51 +01:00
Jonathan A. Sternberg baaa782c95 Modify the default retention policy name and make it configurable
The default retention policy name is changed to "autogen" instead of
"default" since it ends up being ambiguous when we tell a user to check
the default retention policy, it is uncertain if we are referring to the
default retention policy (which can be changed) or the retention policy
with the name "default".

Now the automatically generated retention policy name is "autogen".

The default retention policy is now also configurable through the
configuration file so an administrator can customize what they think
should be the default.

Fixes #3733.
2016-05-24 09:51:23 -04:00
David Norton 9d5c4a0f41 fix #6702: return correct required privileges
Change SelectStatement to return required privileges for all Sources.
2016-05-23 16:44:23 -04:00
Edd Robinson 67e793e512 Add more context to recovered panics 2016-05-23 12:24:05 +01:00
Jonathan A. Sternberg 7d64779e1d Merge pull request #6688 from influxdata/js-5906-dynamic-documentation-link
Dynamically update the documentation link in the admin UI
2016-05-21 16:41:11 -04:00
David Norton b32208a850 fix #2048: check that RPs exist before creating CQ 2016-05-20 12:40:36 -04:00
Jonathan A. Sternberg 26b0b4a104 Dynamically update the documentation link in the admin UI
Use the latest documentation by default if the server version can't be
found. If it can be found, update the documentation link to that
specific version so it always points at the exact documentation relevant
for the server version.

Fixes #5906.
2016-05-20 09:22:40 -04:00
Nathaniel Cook d460be3706 buffered chans, correct stats, drop to slow backends 2016-05-19 10:53:29 -06:00
Nathaniel Cook a7cac1337b writes happen concurrently via chanWriter 2016-05-19 10:53:29 -06:00
Nathaniel Cook 904e8cc825 add config for http timeout 2016-05-19 10:53:29 -06:00
Nathaniel Cook 18d0bfe056 fix some logging issues 2016-05-19 10:53:29 -06:00
Nathaniel Cook d6f40cdb55 initial http subscriber support 2016-05-19 10:53:29 -06:00
Jonathan A. Sternberg 5d9eae61b0 Add https-private-key option to httpd config
The HTTPS configuration for the httpd service only had an option to
specify the certificate file and the same file would be used for both
the certificate and private key file (they could be concatenated
together).

This adds an additional option to specify the files differently from
each other while still allowing the previous behavior. If only
`https-certificate` is specified, the httpd service will try to load the
private key from the `https-certificate` file. If a separate
`https-private-key` file is specified, the private key will be loaded
from there instead.

Fixes #1310.
2016-05-18 21:05:57 -04:00
Jonathan A. Sternberg 451a5205ef Support bound parameters in the parser
The parser can be passed a map of keys to literal values to be replaced
into the query. Parameters are preceded by a dollar sign (`$`). If a
parameter key is missing, an error is thrown by the parser.

Fixes #2926.
2016-05-18 20:10:15 -04:00
Edd Robinson 81dd13eb1d Don't return empty response bodies when gzip encoding requested (#6633)
* golint clean up

* Verb not possible for endpoint

* Don't specify Content-Encoding header for 204s
2016-05-18 22:24:09 +01:00
Edd Robinson ce0064cd88 Add stat for currently active write requests 2016-05-17 22:14:19 +01:00
Edd Robinson 36480c6271 Add client and server status code stats 2016-05-17 16:57:20 +01:00
David Norton 7ed188185d check admin exists instead of user count
When authenticating a request, check that an admin user exists instead
of checking for len(users) > 0. This prevents getting stuck with no
admin user and being unable to create one.
2016-05-17 08:07:37 -04:00
Jonathan A. Sternberg 4fab68b53b Teach the http service how to enforce connection limits
The http connection limit is for any HTTP operation and is independent
of the other connection limits. It should be set to a higher value than
the query limit. The difference between this and the query limit is it
will close out the connection immediately without any further
processing.

This is the equivalent of the `max_connections` option in PostgreSQL.

Also removes some unused config options from the cluster config.

Fixes #6559.
2016-05-12 16:25:16 -04:00
David Norton 2080cfb35d improve error message 2016-05-12 08:20:07 -04:00
David Norton d42f5f8062 require token expiration 2016-05-12 08:20:07 -04:00
David Norton 5842e206ae bump token expiration time up in test 2016-05-12 08:20:07 -04:00
David Norton 4e2605e98d feat #4448: add support for JWT tokens 2016-05-12 08:20:07 -04:00
Jason Wilder 6cc1a34704 Rename cluster package to coordinator 2016-05-11 11:41:05 -06:00
Jason Wilder d8490f1170 Merge pull request #6587 from influxdata/jw-validate-fields
Fix for merge values
2016-05-10 11:56:07 -06:00
Cory LaNou 4484182e86 remove old shard copier service 2016-05-10 08:14:51 -05:00
Jason Wilder adddb0e6b8 Reduce lock contention when creating shard group 2016-05-09 16:10:38 -06:00
Edd Robinson d35fa1ec97 Remove redundant windows build tags 2016-05-03 14:22:02 +01:00
Todd Persen 0ad2d6fd2c Merge pull request #5524 from seiflotfy/fix5516
Remove redundant error return of (Client) Database(name string)
2016-05-02 09:01:01 -07:00
Jonathan A. Sternberg ae10121c71 Add missing whitespace to the httpd log output 2016-05-01 19:30:11 -04:00
Seif Lotfy ced79acac5 Remove redundant error return from (Client) Database(name string) and (Client) Databases() 2016-04-30 17:04:38 -05:00
Jonathan A. Sternberg 6f61c0ea4a Add POST /query endpoint and warning messages for using GET with write operations
In order to follow REST a bit more carefully, all write operations
should go through a POST in the future. We still allow read operations
through either GET or POST (similar to the Graphite /render endpoint),
but write operations will trigger a returned warning as part of the JSON
response and will eventually return an error.

Also updates the Golang client libraries to always use POST instead of
GET.

Fixes #6290.
2016-04-29 09:00:23 -04:00
Vidhuran Harichandra Babu fedb7bfdbf added a button in admin ui to genearate URL for the query 2016-04-27 11:30:47 +03:00
Jon Seymour d2adf3f6f0 improve feedback from admin UI regarding submission of queries.
It can sometimes be difficult to determine if a query submitted by
the UI has been accepted for execution.

In particular, if a query that returns empty results is followed
by a query that takes a long time, then the green success message
from the first query may be misleadingly displayed to the user when,
in fact, the second query is still in progress.

To address this, we always hide the query error and success divs
before the query is submitted. Previously, just the results were
cleared.

Secondly, if the user re-runs a query expecting slightly different results,
it can also be unclear whether the second attempt to execute the command
is simply redisplaying the results of the previous query or the results of the
resubmission, this is particularly true if the queries involved
have short execution times.

To support the ability to distinguish these two cases, we have any attempt
to use the history arrow also clear the results and status divs. This
reduces the ambiguity about whether the next results display is, in fact,
the result of executing a new query.

Signed-off-by: Jon Seymour <jon@wildducktheories.com>
2016-04-26 12:17:05 -07:00
Jonathan A. Sternberg 62c66b788c Improve query sanitization to prevent a password leak in the logs
Sanitizing is now done through pattern matching rather than parsing the
query and replacing the password in the query. This prevents
accidentally redacting the wrong part of a query and revealing what the
password is through association.

Fixes #3883.
2016-04-22 11:27:09 -04:00
Stephen Gutekanst 9dc09c5257 Make logging output location more programmatically configurable (#6213)
This has various benefits:

- Users embedding InfluxDB within other Go programs can specify a different logger / prefix easily.
- More consistent with code used elsewhere in InfluxDB (e.g. services, other `run.Server.*` fields, etc).
- This is also more efficient, because it means `executeQuery` no longer allocates a single `*log.Logger` each time it is called.
2016-04-20 21:07:08 +01:00
Jonathan A. Sternberg 132b9e5496 Merge pull request #6414 from influxdata/js-prevent-query-pipelining
Send the "Connection: close" header for queries
2016-04-19 14:54:07 -04:00
David Norton c3813fc64f Merge pull request #6418 from influxdata/dn-auth
add WriteAuthorizer interface
2016-04-19 08:01:27 -04:00
David Norton 5e9ff7bc93 add WriteAuthorizer interface 2016-04-19 00:03:57 -04:00
Jonathan A. Sternberg 8bdd44107f Send the "Connection: close" header for queries
CloseNotify() is not compatible with pipelining. Adding this header
prevents the client from attempting to pipeline requests.
2016-04-18 15:38:50 -04:00
Aaron Knister 77db7ad618 Prevent goroutine leak in http service for persistent query connections 2016-04-18 15:11:04 -04:00
Jonathan A. Sternberg 42b68d915f Return a deprecated message when IF NOT EXISTS is used
The deprecated message is now attached to a new attribute returned with
the results. This message can then be read by clients to warn a user
about upcoming changes to the query engine.

The `influx` client has already been modified to read this message and
print it out for every format except CSV.

The first warning message is a deprecated message about removing `IF NOT
EXISTS` from `CREATE DATABASE`.

The message will also be printed to the server log.

Fixes #5707.
2016-04-15 09:17:59 -04:00
Chris Ramón a4dd61a2c7 adds shard duration normalization before comparing retention policies 2016-04-13 13:09:20 -05:00
Todd Persen 00ed9ebaf3 Remove TSDB listener accept message 2016-04-12 16:09:31 -07:00
Gunnar cfa50bbd3d Merge pull request #6228 from influxdata/ga-multiple-inputs
Support multiple listeners per input
2016-04-12 15:31:38 -07:00
Valeriy Simonov 57f0d8ffb8 Merge branch 'master' into opentsdb-log-less 2016-04-12 09:08:12 +05:00
Edd Robinson 7c34b1b3f8 Merge pull request #6281 from influxdata/er-fix-admin-docs
Use 'latest' alias for links to documentation
2016-04-11 22:35:12 +01:00
Jonathan A. Sternberg f4d74bcb30 Merge pull request #6241 from influxdata/js-refresh-admin-assets
Refresh admin assets
2016-04-11 12:32:02 -04:00
gunnaraasen eec8cc94c7 Support multiple OpenTSDB inputs 2016-04-11 08:47:38 -07:00
gunnaraasen fccd0943a8 Support multiple collectd inputs 2016-04-11 08:47:38 -07:00
Edd Robinson 44911bc5d8 Use 'latest' alias for links to documentation
Includes a fix for #6280
2016-04-11 14:56:53 +01:00
Cameron Sparr 23e2c40923 Reduce UDP service per-packet allocation size
This will reduce memory pressure and number of GC cycles, my results
sending 100,000 UDP points were:

- udp-payload-size=0: 242 GC cycles
- udp-payload-size=1500: 142 GC cycles
- udp-payload-size=0 (with change): 114 GC cycles
- udp-payload-size=1500 (with change): 112 GC cycles
2016-04-08 13:29:47 -06:00
Cory LaNou 7eaf7f8855 CreateShardGroup was incrementing meta data index even when it was idempotent 2016-04-07 11:39:45 -05:00
Edd Robinson 5327a75a6f Merge pull request #6216 from influxdata/er-scope-proto
Change protobuf package names to avoid clashes
2016-04-07 16:38:21 +01:00
Valeriy Simonov 0154a7008b Remove TSDB listener accept message 2016-04-07 17:15:56 +05:00
Regan Kuchan 87d3800e70 Refresh admin assets 2016-04-06 12:07:18 -04:00
mvadu 94524a7699 Fix failure on Windows
Windows does not allow a file to be renamed while its in use. f.Sync does not close the handle. It needs to be explicitly closed before renaming.
2016-04-06 07:32:37 +05:30
Edd Robinson 184257a10d Scope all internal protobuf packages 2016-04-05 13:54:21 +01:00
Jonathan A. Sternberg e22f098a11 Merge pull request #6187 from influxdata/js-plugin-executor
Cleanup QueryExecutor and add custom query execution code
2016-04-04 14:13:32 -04:00
Todd Persen dba3849c80 Merge pull request #6195 from jonseymour/jss-6193-master
Fix TypeError when processing empty results in admin UI
2016-04-04 11:09:47 -07:00
Jonathan A. Sternberg 37b63cedec Cleanup QueryExecutor and split statement execution code
The QueryExecutor had a lot of dead code made obsolete by the query
engine refactor that has now been removed. The TSDBStore interface has
also been cleaned up so we can have multiple implementations of this
(such as a local and remote version).

A StatementExecutor interface has been created for adding custom
functionality to the QueryExecutor that may not be available in the open
source version. The QueryExecutor delegate all statement execution to
the StatementExecutor and the QueryExecutor will only keep track of
housekeeping. Implementing additional queries is as simple as wrapping
the cluster.StatementExecutor struct or replacing it with something
completely different.

The PointsWriter in the QueryExecutor has been changed to a simple
interface that implements the one method needed by the query executor.
This is to allow different PointsWriter implementations to be used by
the QueryExecutor. It has also been moved into the StatementExecutor
instead.

The TSDBStore interface has now been modified to contain the code for
creating an IteratorCreator. This is so the underlying TSDBStore can
implement different ways of accessing the underlying shards rather than
always having to access each shard individually (such as batch
requests).

Remove the show servers handling. This isn't a valid command in the open
source version of InfluxDB anymore.

The QueryManager interface is now built into QueryExecutor and is no
longer necessary. The StatementExecutor and QueryExecutor split allows
task management to much more easily be built into QueryExecutor rather
than as a separate struct.
2016-04-04 13:27:17 -04:00
Jon Seymour 9c8bd466f7 fix TypeError when processing empty results in admin UI (#6193)
A change to the admin UI prevented the success message being displayed
for empty results. This change restores the original behaviour for
this case.

Signed-off-by: Jon Seymour <jon@wildducktheories.com>
2016-04-04 22:09:11 +10:00
Edd Robinson 712cc3f290 ShardGroup Duration is checked when recreating rp
Fixes #6178.
2016-04-01 11:59:41 +01:00
Cory LaNou 9bd60c3e07 remove node.json as well, change where we check for raft.db in startup 2016-03-31 17:15:58 -05:00
Cory LaNou 6c6fe87a40 add error message if detecting raft.db file 2016-03-31 17:15:42 -05:00
Cory LaNou a961ff9ebf minor restore fixes; fsync meta snapshots 2016-03-31 17:15:42 -05:00
Edd Robinson f68637205c Check ShardGroup Duration when recreating database
Fixes #6153
2016-03-31 18:11:50 +01:00
Jonathan A. Sternberg eb467d8d7f Merge pull request #6126 from influxdata/js-6115-chunked-query-support-mid-series
Support chunking in the middle of a series in the emitter
2016-03-30 20:59:24 -04:00
Jason Wilder 5e6247ac58 Fixed consistency level parameter being ignored
The http handler consistency level parameter was removed and hard-coded
to "any."  It needs to be read and passed through to the points writer.
2016-03-30 16:37:58 -06:00
joelegasse de94acc79d Merge pull request #6163 from influxdata/jl-content-type
Don't use Request.FormValue() in HTTP handler
2016-03-30 17:34:59 -04:00
Joe LeGasse c149906cbf Use r.URL.Query().Get() rather than r.FormValue()
FormValue() would attempt to parse the body of a request when the
content-type is set to `application/x-www-form-urlencoded`. The write
handler never wants url-encoded forms, and should only ever check the
URL for query parameters.

Fixes #6061
2016-03-30 17:16:10 -04:00
Jonathan A. Sternberg d2bc954236 Have an error overwrite previous results when aggregating results in the HTTP handler
It's possible for a single query to send multiple results that get
aggregated in the HTTP handler. If an earlier result passed in data and
a later result had an error, the error would be ignored.

Now an error for a statement will overwrite any previous results for
that statement.
2016-03-30 16:08:56 -04:00
Jonathan A. Sternberg 364dce36ab Support chunking in the middle of a series in the emitter
Limit the maximum size of the return value when chunking is not used to
prevent the server from going out of memory.

Fixes #6115.
2016-03-30 14:36:35 -04:00
Edd Robinson dcb6e318e9 Creating CQ with identical query does not error
Partially fixes #6094.

Prior to this when passing the same query and CQ name in a CREATE
CONTINUOUS QUERY command an error would be returned. This means the
command was not behaving in a similar way to other commands.a

Now when running the command with the same CQ name and query string no
error will be returned. Note, this change does not parse the query, it
simply compares a normalised query string to the existing one on the CQ.
2016-03-29 11:35:04 +01:00
Edd Robinson 828de361c6 Correct the CREATE RETENTION POLICY error behaviour
Partially addresses #6094.

Previously, when creating a retention policy only the name was
considered when deciding if the policy already existed. This meant that
adding a second policy with the same name but different duration or
replica factor returned the original policy and no error.

This commit fixes that and ensures that name, duration and replica
factor are all considered.
2016-03-29 11:07:26 +01:00
Cory LaNou fa05ae038a fix inconsistent route names 2016-03-25 14:10:31 -05:00
Cory LaNou 2a7abbab1f more external routes magic 2016-03-24 13:03:11 -05:00
Cory LaNou b3b4a3764a allow routes to be added outside the httpd package 2016-03-24 13:03:11 -05:00
Tait Clarridge 45b3e61ac7 Add configurable shard duration to retention policies
Allows configuration of shard group duration at database creation, and retention
policy create/alter time.

Query examples:

```
CREATE DATABASE testdb WITH DURATION 90d SHARD DURATION 30m NAME rp_testdb
CREATE RETENTION POLICY rp_testdb2 ON testdb DURATION INF REPLICATION 1 SHARD DURATION 30m
ALTER RETENTION POLICY rp_testdb2 ON testdb SHARD DURATION 1h
```

This can be useful with long duration retention policies with lots of data, where
you can split into smaller shards to relieve memory pressure.
2016-03-24 00:25:49 -04:00
Jonathan A. Sternberg b12cf04a73 Merge pull request #5950 from influxdata/js-5939-query-manager
Implement a query manager for running queries
2016-03-21 16:02:26 -04:00
David Norton 8afff49e2b Merge pull request #6072 from influxdata/dn-rename-func
rename CloneUserInfos to CloneUsers
2016-03-21 12:54:19 -04:00
David Norton b2d30e4593 rename CloneUserInfos to CloneUsers 2016-03-21 12:16:53 -04:00
Jonathan A. Sternberg 9246816c66 Implement http.CloseNotifier interface for gzipResponseWriter
This seems to have been an oversight since all of the response writers
are supposed to implement this interface, but the gzipResponseWriter
didn't implement this interface for some reason.
2016-03-21 12:07:07 -04:00
Jonathan A. Sternberg 117f62c33e Implement a simple task manager for queries
The currently running queries can be listed with the command
`SHOW QUERIES` and it will display the current commands that have been
run, the database they were run against, and how long they have been
running.
2016-03-21 12:06:06 -04:00
mvadu 8af9170c1d Add platform specific override for file rename.
In windows, it first deletes the target file (if it already exists).
2016-03-18 21:45:29 +05:30
Gunnar c6ff2588d1 Merge pull request #6025 from influxdata/ga-remove-json
Remove deprecated JSON write path
2016-03-16 14:17:23 -07:00
Todd Persen 58d1c8ded0 Display results of multiple queries in Admin UI, Not just the first query. 2016-03-16 13:22:37 -07:00
David Norton 515dadb1ef remove commented code 2016-03-16 08:58:55 -04:00
gunnaraasen d96eef4c52 Remove deprecated JSON write path 2016-03-15 19:52:41 -07:00