Commit Graph

720 Commits (3ecc913c887f49c86579cedc24e113d4534fe13d)

Author SHA1 Message Date
Jonathan A. Sternberg 4f37bc5a40 Support specifying a retention policy for the graphite service
The graphite service will attempt to create the retention policy and use
it. If the retention policy doesn't exist, it will be created with the
default options.

Fixes #5655.
2016-06-08 17:01:44 -05:00
Joe LeGasse 647210c57a Add TruncatedAt field to meta.ShardGroupInfo 2016-06-07 18:24:24 -04:00
Jonathan A. Sternberg b972c220aa Merge pull request #6757 from influxdata/js-refactor-execute-query
Refactor ExecuteQuery to take options as a struct
2016-06-07 10:35:52 -05:00
Jonathan A. Sternberg 1e562408f1 Merge pull request #6779 from joe2far/patch-1
Fixed typo in docstring
2016-06-07 08:26:23 -05:00
Jonathan A. Sternberg b8e22d9d79 Merge pull request #6586 from influxdata/js-3733-rename-default-retention-policy
Modify the default retention policy name and make it configurable
2016-06-06 15:05:29 -05:00
Rubycut a849f42cc2 Lock auth properly, code suggested by @jwilder 2016-06-04 09:05:10 +02:00
Joe Farrell bb8b08e2ca Fixed typo in docstring 2016-06-03 18:01:04 +01:00
Jonathan A. Sternberg 5c7bcda82c Set X-Influxdb-Version header on every request (even 404 requests)
Fixes #6756.
2016-06-03 10:29:50 -05:00
minkikim89 0e1e5e28a0 navbar-brand page refresh 2016-06-03 12:10:36 +09:00
Jonathan A. Sternberg 71c8e9e567 Refactor ExecuteQuery to take options as a struct
This allows us to add additional options to ExecuteQuery without
creating parameter bloat.

Removing the unused Series structs. Their necessity was removed by a
previous commit, but the structs were not removed yet.

Add another type of interrupt iterator that monitors the interrupt
channel and calls `Close()` on the iterator when the interrupt happens.
It will primarily be used for asynchronously closing the ReaderIterator,
but it will only close the read side of the connection properly. More
work needs to be done to allow closing the write side efficiently.
2016-06-01 12:30:52 -05:00
Edd Robinson f4fc905fa9 Reject timestamps too far in future 2016-05-27 11:07:48 +01:00
Jonathan A. Sternberg 907c88d4b5 Truncate the shard group end time if it exceeds MaxNanoTime
Related to #6599.
2016-05-25 21:25:10 -04:00
Edd Robinson ef4b36c063 Merge pull request #6695 from influxdata/er-trace
Add more context to recovered panics
2016-05-24 15:16:51 +01:00
Jonathan A. Sternberg baaa782c95 Modify the default retention policy name and make it configurable
The default retention policy name is changed to "autogen" instead of
"default" since it ends up being ambiguous when we tell a user to check
the default retention policy, it is uncertain if we are referring to the
default retention policy (which can be changed) or the retention policy
with the name "default".

Now the automatically generated retention policy name is "autogen".

The default retention policy is now also configurable through the
configuration file so an administrator can customize what they think
should be the default.

Fixes #3733.
2016-05-24 09:51:23 -04:00
David Norton 9d5c4a0f41 fix #6702: return correct required privileges
Change SelectStatement to return required privileges for all Sources.
2016-05-23 16:44:23 -04:00
Edd Robinson 67e793e512 Add more context to recovered panics 2016-05-23 12:24:05 +01:00
Jonathan A. Sternberg 7d64779e1d Merge pull request #6688 from influxdata/js-5906-dynamic-documentation-link
Dynamically update the documentation link in the admin UI
2016-05-21 16:41:11 -04:00
David Norton b32208a850 fix #2048: check that RPs exist before creating CQ 2016-05-20 12:40:36 -04:00
Jonathan A. Sternberg 26b0b4a104 Dynamically update the documentation link in the admin UI
Use the latest documentation by default if the server version can't be
found. If it can be found, update the documentation link to that
specific version so it always points at the exact documentation relevant
for the server version.

Fixes #5906.
2016-05-20 09:22:40 -04:00
Nathaniel Cook d460be3706 buffered chans, correct stats, drop to slow backends 2016-05-19 10:53:29 -06:00
Nathaniel Cook a7cac1337b writes happen concurrently via chanWriter 2016-05-19 10:53:29 -06:00
Nathaniel Cook 904e8cc825 add config for http timeout 2016-05-19 10:53:29 -06:00
Nathaniel Cook 18d0bfe056 fix some logging issues 2016-05-19 10:53:29 -06:00
Nathaniel Cook d6f40cdb55 initial http subscriber support 2016-05-19 10:53:29 -06:00
Jonathan A. Sternberg 5d9eae61b0 Add https-private-key option to httpd config
The HTTPS configuration for the httpd service only had an option to
specify the certificate file and the same file would be used for both
the certificate and private key file (they could be concatenated
together).

This adds an additional option to specify the files differently from
each other while still allowing the previous behavior. If only
`https-certificate` is specified, the httpd service will try to load the
private key from the `https-certificate` file. If a separate
`https-private-key` file is specified, the private key will be loaded
from there instead.

Fixes #1310.
2016-05-18 21:05:57 -04:00
Jonathan A. Sternberg 451a5205ef Support bound parameters in the parser
The parser can be passed a map of keys to literal values to be replaced
into the query. Parameters are preceded by a dollar sign (`$`). If a
parameter key is missing, an error is thrown by the parser.

Fixes #2926.
2016-05-18 20:10:15 -04:00
Edd Robinson 81dd13eb1d Don't return empty response bodies when gzip encoding requested (#6633)
* golint clean up

* Verb not possible for endpoint

* Don't specify Content-Encoding header for 204s
2016-05-18 22:24:09 +01:00
Edd Robinson ce0064cd88 Add stat for currently active write requests 2016-05-17 22:14:19 +01:00
Edd Robinson 36480c6271 Add client and server status code stats 2016-05-17 16:57:20 +01:00
David Norton 7ed188185d check admin exists instead of user count
When authenticating a request, check that an admin user exists instead
of checking for len(users) > 0. This prevents getting stuck with no
admin user and being unable to create one.
2016-05-17 08:07:37 -04:00
Jonathan A. Sternberg 4fab68b53b Teach the http service how to enforce connection limits
The http connection limit is for any HTTP operation and is independent
of the other connection limits. It should be set to a higher value than
the query limit. The difference between this and the query limit is it
will close out the connection immediately without any further
processing.

This is the equivalent of the `max_connections` option in PostgreSQL.

Also removes some unused config options from the cluster config.

Fixes #6559.
2016-05-12 16:25:16 -04:00
David Norton 2080cfb35d improve error message 2016-05-12 08:20:07 -04:00
David Norton d42f5f8062 require token expiration 2016-05-12 08:20:07 -04:00
David Norton 5842e206ae bump token expiration time up in test 2016-05-12 08:20:07 -04:00
David Norton 4e2605e98d feat #4448: add support for JWT tokens 2016-05-12 08:20:07 -04:00
Jason Wilder 6cc1a34704 Rename cluster package to coordinator 2016-05-11 11:41:05 -06:00
Jason Wilder d8490f1170 Merge pull request #6587 from influxdata/jw-validate-fields
Fix for merge values
2016-05-10 11:56:07 -06:00
Cory LaNou 4484182e86 remove old shard copier service 2016-05-10 08:14:51 -05:00
Jason Wilder adddb0e6b8 Reduce lock contention when creating shard group 2016-05-09 16:10:38 -06:00
Edd Robinson d35fa1ec97 Remove redundant windows build tags 2016-05-03 14:22:02 +01:00
Todd Persen 0ad2d6fd2c Merge pull request #5524 from seiflotfy/fix5516
Remove redundant error return of (Client) Database(name string)
2016-05-02 09:01:01 -07:00
Jonathan A. Sternberg ae10121c71 Add missing whitespace to the httpd log output 2016-05-01 19:30:11 -04:00
Seif Lotfy ced79acac5 Remove redundant error return from (Client) Database(name string) and (Client) Databases() 2016-04-30 17:04:38 -05:00
Jonathan A. Sternberg 6f61c0ea4a Add POST /query endpoint and warning messages for using GET with write operations
In order to follow REST a bit more carefully, all write operations
should go through a POST in the future. We still allow read operations
through either GET or POST (similar to the Graphite /render endpoint),
but write operations will trigger a returned warning as part of the JSON
response and will eventually return an error.

Also updates the Golang client libraries to always use POST instead of
GET.

Fixes #6290.
2016-04-29 09:00:23 -04:00
Vidhuran Harichandra Babu fedb7bfdbf added a button in admin ui to genearate URL for the query 2016-04-27 11:30:47 +03:00
Jon Seymour d2adf3f6f0 improve feedback from admin UI regarding submission of queries.
It can sometimes be difficult to determine if a query submitted by
the UI has been accepted for execution.

In particular, if a query that returns empty results is followed
by a query that takes a long time, then the green success message
from the first query may be misleadingly displayed to the user when,
in fact, the second query is still in progress.

To address this, we always hide the query error and success divs
before the query is submitted. Previously, just the results were
cleared.

Secondly, if the user re-runs a query expecting slightly different results,
it can also be unclear whether the second attempt to execute the command
is simply redisplaying the results of the previous query or the results of the
resubmission, this is particularly true if the queries involved
have short execution times.

To support the ability to distinguish these two cases, we have any attempt
to use the history arrow also clear the results and status divs. This
reduces the ambiguity about whether the next results display is, in fact,
the result of executing a new query.

Signed-off-by: Jon Seymour <jon@wildducktheories.com>
2016-04-26 12:17:05 -07:00
Jonathan A. Sternberg 62c66b788c Improve query sanitization to prevent a password leak in the logs
Sanitizing is now done through pattern matching rather than parsing the
query and replacing the password in the query. This prevents
accidentally redacting the wrong part of a query and revealing what the
password is through association.

Fixes #3883.
2016-04-22 11:27:09 -04:00
Stephen Gutekanst 9dc09c5257 Make logging output location more programmatically configurable (#6213)
This has various benefits:

- Users embedding InfluxDB within other Go programs can specify a different logger / prefix easily.
- More consistent with code used elsewhere in InfluxDB (e.g. services, other `run.Server.*` fields, etc).
- This is also more efficient, because it means `executeQuery` no longer allocates a single `*log.Logger` each time it is called.
2016-04-20 21:07:08 +01:00
Jonathan A. Sternberg 132b9e5496 Merge pull request #6414 from influxdata/js-prevent-query-pipelining
Send the "Connection: close" header for queries
2016-04-19 14:54:07 -04:00
David Norton c3813fc64f Merge pull request #6418 from influxdata/dn-auth
add WriteAuthorizer interface
2016-04-19 08:01:27 -04:00
David Norton 5e9ff7bc93 add WriteAuthorizer interface 2016-04-19 00:03:57 -04:00
Jonathan A. Sternberg 8bdd44107f Send the "Connection: close" header for queries
CloseNotify() is not compatible with pipelining. Adding this header
prevents the client from attempting to pipeline requests.
2016-04-18 15:38:50 -04:00
Aaron Knister 77db7ad618 Prevent goroutine leak in http service for persistent query connections 2016-04-18 15:11:04 -04:00
Jonathan A. Sternberg 42b68d915f Return a deprecated message when IF NOT EXISTS is used
The deprecated message is now attached to a new attribute returned with
the results. This message can then be read by clients to warn a user
about upcoming changes to the query engine.

The `influx` client has already been modified to read this message and
print it out for every format except CSV.

The first warning message is a deprecated message about removing `IF NOT
EXISTS` from `CREATE DATABASE`.

The message will also be printed to the server log.

Fixes #5707.
2016-04-15 09:17:59 -04:00
Chris Ramón a4dd61a2c7 adds shard duration normalization before comparing retention policies 2016-04-13 13:09:20 -05:00
Todd Persen 00ed9ebaf3 Remove TSDB listener accept message 2016-04-12 16:09:31 -07:00
Gunnar cfa50bbd3d Merge pull request #6228 from influxdata/ga-multiple-inputs
Support multiple listeners per input
2016-04-12 15:31:38 -07:00
Valeriy Simonov 57f0d8ffb8 Merge branch 'master' into opentsdb-log-less 2016-04-12 09:08:12 +05:00
Edd Robinson 7c34b1b3f8 Merge pull request #6281 from influxdata/er-fix-admin-docs
Use 'latest' alias for links to documentation
2016-04-11 22:35:12 +01:00
Jonathan A. Sternberg f4d74bcb30 Merge pull request #6241 from influxdata/js-refresh-admin-assets
Refresh admin assets
2016-04-11 12:32:02 -04:00
gunnaraasen eec8cc94c7 Support multiple OpenTSDB inputs 2016-04-11 08:47:38 -07:00
gunnaraasen fccd0943a8 Support multiple collectd inputs 2016-04-11 08:47:38 -07:00
Edd Robinson 44911bc5d8 Use 'latest' alias for links to documentation
Includes a fix for #6280
2016-04-11 14:56:53 +01:00
Cameron Sparr 23e2c40923 Reduce UDP service per-packet allocation size
This will reduce memory pressure and number of GC cycles, my results
sending 100,000 UDP points were:

- udp-payload-size=0: 242 GC cycles
- udp-payload-size=1500: 142 GC cycles
- udp-payload-size=0 (with change): 114 GC cycles
- udp-payload-size=1500 (with change): 112 GC cycles
2016-04-08 13:29:47 -06:00
Cory LaNou 7eaf7f8855 CreateShardGroup was incrementing meta data index even when it was idempotent 2016-04-07 11:39:45 -05:00
Edd Robinson 5327a75a6f Merge pull request #6216 from influxdata/er-scope-proto
Change protobuf package names to avoid clashes
2016-04-07 16:38:21 +01:00
Valeriy Simonov 0154a7008b Remove TSDB listener accept message 2016-04-07 17:15:56 +05:00
Regan Kuchan 87d3800e70 Refresh admin assets 2016-04-06 12:07:18 -04:00
mvadu 94524a7699 Fix failure on Windows
Windows does not allow a file to be renamed while its in use. f.Sync does not close the handle. It needs to be explicitly closed before renaming.
2016-04-06 07:32:37 +05:30
Edd Robinson 184257a10d Scope all internal protobuf packages 2016-04-05 13:54:21 +01:00
Jonathan A. Sternberg e22f098a11 Merge pull request #6187 from influxdata/js-plugin-executor
Cleanup QueryExecutor and add custom query execution code
2016-04-04 14:13:32 -04:00
Todd Persen dba3849c80 Merge pull request #6195 from jonseymour/jss-6193-master
Fix TypeError when processing empty results in admin UI
2016-04-04 11:09:47 -07:00
Jonathan A. Sternberg 37b63cedec Cleanup QueryExecutor and split statement execution code
The QueryExecutor had a lot of dead code made obsolete by the query
engine refactor that has now been removed. The TSDBStore interface has
also been cleaned up so we can have multiple implementations of this
(such as a local and remote version).

A StatementExecutor interface has been created for adding custom
functionality to the QueryExecutor that may not be available in the open
source version. The QueryExecutor delegate all statement execution to
the StatementExecutor and the QueryExecutor will only keep track of
housekeeping. Implementing additional queries is as simple as wrapping
the cluster.StatementExecutor struct or replacing it with something
completely different.

The PointsWriter in the QueryExecutor has been changed to a simple
interface that implements the one method needed by the query executor.
This is to allow different PointsWriter implementations to be used by
the QueryExecutor. It has also been moved into the StatementExecutor
instead.

The TSDBStore interface has now been modified to contain the code for
creating an IteratorCreator. This is so the underlying TSDBStore can
implement different ways of accessing the underlying shards rather than
always having to access each shard individually (such as batch
requests).

Remove the show servers handling. This isn't a valid command in the open
source version of InfluxDB anymore.

The QueryManager interface is now built into QueryExecutor and is no
longer necessary. The StatementExecutor and QueryExecutor split allows
task management to much more easily be built into QueryExecutor rather
than as a separate struct.
2016-04-04 13:27:17 -04:00
Jon Seymour 9c8bd466f7 fix TypeError when processing empty results in admin UI (#6193)
A change to the admin UI prevented the success message being displayed
for empty results. This change restores the original behaviour for
this case.

Signed-off-by: Jon Seymour <jon@wildducktheories.com>
2016-04-04 22:09:11 +10:00
Edd Robinson 712cc3f290 ShardGroup Duration is checked when recreating rp
Fixes #6178.
2016-04-01 11:59:41 +01:00
Cory LaNou 9bd60c3e07 remove node.json as well, change where we check for raft.db in startup 2016-03-31 17:15:58 -05:00
Cory LaNou 6c6fe87a40 add error message if detecting raft.db file 2016-03-31 17:15:42 -05:00
Cory LaNou a961ff9ebf minor restore fixes; fsync meta snapshots 2016-03-31 17:15:42 -05:00
Edd Robinson f68637205c Check ShardGroup Duration when recreating database
Fixes #6153
2016-03-31 18:11:50 +01:00
Jonathan A. Sternberg eb467d8d7f Merge pull request #6126 from influxdata/js-6115-chunked-query-support-mid-series
Support chunking in the middle of a series in the emitter
2016-03-30 20:59:24 -04:00
Jason Wilder 5e6247ac58 Fixed consistency level parameter being ignored
The http handler consistency level parameter was removed and hard-coded
to "any."  It needs to be read and passed through to the points writer.
2016-03-30 16:37:58 -06:00
joelegasse de94acc79d Merge pull request #6163 from influxdata/jl-content-type
Don't use Request.FormValue() in HTTP handler
2016-03-30 17:34:59 -04:00
Joe LeGasse c149906cbf Use r.URL.Query().Get() rather than r.FormValue()
FormValue() would attempt to parse the body of a request when the
content-type is set to `application/x-www-form-urlencoded`. The write
handler never wants url-encoded forms, and should only ever check the
URL for query parameters.

Fixes #6061
2016-03-30 17:16:10 -04:00
Jonathan A. Sternberg d2bc954236 Have an error overwrite previous results when aggregating results in the HTTP handler
It's possible for a single query to send multiple results that get
aggregated in the HTTP handler. If an earlier result passed in data and
a later result had an error, the error would be ignored.

Now an error for a statement will overwrite any previous results for
that statement.
2016-03-30 16:08:56 -04:00
Jonathan A. Sternberg 364dce36ab Support chunking in the middle of a series in the emitter
Limit the maximum size of the return value when chunking is not used to
prevent the server from going out of memory.

Fixes #6115.
2016-03-30 14:36:35 -04:00
Edd Robinson dcb6e318e9 Creating CQ with identical query does not error
Partially fixes #6094.

Prior to this when passing the same query and CQ name in a CREATE
CONTINUOUS QUERY command an error would be returned. This means the
command was not behaving in a similar way to other commands.a

Now when running the command with the same CQ name and query string no
error will be returned. Note, this change does not parse the query, it
simply compares a normalised query string to the existing one on the CQ.
2016-03-29 11:35:04 +01:00
Edd Robinson 828de361c6 Correct the CREATE RETENTION POLICY error behaviour
Partially addresses #6094.

Previously, when creating a retention policy only the name was
considered when deciding if the policy already existed. This meant that
adding a second policy with the same name but different duration or
replica factor returned the original policy and no error.

This commit fixes that and ensures that name, duration and replica
factor are all considered.
2016-03-29 11:07:26 +01:00
Cory LaNou fa05ae038a fix inconsistent route names 2016-03-25 14:10:31 -05:00
Cory LaNou 2a7abbab1f more external routes magic 2016-03-24 13:03:11 -05:00
Cory LaNou b3b4a3764a allow routes to be added outside the httpd package 2016-03-24 13:03:11 -05:00
Tait Clarridge 45b3e61ac7 Add configurable shard duration to retention policies
Allows configuration of shard group duration at database creation, and retention
policy create/alter time.

Query examples:

```
CREATE DATABASE testdb WITH DURATION 90d SHARD DURATION 30m NAME rp_testdb
CREATE RETENTION POLICY rp_testdb2 ON testdb DURATION INF REPLICATION 1 SHARD DURATION 30m
ALTER RETENTION POLICY rp_testdb2 ON testdb SHARD DURATION 1h
```

This can be useful with long duration retention policies with lots of data, where
you can split into smaller shards to relieve memory pressure.
2016-03-24 00:25:49 -04:00
Jonathan A. Sternberg b12cf04a73 Merge pull request #5950 from influxdata/js-5939-query-manager
Implement a query manager for running queries
2016-03-21 16:02:26 -04:00
David Norton 8afff49e2b Merge pull request #6072 from influxdata/dn-rename-func
rename CloneUserInfos to CloneUsers
2016-03-21 12:54:19 -04:00
David Norton b2d30e4593 rename CloneUserInfos to CloneUsers 2016-03-21 12:16:53 -04:00
Jonathan A. Sternberg 9246816c66 Implement http.CloseNotifier interface for gzipResponseWriter
This seems to have been an oversight since all of the response writers
are supposed to implement this interface, but the gzipResponseWriter
didn't implement this interface for some reason.
2016-03-21 12:07:07 -04:00
Jonathan A. Sternberg 117f62c33e Implement a simple task manager for queries
The currently running queries can be listed with the command
`SHOW QUERIES` and it will display the current commands that have been
run, the database they were run against, and how long they have been
running.
2016-03-21 12:06:06 -04:00
mvadu 8af9170c1d Add platform specific override for file rename.
In windows, it first deletes the target file (if it already exists).
2016-03-18 21:45:29 +05:30
Gunnar c6ff2588d1 Merge pull request #6025 from influxdata/ga-remove-json
Remove deprecated JSON write path
2016-03-16 14:17:23 -07:00
Todd Persen 58d1c8ded0 Display results of multiple queries in Admin UI, Not just the first query. 2016-03-16 13:22:37 -07:00
David Norton 515dadb1ef remove commented code 2016-03-16 08:58:55 -04:00