Commit Graph

624 Commits (fd840f242c7a3afc90e9198d68b4a97225f78299)

Author SHA1 Message Date
Jonathan A. Sternberg 62c66b788c Improve query sanitization to prevent a password leak in the logs
Sanitizing is now done through pattern matching rather than parsing the
query and replacing the password in the query. This prevents
accidentally redacting the wrong part of a query and revealing what the
password is through association.

Fixes #3883.
2016-04-22 11:27:09 -04:00
Stephen Gutekanst 9dc09c5257 Make logging output location more programmatically configurable (#6213)
This has various benefits:

- Users embedding InfluxDB within other Go programs can specify a different logger / prefix easily.
- More consistent with code used elsewhere in InfluxDB (e.g. services, other `run.Server.*` fields, etc).
- This is also more efficient, because it means `executeQuery` no longer allocates a single `*log.Logger` each time it is called.
2016-04-20 21:07:08 +01:00
Jonathan A. Sternberg 132b9e5496 Merge pull request #6414 from influxdata/js-prevent-query-pipelining
Send the "Connection: close" header for queries
2016-04-19 14:54:07 -04:00
David Norton c3813fc64f Merge pull request #6418 from influxdata/dn-auth
add WriteAuthorizer interface
2016-04-19 08:01:27 -04:00
David Norton 5e9ff7bc93 add WriteAuthorizer interface 2016-04-19 00:03:57 -04:00
Jonathan A. Sternberg 8bdd44107f Send the "Connection: close" header for queries
CloseNotify() is not compatible with pipelining. Adding this header
prevents the client from attempting to pipeline requests.
2016-04-18 15:38:50 -04:00
Aaron Knister 77db7ad618 Prevent goroutine leak in http service for persistent query connections 2016-04-18 15:11:04 -04:00
Jonathan A. Sternberg 42b68d915f Return a deprecated message when IF NOT EXISTS is used
The deprecated message is now attached to a new attribute returned with
the results. This message can then be read by clients to warn a user
about upcoming changes to the query engine.

The `influx` client has already been modified to read this message and
print it out for every format except CSV.

The first warning message is a deprecated message about removing `IF NOT
EXISTS` from `CREATE DATABASE`.

The message will also be printed to the server log.

Fixes #5707.
2016-04-15 09:17:59 -04:00
Chris Ramón a4dd61a2c7 adds shard duration normalization before comparing retention policies 2016-04-13 13:09:20 -05:00
Todd Persen 00ed9ebaf3 Remove TSDB listener accept message 2016-04-12 16:09:31 -07:00
Gunnar cfa50bbd3d Merge pull request #6228 from influxdata/ga-multiple-inputs
Support multiple listeners per input
2016-04-12 15:31:38 -07:00
Valeriy Simonov 57f0d8ffb8 Merge branch 'master' into opentsdb-log-less 2016-04-12 09:08:12 +05:00
Edd Robinson 7c34b1b3f8 Merge pull request #6281 from influxdata/er-fix-admin-docs
Use 'latest' alias for links to documentation
2016-04-11 22:35:12 +01:00
Jonathan A. Sternberg f4d74bcb30 Merge pull request #6241 from influxdata/js-refresh-admin-assets
Refresh admin assets
2016-04-11 12:32:02 -04:00
gunnaraasen eec8cc94c7 Support multiple OpenTSDB inputs 2016-04-11 08:47:38 -07:00
gunnaraasen fccd0943a8 Support multiple collectd inputs 2016-04-11 08:47:38 -07:00
Edd Robinson 44911bc5d8 Use 'latest' alias for links to documentation
Includes a fix for #6280
2016-04-11 14:56:53 +01:00
Cameron Sparr 23e2c40923 Reduce UDP service per-packet allocation size
This will reduce memory pressure and number of GC cycles, my results
sending 100,000 UDP points were:

- udp-payload-size=0: 242 GC cycles
- udp-payload-size=1500: 142 GC cycles
- udp-payload-size=0 (with change): 114 GC cycles
- udp-payload-size=1500 (with change): 112 GC cycles
2016-04-08 13:29:47 -06:00
Cory LaNou 7eaf7f8855 CreateShardGroup was incrementing meta data index even when it was idempotent 2016-04-07 11:39:45 -05:00
Edd Robinson 5327a75a6f Merge pull request #6216 from influxdata/er-scope-proto
Change protobuf package names to avoid clashes
2016-04-07 16:38:21 +01:00
Valeriy Simonov 0154a7008b Remove TSDB listener accept message 2016-04-07 17:15:56 +05:00
Regan Kuchan 87d3800e70 Refresh admin assets 2016-04-06 12:07:18 -04:00
mvadu 94524a7699 Fix failure on Windows
Windows does not allow a file to be renamed while its in use. f.Sync does not close the handle. It needs to be explicitly closed before renaming.
2016-04-06 07:32:37 +05:30
Edd Robinson 184257a10d Scope all internal protobuf packages 2016-04-05 13:54:21 +01:00
Jonathan A. Sternberg e22f098a11 Merge pull request #6187 from influxdata/js-plugin-executor
Cleanup QueryExecutor and add custom query execution code
2016-04-04 14:13:32 -04:00
Todd Persen dba3849c80 Merge pull request #6195 from jonseymour/jss-6193-master
Fix TypeError when processing empty results in admin UI
2016-04-04 11:09:47 -07:00
Jonathan A. Sternberg 37b63cedec Cleanup QueryExecutor and split statement execution code
The QueryExecutor had a lot of dead code made obsolete by the query
engine refactor that has now been removed. The TSDBStore interface has
also been cleaned up so we can have multiple implementations of this
(such as a local and remote version).

A StatementExecutor interface has been created for adding custom
functionality to the QueryExecutor that may not be available in the open
source version. The QueryExecutor delegate all statement execution to
the StatementExecutor and the QueryExecutor will only keep track of
housekeeping. Implementing additional queries is as simple as wrapping
the cluster.StatementExecutor struct or replacing it with something
completely different.

The PointsWriter in the QueryExecutor has been changed to a simple
interface that implements the one method needed by the query executor.
This is to allow different PointsWriter implementations to be used by
the QueryExecutor. It has also been moved into the StatementExecutor
instead.

The TSDBStore interface has now been modified to contain the code for
creating an IteratorCreator. This is so the underlying TSDBStore can
implement different ways of accessing the underlying shards rather than
always having to access each shard individually (such as batch
requests).

Remove the show servers handling. This isn't a valid command in the open
source version of InfluxDB anymore.

The QueryManager interface is now built into QueryExecutor and is no
longer necessary. The StatementExecutor and QueryExecutor split allows
task management to much more easily be built into QueryExecutor rather
than as a separate struct.
2016-04-04 13:27:17 -04:00
Jon Seymour 9c8bd466f7 fix TypeError when processing empty results in admin UI (#6193)
A change to the admin UI prevented the success message being displayed
for empty results. This change restores the original behaviour for
this case.

Signed-off-by: Jon Seymour <jon@wildducktheories.com>
2016-04-04 22:09:11 +10:00
Edd Robinson 712cc3f290 ShardGroup Duration is checked when recreating rp
Fixes #6178.
2016-04-01 11:59:41 +01:00
Cory LaNou 9bd60c3e07 remove node.json as well, change where we check for raft.db in startup 2016-03-31 17:15:58 -05:00
Cory LaNou 6c6fe87a40 add error message if detecting raft.db file 2016-03-31 17:15:42 -05:00
Cory LaNou a961ff9ebf minor restore fixes; fsync meta snapshots 2016-03-31 17:15:42 -05:00
Edd Robinson f68637205c Check ShardGroup Duration when recreating database
Fixes #6153
2016-03-31 18:11:50 +01:00
Jonathan A. Sternberg eb467d8d7f Merge pull request #6126 from influxdata/js-6115-chunked-query-support-mid-series
Support chunking in the middle of a series in the emitter
2016-03-30 20:59:24 -04:00
Jason Wilder 5e6247ac58 Fixed consistency level parameter being ignored
The http handler consistency level parameter was removed and hard-coded
to "any."  It needs to be read and passed through to the points writer.
2016-03-30 16:37:58 -06:00
joelegasse de94acc79d Merge pull request #6163 from influxdata/jl-content-type
Don't use Request.FormValue() in HTTP handler
2016-03-30 17:34:59 -04:00
Joe LeGasse c149906cbf Use r.URL.Query().Get() rather than r.FormValue()
FormValue() would attempt to parse the body of a request when the
content-type is set to `application/x-www-form-urlencoded`. The write
handler never wants url-encoded forms, and should only ever check the
URL for query parameters.

Fixes #6061
2016-03-30 17:16:10 -04:00
Jonathan A. Sternberg d2bc954236 Have an error overwrite previous results when aggregating results in the HTTP handler
It's possible for a single query to send multiple results that get
aggregated in the HTTP handler. If an earlier result passed in data and
a later result had an error, the error would be ignored.

Now an error for a statement will overwrite any previous results for
that statement.
2016-03-30 16:08:56 -04:00
Jonathan A. Sternberg 364dce36ab Support chunking in the middle of a series in the emitter
Limit the maximum size of the return value when chunking is not used to
prevent the server from going out of memory.

Fixes #6115.
2016-03-30 14:36:35 -04:00
Edd Robinson dcb6e318e9 Creating CQ with identical query does not error
Partially fixes #6094.

Prior to this when passing the same query and CQ name in a CREATE
CONTINUOUS QUERY command an error would be returned. This means the
command was not behaving in a similar way to other commands.a

Now when running the command with the same CQ name and query string no
error will be returned. Note, this change does not parse the query, it
simply compares a normalised query string to the existing one on the CQ.
2016-03-29 11:35:04 +01:00
Edd Robinson 828de361c6 Correct the CREATE RETENTION POLICY error behaviour
Partially addresses #6094.

Previously, when creating a retention policy only the name was
considered when deciding if the policy already existed. This meant that
adding a second policy with the same name but different duration or
replica factor returned the original policy and no error.

This commit fixes that and ensures that name, duration and replica
factor are all considered.
2016-03-29 11:07:26 +01:00
Cory LaNou fa05ae038a fix inconsistent route names 2016-03-25 14:10:31 -05:00
Cory LaNou 2a7abbab1f more external routes magic 2016-03-24 13:03:11 -05:00
Cory LaNou b3b4a3764a allow routes to be added outside the httpd package 2016-03-24 13:03:11 -05:00
Tait Clarridge 45b3e61ac7 Add configurable shard duration to retention policies
Allows configuration of shard group duration at database creation, and retention
policy create/alter time.

Query examples:

```
CREATE DATABASE testdb WITH DURATION 90d SHARD DURATION 30m NAME rp_testdb
CREATE RETENTION POLICY rp_testdb2 ON testdb DURATION INF REPLICATION 1 SHARD DURATION 30m
ALTER RETENTION POLICY rp_testdb2 ON testdb SHARD DURATION 1h
```

This can be useful with long duration retention policies with lots of data, where
you can split into smaller shards to relieve memory pressure.
2016-03-24 00:25:49 -04:00
Jonathan A. Sternberg b12cf04a73 Merge pull request #5950 from influxdata/js-5939-query-manager
Implement a query manager for running queries
2016-03-21 16:02:26 -04:00
David Norton 8afff49e2b Merge pull request #6072 from influxdata/dn-rename-func
rename CloneUserInfos to CloneUsers
2016-03-21 12:54:19 -04:00
David Norton b2d30e4593 rename CloneUserInfos to CloneUsers 2016-03-21 12:16:53 -04:00
Jonathan A. Sternberg 9246816c66 Implement http.CloseNotifier interface for gzipResponseWriter
This seems to have been an oversight since all of the response writers
are supposed to implement this interface, but the gzipResponseWriter
didn't implement this interface for some reason.
2016-03-21 12:07:07 -04:00
Jonathan A. Sternberg 117f62c33e Implement a simple task manager for queries
The currently running queries can be listed with the command
`SHOW QUERIES` and it will display the current commands that have been
run, the database they were run against, and how long they have been
running.
2016-03-21 12:06:06 -04:00