Commit Graph

816 Commits (7e0ed1f5e51f92edc97983895898b3a05bb85c1e)

Author SHA1 Message Date
Marc 7d277d0def Add support for collectd security policy
There are 2 new keys in the configuration file.
- security-level: "none", "sign", or "encrypt".
- auth-file: The location of the user/password file.

Please see the collectd network doc for more details.
2016-11-08 08:42:09 +08:00
Marc 0260dfb575 Replace kimor79/gollectd with collectd.org
At this point this is a like-for-like swap.

collectd.org is more active and supports more features.
2016-11-08 08:24:57 +08:00
Jason Wilder b6863b04d1 Disable/deprecate admin interface 2016-11-07 12:04:51 -07:00
Cory LaNou e112de2ee0 Merge pull request #7554 from influxdata/cjl-godeps
update latest dependencies with Godeps
2016-11-07 09:10:30 -06:00
Cory LaNou 3774d5e734
update latest dependencies with Godeps 2016-11-07 08:49:29 -06:00
Cory LaNou 6e290040bb
remove SetDefaultRetentionPolicy method from meta.Client 2016-11-03 09:39:41 -05:00
Cory LaNou cd272ce6c3 fix retention policy creation inconsistencies 2016-11-03 09:09:43 -05:00
Jonathan A. Sternberg 3e29d3d9ca Truncate the version string when linking to the documentation
The admin console would dynamically discover the version from the
InfluxDB server, but for patch releases, it included the patch in the
link to the documentation and that wasn't a valid link.

Truncate the version so the documentation url is correct since we only
do documentation for `major.minor`.
2016-10-27 16:40:08 -05:00
Jason Wilder 6fa145943a Merge pull request #7483 from hvnsweeting/doc-fixes
Doc fixes
2016-10-24 11:38:59 -06:00
Cory LaNou 5b72b874d8
remove ProcessContinousQueries from httpd endpoint 2016-10-20 11:22:36 -05:00
Hung Nguyen Viet 2cb6c64787 Sections headers should be all capitals 2016-10-20 18:57:13 +07:00
Hung Nguyen Viet 1f26984181 Graphite 2016-10-20 18:57:13 +07:00
Hung Nguyen Viet cc7b61d385 fix grammar error 2016-10-20 18:57:13 +07:00
Hung Nguyen Viet 179f5b02cc TSM is the only engine now 2016-10-20 18:57:05 +07:00
Hung Nguyen Viet 1f31e8b707 clarify 2016-10-20 18:48:56 +07:00
Hung Nguyen Viet 4751ca1f43 use markdown 2016-10-20 18:47:54 +07:00
Hung Nguyen Viet dcb494ded5 use plural 2016-10-20 18:47:54 +07:00
Edd Robinson 168c91cc67 graphite service should retry database creation 2016-10-18 16:50:07 +01:00
Edd Robinson 73c267abc1 udp service should retry database creation 2016-10-18 16:50:07 +01:00
Edd Robinson 0470f38478 opentsdb should retry creating its database 2016-10-18 16:50:07 +01:00
Edd Robinson cc60647827 collectd should retry creating database 2016-10-18 16:50:07 +01:00
Edd Robinson 6c149b31ef Remove unused exported methods 2016-10-18 16:19:44 +01:00
Edd Robinson ad2d33f859 Ensure input services can be safely opened and closed 2016-10-18 10:35:38 +01:00
zhexuany 931a6c6d08 fixed two typo 2016-10-13 13:43:38 +08:00
Jason Wilder bbecb3f03d Drop points that would execeed limits
This changes the behavior of the max-series-per-database and
max-values-per-tag limits to drop points that would exceed the limits
and allow the remaining points to be written.  Previously, the whole
batch would fail and return and 500 error to the client.

This now will write the allow points and return a `partial write`
error indicating some of the points were dropped, how many were
dropped and one of the problem measureent and tags.
2016-10-10 11:42:15 -06:00
Jason Wilder ebf50d06ca Merge branch '1.0' into jw-merge-102 2016-10-06 09:51:35 -06:00
Jason Wilder ea21588b9e Fix subscriber service dropping writes under high write load
The subscriber write goroutine would drop points if the write load
was higher than it could process.  This could happen with a just
a few writers to the server.

Instead, process the channel with multiple writers to avoid dropping
writes so easily.  This also adds some config options to control how
large the channel buffer is as well as how many goroutines are started.

Fixes #7330
2016-10-04 14:47:36 -06:00
Nathaniel Cook 91645c05cf Merge pull request #7392 from mglazer/feature/https-subscriber
Fix Kapacitor Issue #942: HTTPS subscriptions don't work
2016-10-04 14:45:05 -06:00
Mike Glazer fa3609b251 Fix Kapacitor Issue #942: HTTPS subscriptions don't work 2016-10-04 20:58:31 +02:00
Edd Robinson 537316eb7b Ensure pprof-enabled config option is respected 2016-09-30 18:41:38 +01:00
Joe LeGasse 9d26cc0732 Stablize the sorting of meta.ShardGroupInfos 2016-09-30 13:58:03 +01:00
Jason Wilder 6e7da3d24a Merge pull request #7379 from influxdata/jw-meta-clone
Don't clone the cached meta store
2016-09-28 11:52:43 -06:00
Jonathan A. Sternberg bb8c4efa95 Do not automatically reset the shard duration when using ALTER RETENTION POLICY 2016-09-28 12:26:03 -05:00
Jason Wilder bc00490b92 Don't clone the cached meta store
The cacheData is read-only and does not need to be cloned.  Cloning
it for read operations can be expensive with lots of shards/database/RPs/etc.
2016-09-28 10:35:59 -06:00
Jonathan A. Sternberg 3afdf3cd94 Merge tag 'v1.0.1' 2016-09-27 17:53:33 -05:00
joelegasse 1eda0277a7 Merge pull request #7365 from influxdata/jl-shardgroup-sort
Stablize the sorting of meta.ShardGroupInfos
2016-09-27 11:02:47 -04:00
Jason Wilder 14b0f6e496 Fix panic in subscriber.Statistics
The tags passed into Statistics() calls are not supposed to be modified.
The balanceWriter in subscribers tried to modify them triggering a panic
because they can be nil.
2016-09-16 11:49:15 -06:00
Jonathan A. Sternberg 477d6231db Update source files to pass vet checks for go 1.7
The vet checks for some files did not pass for go 1.7. As part of a
preliminary start to making go 1.7 work with this software, go vet
should pass.

Also updated the gogo/protobuf dependency which fixed the code generator
to work with go 1.7 too. Ran `go generate` on the entire repository to
ensure every file was up to date.
2016-09-14 15:01:22 -05:00
Todd Persen b8192d079e Regenerate static assets and update CHANGELOG for PR #7286. 2016-09-13 23:38:36 -07:00
Dan Dascalescu 858060375e Proper password field 2016-09-09 22:30:17 -07:00
Jonathan A. Sternberg ab4bca8495 Report cmdline and memstats in /debug/vars
When we refactored expvar, the cmdline and memstats sections were not
readded to the output. This adds it back if they can be found inside of
`expvar`.

It also stops trying to sort the output of the statistics so they get
returned faster. JSON doesn't need them to be sorted and it causes
enough latency problems that sorting them hurts performance.
2016-09-09 14:32:43 -05:00
Jonathan A. Sternberg dc2527ce86 Merge branch '1.0' 2016-08-31 14:45:57 -05:00
Jonathan A. Sternberg 23f2d50ecb Use defaults from `meta` package for `CREATE DATABASE`
Instead of having the parser set the defaults, the command will set the
defaults so that the constants for that are actually used. This way we
can also identify which things the user provided and which ones we are
filling with default values.

This allows the meta client to be able to make smarter decisions when
determining if the user requested a conflict or if the requested
capabilities match with what is currently available. If you just say
`CREATE DATABASE WITH NAME myrp`, the user doesn't really care what the
duration of the retention policy is and just wants to use the default.
Now, we can use that information to determine if an existing retention
policy would conflict with what the user requested rather than returning
an error if a default value ever gets changed since the meta client
command can communicate intent more easily.
2016-08-30 13:23:49 -05:00
Jonathan A. Sternberg c05c7f6360 Revert "limit shard concurrency"
This reverts commit 6c7d56d4bc.
2016-08-29 12:39:52 -05:00
Edd Robinson 6cafdbc604 Ensure we don't mutate provided statistics tags 2016-08-24 11:40:13 +01:00
Edd Robinson 90ff713f21 Fix base64 encoding issue in stats
Fixes #7177.
2016-08-22 15:21:31 +01:00
Jonathan A. Sternberg 9b28f7704f Merge pull request #7171 from influxdata/js-1.0-merge-response-formatter-bugfix
Merge branch '1.0'
2016-08-17 19:13:09 -05:00
Jonathan A. Sternberg 5e468fac0b Merge branch '1.0' into master 2016-08-17 13:27:02 -05:00
Jonathan A. Sternberg d2746ee8f2 The number of bytes recorded when using chunking was off by one
Previously, we implicitly added a newline and had to add one to the
number of bytes transmitted because we added that byte. That was removed
at some point and the metric was not updated to record the correct
value.
2016-08-17 11:54:19 -05:00
Jonathan A. Sternberg 71b54768d0 Remove redundant code the from response formatter and fix CloseNotify
The query killing functionality depends on the ResponseWriter exposing a
CloseNotify method. Since we wrap the http.ResponseWriter, the new
struct does not have that method and the HTTP handler would skip past
calling that method.

Instead of duplicating `Flush()` and `CloseNotify()` for every response
formatter, we will unify all of that under a single struct and create
formatters instead.

Also, fixes a bug where the header information from a query would not be
returned until some other data was returned with it because of
buffering and another bug in the gzipResponseWriter that wouldn't flush
the actual underlying ResponseWriter.
2016-08-16 15:54:03 -05:00
Ben Johnson 8aa224b22d
reduce memory allocations in index
This commit changes the index to point to index data in the shards
instead of keeping it in-memory on the heap.
2016-08-16 14:09:00 -06:00
Edd Robinson cebeda817c Update jwt-go to v3 2016-08-12 17:35:57 +01:00
Jonathan A. Sternberg 87f7c66b8a Merge pull request #7119 from influxdata/js-create-database-use-defaults
Use defaults from `meta` package for `CREATE DATABASE`
2016-08-11 10:34:22 -05:00
Jonathan A. Sternberg f58a50c231 Allow queries to be uploaded as a file rather than forcing a text parameter
The query can be uploaded from a file using `multipart/form-data` and
setting the file name to `q`. An example of using curl to execute an
async query would be:

    curl -F "q=@database.iql" -F "async=true" http://localhost:8086/query
2016-08-10 15:34:04 -05:00
Jonathan A. Sternberg 760767b033 Support running a command in async mode
It will return a 204 No Content as long as the query is accepted
(immediate errors will be returned, but not individual errors with
specific queries). The only way to kill the query is by using the task
manager.
2016-08-10 15:34:04 -05:00
Jonathan A. Sternberg a4e49963f5 Implement text/csv content encoding for the response writer
CSV doesn't offer a way to separate different sheets from each other and
it doesn't really have a standard format. We separate sheets with a
newline so they can be imported into something like Excel or LibreOffice
more easily.

The number of columns for each sheet is inferred from the first returned
row in each statement since they should all be the same.
2016-08-10 15:15:33 -05:00
kun 6945655be2 add support for unix socket binding service 2016-08-11 02:20:54 +08:00
Jonathan A. Sternberg 530b00bd76 Use defaults from `meta` package for `CREATE DATABASE`
Instead of having the parser set the defaults, the command will set the
defaults so that the constants for that are actually used. This way we
can also identify which things the user provided and which ones we are
filling with default values.

This allows the meta client to be able to make smarter decisions when
determining if the user requested a conflict or if the requested
capabilities match with what is currently available. If you just say
`CREATE DATABASE WITH NAME myrp`, the user doesn't really care what the
duration of the retention policy is and just wants to use the default.
Now, we can use that information to determine if an existing retention
policy would conflict with what the user requested rather than returning
an error if a default value ever gets changed since the meta client
command can communicate intent more easily.
2016-08-09 12:00:06 -05:00
Jonathan A. Sternberg b98763a3d8 Merge pull request #7118 from influxdata/js-go-generate
go generate on every package to ensure they are generated with the correct dependency
2016-08-08 09:02:32 -05:00
Jonathan A. Sternberg 4cdfc3280d Move the CQ interval by the group by offset
This will make the period selected by the CQ system work correctly for a
query with an offset.
2016-08-05 14:39:52 -05:00
Jonathan A. Sternberg 0879d43b74 Do not run continuous queries that have no time span 2016-08-05 14:39:52 -05:00
Jonathan A. Sternberg ed2f81357f go generate on every package to ensure they are generated with the correct dependency 2016-08-05 14:35:07 -05:00
Ben Johnson 6c7d56d4bc
limit shard concurrency
This commit limits queries to only process one shard at a time.
However, within a shard, multiple series can still be processed in
parallel. Shard iterators are lazily instantiated during query
execution to limit the amount of memory a given query uses.
2016-08-05 09:45:57 -06:00
Jonathan A. Sternberg 73bbbf24b1 Use the Accept header for determining the Content-Type instead of Content-Type
The `Content-Type` header is meant to say what the content type of the
request is supposed to be and `Accept` is used to ask for a specific
content type. I messed this up and used `Content-Type` instead of
`Accept`. This works with `GET` requests accidentally, but `POST`
requests stop working.
2016-07-29 14:29:39 -05:00
Jonathan A. Sternberg d85072ada9 Setup an interface for writing http responses based on the Content-Type header
Only `application/json` is supported right now, but this opens up the
easier possibility of additional content types to be returned from the
server.
2016-07-29 13:11:51 -05:00
Edd Robinson 9a2efafa13 Ensure CQ id names can't clash 2016-07-28 16:26:05 +01:00
Cameron Sparr d688676266 Update UDP buffer setting doc to 25MB 2016-07-28 12:02:56 +01:00
Cameron Sparr 6afdcf9572 Fix incrementing pointsRx twice 2016-07-28 11:06:50 +01:00
David Norton e3328117a9 fix #7081: hardcode auto gen RP name 2016-07-27 16:49:46 -04:00
Joe LeGasse b3341e8f66 Stablize the sorting of meta.ShardGroupInfos 2016-07-27 16:17:52 -04:00
David Norton b1079c9080 revert previous DefaultRetentionPolicyName change 2016-07-27 11:38:42 -04:00
David Norton 8fbfdc5616 fix #7068: use DefaultRetentionPolicyName from cfg
Use DefaultRetentionPolicyName from the config instead of passing
through meta data.
2016-07-26 18:53:55 -04:00
Jonathan A. Sternberg 4bdf57534b Log the CQ execution time when continuous query logging is enabled 2016-07-21 12:16:53 -05:00
Jonathan A. Sternberg f6ff8a572d Run continuous query for multiple buckets rather than one per bucket 2016-07-14 11:23:05 -05:00
Jonathan A. Sternberg 83caeec3a4 Support loading a folder for collectd typesdb files 2016-07-07 20:57:57 -05:00
Jonathan A. Sternberg 7a3bd19926 Properly use the 401 and 403 HTTP status codes
According to the HTTP standard, a lack of authentication credentials or
incorrect authentication credentials should send back a 401
(Unauthorized) with a `WWW-Authenticate` header with a challenge that
can be used to authenticate. This is because a 401 status should be sent
when an authentication attempt can be retried by the browser.

The 403 (Forbidden) status code should be sent when authentication
succeeded, but the user does not have the necessary authorization.
Previously, the server would always send a 401 status code.
2016-07-07 20:30:04 -05:00
Jonathan A. Sternberg 837a9804cf Refactoring the monitor service to avoid expvar
Truncate the time interval output of the monitor service to be on even
time intervals rather than on every minute based on the start time. This
normalizes the output from the monitor service.
2016-07-07 11:13:58 -05:00
Edd Robinson 2c85d498ef Remove unused errors 2016-07-06 15:04:39 +01:00
Mikael Taina b9d3ae085c Don't force root path on getClientVersion
Similar to #1339 .

Fix getClientVersion() on setups where the admin interface resides on some
other path than root path due to some reverse proxy setup.
2016-07-05 22:47:48 +03:00
Cole Mackenzie 6d926722b5 added favicon.ico and regenerated statik.go 2016-07-01 08:53:56 +00:00
Jonathan A. Sternberg b8e52ce39a Merge pull request #6889 from influxdata/js-update-config-options
Update help and remove unused config options from the configuration file
2016-06-23 11:49:10 -05:00
kun 8aa2fe8ede fix ClusterID generation bug 2016-06-23 14:52:58 +08:00
Jonathan A. Sternberg 22173acb70 Update help and remove unused config options from the configuration file
Normalize the output for the various help options so they all follow the
same format and display all relevant options.

Removing some of the unused config options from the configuration file
and updating the help documentation. Removing some remaining references
to clustering within the open source version.
2016-06-21 14:06:05 -05:00
Jonathan A. Sternberg 497db2a6d3 Removing dead code from every package except influxql
The tsdb package had a substantial amount of dead code related to the
old query engine still in there. It is no longer used, so it was removed
since it was left unmaintained. There is likely still more code that is
the same, but wasn't found as part of this code cleanup.

influxql has dead code show up because of the code generation so it is
not included in this pruning.
2016-06-20 22:41:07 -05:00
Edd Robinson 9603bad14c Merge pull request #6868 from smunier01/continuous-queries-admin-interface
Add continuous queries templates in admin interface
2016-06-20 11:35:47 +01:00
Edd Robinson 3d8cde2346 Merge pull request #6774 from minkikim89/patch-1
Fixed admin page reloaded when navbar-brand clicked
2016-06-20 11:34:08 +01:00
simon f007fccb4a Add continuous queries templates in admin interface 2016-06-18 16:33:41 +02:00
Jonathan A. Sternberg 8e1b036b0a Modify the max nanosecond time to be one nanosecond less
The highest time represented by a nanosecond needs to be used for an
exclusive range, so the maximum time needs to be one less than the
possible maximum number of nanoseconds representable by an int64 so that
we don't lose a point at that one time.

Previously worked in the open source version because the timestamp used
for finding a shard would be truncated by the retention policy so the
lookup time didn't run into this edge case because it didn't rest on the
truncation boundary. Since that point didn't really belong in that shard
group and was placed there by mistake, it's best to fix this bug since
the timestamp used to create the shard group should be capable of
retrieving it.
2016-06-16 12:15:41 -05:00
Jack Zampolin a96f96c024 Merge pull request #6834 from influxdata/jz-graphite
Add `port` to `[graphite]` log output
2016-06-14 11:35:19 -07:00
Cameron Sparr b9a3f2e1e5 Make httpd logger closer to Common Log Format
changes the httpd log lines from this:

    [httpd] 2016/06/08 14:06:39 ::1 - - [08/Jun/2016:14:06:39 +0100] POST /write?consistency=any&db=telegraf&precision=s&rp= HTTP/1.1 204 0 - InfluxDBClient d6aa01fc-2d79-11e6-8024-000000000000 2.751391ms

to this:

    [httpd] ::1 - - [08/Jun/2016:14:06:39 +0100] "POST /write?consistency=any&db=telegraf&precision=s&rp= HTTP/1.1" 204 0 "-" "InfluxDBClient" d6aa01fc-2d79-11e6-8024-000000000000 2751

So it changes a few things:

1. Remove the logger timestamp at the beginning which isn't very relevant anyways
2. adds quotes around "METHOD URI PROTOCOL", because this is part of the
common log format.
3. adds quotes around "AGENT" and "REFERRER" because this is part of the
"combined" log format.
4. Puts the response time in integer microseconds, because this is
consistent with apache's %D config mod option.

Compared with CLF, our logs now look like this:

    [httpd] %{COMMON_LOG_FORMAT} "<agent>" "<referrer>" <request_uuid> <response_time_µs>

For reference, see:
https://en.wikipedia.org/wiki/Common_Log_Format
http://httpd.apache.org/docs/current/mod/mod_log_config.html
2016-06-14 14:48:24 +01:00
Jack Zampolin f1db2f2ca5 Add port to all graphite log output to help with debugging multiple endpoints 2016-06-13 11:59:56 -07:00
Jason Wilder fd262697b9 Initialize subscriber channel on open
If the subscriber service was closed and re-open, it would panic
with panic: send on closed channel
2016-06-10 18:00:44 -06:00
Ben Johnson 48f1a6d858 Merge pull request #6820 from benbjohnson/http-query-node-id
Add NodeID to execution options
2016-06-10 13:10:15 -06:00
Jonathan A. Sternberg 9db82e6bf0 Switch ExecutionContext to be passed by value 2016-06-10 12:31:51 -05:00
Jonathan A. Sternberg 55973d2815 Separate the task manager from the query executor
The task manager now acts as its own statement executor so that a custom
statement executor can perform custom actions for KillQueryStatement and
ShowQueriesStatement.
2016-06-10 12:30:51 -05:00
Ben Johnson 7d4bea7153
add node id to execution options
This commit changes the `ExecutionOptions` and `SelectOptions` to
allow a `NodeID` for specifying an exact node to query against.
2016-06-10 09:20:44 -06:00
Jonathan A. Sternberg 4f37bc5a40 Support specifying a retention policy for the graphite service
The graphite service will attempt to create the retention policy and use
it. If the retention policy doesn't exist, it will be created with the
default options.

Fixes #5655.
2016-06-08 17:01:44 -05:00
Joe LeGasse 647210c57a Add TruncatedAt field to meta.ShardGroupInfo 2016-06-07 18:24:24 -04:00
Jonathan A. Sternberg b972c220aa Merge pull request #6757 from influxdata/js-refactor-execute-query
Refactor ExecuteQuery to take options as a struct
2016-06-07 10:35:52 -05:00
Jonathan A. Sternberg 1e562408f1 Merge pull request #6779 from joe2far/patch-1
Fixed typo in docstring
2016-06-07 08:26:23 -05:00