Commit Graph

372 Commits (82e16a18990127283fd70a17a5fc96f47d69e200)

Author SHA1 Message Date
Paul Dix 59fbd371fc Implement backup/restore for TSM.
This changes backup and restore to work for TSM. It breaks it for b1 and bz1, but since those are getting removed it's ok.

The backup runs against any host that is specified and can backup either the metasstore, a database, specific retention policy, or a specific shard. It can also take incremental backups with the `since` flag, which will only backup TSM files that have been created since that timestamp.

The backup is safe to run online. However, for shards that are still hot for writes, they won't be able to create new TSM files while the backup for that single shard runs. If the backup isn't too large and the write throughput isn't too high this shouldn't be a problem since the writes will just go into the WAL cache.
2015-12-30 18:06:50 -05:00
Jonathan A. Sternberg 5d4ecf853c Add continuous query option for customizing resampling
This makes the following syntax possible:

    CREATE CONTINUOUS QUERY mycq ON mydb
        RESAMPLE EVERY 1m FOR 1h
        BEGIN
          SELECT mean(value) INTO cpu_mean FROM cpu GROUP BY time(5m)
        END

The RESAMPLE option customizes how often an interval will be sampled and
the duration. The interval is customized with EVERY. Any intervals
within the resampling duration on a multiple of the resample interval
will be updated with the new results from the query.

The duration is customized with FOR. This determines how long an
interval will participate in resampling.

Both options are optional. If RESAMPLE is in the syntax, at least one of
the two needs to be given. The default for both is the interval of the
continuous query.

The service also improves tracking of the last run time and the logic of
when a query for an interval should be run. When determining the oldest
interval to run for a query, the continuous query service determines
what would have been the optimal time to perform the next query based on
the last run time. It then uses this time to determine the oldest
interval that should be run using the resample duration and will
resample all intervals between this time and the current time as opposed
to potentially forgetting about the last run in an interval if the
continuous query service gets delayed for some reason.

This removes the previous config options for customizing continuous
queries since they are no longer relevant and adds a new option of
customizing the run interval. The run interval determines how often the
continuous query service polls for when it should execute a query. This
option defaults to 1s, but can be set to 1m if the least common factor
of all continuous queries' intervals is a higher value (like 1m).
2015-12-28 16:43:49 -05:00
Sebastian Borza d885f4156b removing golang reference 2015-12-22 12:54:43 -05:00
Sebastian Borza 3bf5d9c969 moving UDP payload size to config option, improve throughput performance
moving UDP payload size to optional config choice to imporove throughput performance
2015-12-22 09:14:50 -05:00
Nathaniel Cook 8214676363 fix panic when merging empty series 2015-12-15 11:32:51 -07:00
Tim Raymond 718fb92cf5 Remove registration, add 'usage-client'
Server registration and stats reporting has been removed from what was
once http://enterprise.influxdata.com. The app that lived there, now
runs at http://usage.influxdata.com, so that the subdomain can
eventually be repurposed. Because we also want to repurpose the
`enterprise-client` repo, we have also renamed that to `usage-client`.
InfluxDB no longer needs the `registration` service now, since all of
the endpoints it communicates with simply discard the data provided to
them.
2015-12-10 15:58:34 -05:00
Nathaniel Cook eb080785d4 add bounds checking for shard group timestamps
enforce strict valid time ranges on points

small
2015-12-09 13:07:13 -07:00
Philip O'Toole a5cb913035 Drop UDP point on bad parse and keep going
Fixes issue #4992.
2015-12-07 19:47:59 -08:00
Cory LaNou cf4e901078 silence registration logging for testing 2015-12-03 08:04:10 -06:00
Paul Dix 6e1d0f47d4 Merge pull request #4921 from pires/3170-db_not_found
Handler errors should be formatted as JSON.
2015-12-02 11:36:53 -05:00
Cory LaNou 3cd8056664 Merge pull request #4876 from e-dard/lint
Lint
2015-12-02 08:55:47 -06:00
Edd Robinson 8384ba3e84 Update comments. 2015-12-02 12:35:58 +00:00
Paulo Pires 4587a5387d Handler errors should be formatted as JSON. Fixes #3170 2015-11-27 21:12:05 +00:00
Cory LaNou be488b7d12 implement close notifier and timeout on executors 2015-11-24 21:07:18 -06:00
Данило Глинський (Danylo Hlynskyi) 8ba35c3b8c fix collectd manual typo 2015-11-23 18:54:47 +02:00
Edd Robinson ffbd6037e2 Initial lint for all packages under services. Supports #4098 2015-11-22 19:23:56 +00:00
Philip O'Toole b1c5a36b0d Merge pull request #4846 from jsternberg/feature/allow-nan-in-graphite-service
Allow NaN as a valid value on the graphite service
2015-11-20 08:09:51 -08:00
Jonathan A. Sternberg 5fa36639db Allow NaN as a valid value on the graphite service
The canonical graphite implementation will read and discard NaN values
instead of throwing an error when reading on the line receiver protocol.
Since this is the default behavior for graphite, InfluxDB should have
the same behavior for compatibility.

Previously, a NaN value would result in an error printed to the console.
When you have a large number of NaN values being sent every minute, this
results in the log file filling with useless messages.
2015-11-20 11:07:25 -05:00
Philip O'Toole d86a02eeab Fix minor typo in variable name 2015-11-20 07:48:42 -08:00
Paweł Kowalak 8c2f6eb7e0 Comment additions to services to satisfy golint 2015-11-19 13:25:07 +01:00
Philip O'Toole ba807a268b Merge pull request #4835 from orthogonous/master
Comment additions to conform to golint
2015-11-18 17:47:51 -08:00
orthogonous 65f4060de3 Comment additions to conform to golint
Also 1 slight code change that fixed an if statement
that golint didn't like.
2015-11-19 11:26:21 +10:00
Philip O'Toole 06cd9b1e4d Merge pull request #4411 from gobwas/master
Update handler.go
2015-11-18 16:12:18 -08:00
Sebastian Borza e17e6394c9 fixing typo for pointsWrittenOK 2015-11-18 14:28:39 -05:00
John Nguyen 63e958f06e fixed export lint issues in services/admin 2015-11-16 17:31:53 +08:00
Philip O'Toole 69803ddc6f Merge pull request #4791 from orthogonous/master
Added some comments to the udp service so golint passes. Ref #4098
2015-11-13 19:10:49 -08:00
Philip O'Toole 7ea9b3e49e Allow openTSDB point errors logging to be disabled 2015-11-13 14:54:54 -08:00
orthogonous 434d06008e Added some comments to the udp service so golint passes. Ref #4098 2015-11-14 08:46:26 +10:00
Philip O'Toole bc1091ec31 Revert "Merge pull request #4684 from nkatsaros/default-config"
This reverts commit 87441584f3, reversing
changes made to a14829bc1d.
2015-11-11 09:29:06 -08:00
Cory LaNou 6ecb62e4d2 Merge pull request #4737 from ch33hau/4283-hh-throws-error-even-if-disabled
Disable HintedHandoff if configuration is not set. #4283
2015-11-10 19:07:27 -06:00
Cameron Sparr 9625953d3e Add UDP OS buffer size recommendations 2015-11-10 11:00:11 -07:00
Cameron Sparr 727b9f6d8b Documentation updates for UDP, graphite, collectd listeners 2015-11-10 10:35:21 -07:00
Cameron Sparr 0a683144fb Update collectd and graphite UDP listeners with perf enhancements
closes #4678
2015-11-10 10:35:21 -07:00
ch33hau 8bfdfbda0b Disable HintedHandoff if configuration is not set. #4283 2015-11-11 01:12:34 +08:00
Philip O'Toole bece8fed2a Better retention enforcement logging
Fixes issue #4727.
2015-11-09 17:22:24 -08:00
Cameron Sparr 78e6979be5 Make parser/writer internal func names consistent 2015-11-06 16:43:35 -07:00
Cameron Sparr 5c28048132 UDP service listener performance enhancements 2015-11-06 15:53:47 -07:00
Mark Bates 8ac958f8a0 Merge pull request #4683 from influxdb/enterprise-client
Updated to use the new Enterprise Client
2015-11-06 13:13:15 -05:00
Mark Bates 377d926aba Fixed a logging message posting to Enterprise 2015-11-06 10:21:18 -05:00
Mark Bates b1df4bd76e Updated to use the new Enterprise Client 2015-11-05 16:46:45 -05:00
Nicholas Katsaros 4f8b0aca9a Add graphite and udp services to the default config generator 2015-11-05 16:42:21 -05:00
Cameron Sparr e2db577e0f Implement a UDP client
closes #4647
2015-11-04 14:18:14 -07:00
Philip O'Toole 3ea4366e4e Unit test handling bad write requests 2015-10-30 11:05:47 -07:00
oiooj d9a60d7e4c fix write points panic 2015-10-30 10:52:50 -07:00
David Howden 5044b9d42c Use HTTPS instead of Https in variable names 2015-10-30 08:49:37 +11:00
Philip O'Toole de7919240f Migrate internal stats to consistent names
Go style -- and existing runtime stats -- do not use underscores, but
instead use camel case. This change makes the internal stats adhere to
that convention.
2015-10-28 21:07:45 -07:00
Philip O'Toole ef190b4013 Merge pull request #4600 from influxdb/wait_for_leader
ping endpoint can now wait for leader
2015-10-28 12:04:53 -07:00
Philip O'Toole 1f6f9b35d1 Unit test ping endpoint 2015-10-28 11:37:25 -07:00
Philip O'Toole 2046556b71 Ping endpoint supports waiting for leader 2015-10-28 11:37:07 -07:00
Jason Wilder 7d6d23e661 Write as many points as we can parse via line protocol
This changes the HTTP line protocol handler to behave similar to the other
handler in that they will write as many points as possible.  Previously, we
would fail the entire batch if one point failed.  This can happen more frequently
now with NaN being more explicitly unsupported.  Now it will write as many points
that parse successfully and return a "partial write" error to the client with the
lines that failed to parse.
2015-10-27 17:12:57 -06:00