Commit Graph

1765 Commits (dff34939f8e2176955052de7ace2c47c08e7fd19)

Author SHA1 Message Date
Jonathan A. Sternberg dff34939f8
Merge pull request #9028 from cbushko/lint-the-c-dirs
Fix linting on cmd files
2018-04-23 10:41:05 -05:00
Ben Johnson 9a58835b82
Merge pull request #9743 from influxdata/bj-delete-tsm
Add 'influx_inspect deletetsm'
2018-04-20 07:41:25 -06:00
Stuart Carnie 14dcc5d6e7 PR feedback 2018-04-19 18:05:55 -07:00
Stuart Carnie 29ce031fc2 influx-tools/export: driver for reshaping and exporting data
The only file require in Enterprise is a reimplementation of main.go,
the server.Interface and server.MetaClient interfaces.
2018-04-19 18:05:55 -07:00
Stuart Carnie 9e79b5cbaf influx-tools/format: implementation of various series and value writers
A format.Writer is an abstraction for reading data from
storage.ResultSet and writing to various formats. Those included are

* binary: efficient binary format using protocol buffers. This is the
  expected format for the import tooling. The data is written in the
  desired shard group shape so that it can be read and streams to
  TSM files without further transformation.

* line: line protocol use for exporting field type conflicts or as an
  alternative, lossless export format

* text: two debugging modes for outputting series or series and values
  in a more efficient format that line protocol.

* discard: reads and discards the source data. This can be useful for
  benchmarking and profiling the read and decode performance.
2018-04-19 18:05:55 -07:00
Stuart Carnie 338d6b463d influx-tools/server: abstractions to simplify usage in Enterprise
This package is the only component that must be implemented to
use the influx-tools in the Enterprise edition.
2018-04-19 18:05:55 -07:00
Stuart Carnie a8f3c8ed5b import-tools: minimal storage API to read shard data efficiently
* this implementation produces a cursor per shard to permit the export
  tool to distinguish field type conflicts
2018-04-19 18:05:55 -07:00
Ben Johnson 8c9d179a94
Add 'influx_inspect deletetsm' 2018-04-19 10:36:08 -06:00
Adam a00e4dc70b
Changed file download retry rate to have exponential wait times (#9706)
* Changed file download retry rate to have exponential wait times

* Fixes from PR review
2018-04-12 15:21:02 -04:00
Ben Johnson fd491b28a6
Check for root user when running buildtsi.
This commit also adds the `max-log-file-size` flag.
2018-04-04 09:58:02 -06:00
Mark Rushakoff b3c2d9290f Log error encountered when reading WAL files
Inspired by #9657.
2018-03-30 09:40:58 -07:00
Ben Johnson eb6215acbf
Fix buildtsi partition key. 2018-03-29 14:27:28 -06:00
Adam ca9f2510b8
check for failure case where backup dir has no manifest files (#9634) 2018-03-27 12:05:29 -04:00
Stuart Carnie ee3e2ad67f rename Tenant -> OrgID 2018-03-23 12:26:55 -07:00
Stuart Carnie f69681c7c2 store command: add support for multi-tenant queries; fix validation 2018-03-23 12:26:55 -07:00
Stuart Carnie aa61359cc7 Storage RPC API improvements. See PR for details
* reduce # allocations (115M -> 22M)
* reduce size allocations (53GB -> 1.3GB)
* reduce RPC query time (45s -> 12.9s)
2018-03-21 13:46:09 -07:00
Mark Rushakoff fbddcf7cad Move ApplyEnvOverrides to toml package
This code has been duplicated to other projects and its implementations
have grown out of sync. Now the code can live as a package-level
function rather than a method coupled with particular structs.
2018-03-14 15:23:46 -07:00
Adam b213b31cd4
Bugfix for shard collisions resulting from attempted _internal import (#9569) 2018-03-14 11:51:57 -04:00
Jonathan A. Sternberg 9cafd4b1f5
Merge pull request #9429 from influxdata/js-9297-support-http-proxy-in-clients
Support proxy environment variables in the influx client
2018-03-12 11:40:09 -05:00
Jonathan A. Sternberg 5d8d460034 Support setting the log level through the environment variable 2018-03-12 11:10:32 -05:00
Jonathan A. Sternberg ed27a00255
Merge pull request #9524 from influxdata/js-remove-query-prefix
Rename some of the structs and interfaces in the query package
2018-03-07 18:00:37 -06:00
Stuart Carnie 044ca2ac89 ensure ENV overrides are applied when configuring logging
Additionally, the config file is only parsed once.
2018-03-03 07:38:15 -07:00
Jonathan A. Sternberg de4390ae83 Rename some of the structs and interfaces in the query package
Remove the `Query` prefix from some structs and interfaces. They were
there so when the query engine was in the same package as influxql,
these would be differentiated. Now that the package name is query, the
extra prefix seems redundant.
2018-03-02 09:44:12 -06:00
Ben Johnson b3fcc63a78
Add dumptsi path error handling. 2018-02-27 08:30:12 -07:00
Stuart Carnie d135aecf02 Generate trace logs for a number of significant influx operations
* tsdb Store.Open traces all events related to opening files
    * op.name : tsdb.open
* retention policy shard deletions
    * op.name : retention.delete_check
* all TSM compaction strategies
    * op.name : tsm1.compact_group
* series file compactions
    * op.name : series_partition.compaction
* continuous query execution (if logging enabled)
    * op.name : continuous_querier.execute
* TSI log file compaction
    * op_name: index.tsi.compact_log_file
* TSI level compaction
    * op.name: index.tsi.compact_to_level
2018-02-21 15:08:49 -07:00
Jonathan A. Sternberg 2bbd96768d Update logging calls to take advantage of structured logging
Includes a style guide that details the basics of how to log.
2018-02-20 10:04:19 -06:00
Stuart Carnie 1303030184 Suppress the InfluxDB banner if the log output is not a TTY 2018-02-14 15:23:58 -07:00
Jonathan A. Sternberg 2bb5b68833 Add an auto choice that will choose the most appropriate logger
This restricts the logging format options so console cannot be chosen.
This way, logs written to a non-TTY will never be the console format.
2018-02-14 11:56:29 -05:00
Jonathan A. Sternberg 2efc9ddbac Add an option to suppress the initial logo
For any systems that want to read the log file in the specific format,
the logo being printed on restart may not be good for those parsers
since the log parser would have to be aware of the logos existance or
capable of just ignoring lines it couldn't parse.

This gives an option to disable the printed logo if required.

Like other logging options, this will fail if the configuration file
itself is invalid.
2018-02-14 10:11:02 -05:00
Jonathan A. Sternberg ccf2e549b8 Add logging configuration to the main configuration file
Make logfmt the default log output format with json and a more user
friendly console into possible options.
2018-02-14 10:11:02 -05:00
Jonathan A. Sternberg c67a0548a0 Support proxy environment variables in the influx client
This also adds support for both the v1 and v2 APIs to set the Proxy
function on the underlying http.Transport.

The functionality for proxy environment variables is provided by the
[http.ProxyFromEnvironment](https://godoc.org/net/http#ProxyFromEnvironment).
2018-02-11 21:51:58 -06:00
Stuart Carnie 41dc96ca91 restore `MetaClient`, which is needed by store
* Switch from an anonymous type to avoid false positives with
  `megacheck`
2018-02-08 12:13:13 -07:00
Ben Johnson 0af6fb8e10
Merge pull request #9412 from influxdata/er-series-sketches
Sketches, Cont'd
2018-02-08 11:41:39 -07:00
kun 7bc52b7a81 Fix windows history file location 2018-02-09 00:20:52 +08:00
Edd Robinson 7a55735562
Add option to set LogFile compaction size 2018-02-07 14:52:13 -07:00
Adam 43fdb866dd
update to logs to print start/end values when relevant (#9388) 2018-02-02 13:32:43 -05:00
Adam f11d6fcd29
Removed tabs from help text (#9382) 2018-01-31 09:28:15 -05:00
Ben Johnson b42245acbe
Refactoring 'buildtsi' command. 2018-01-30 11:01:35 -07:00
Edd Robinson 13a1505ba2 Cleanup cmd package 2018-01-21 12:08:25 -08:00
Edd Robinson 6a66b5faf0 Cleanup services package 2018-01-21 10:52:37 -08:00
Adam 28f3912491
Renamed references for "enterprise" backup/restore mode to "portable" (#9346) 2018-01-19 16:53:43 -05:00
Adam 938db68198
Update restore functionality to run in online mode, consume Enterprise backup files. (#9207)
* Live Restore + Enterprise data format compatability

* Extended ImportData to import all DB's if no db name given

* Added a new enterprise data test, and backup command now prints the backup file paths at conclusion

* Added whole-system backup test

* Update to use protobuf in all enterprise data cases

* Update to test to do cross-testing with enterprise version

* incremental enterprise backup format support
2018-01-10 13:59:18 -05:00
Ben Johnson 98486a284a
Merge pull request #9265 from benbjohnson/series-file-compaction
Sequential series file id & series file segmentation
2018-01-03 10:05:59 -07:00
Edd Robinson 5efeed4ccc Add import 2018-01-03 16:50:44 +00:00
Ben Johnson 52630e69d7
Integrate SeriesFileCompactor 2018-01-02 12:20:03 -07:00
Jonathan A. Sternberg af23897940 Allow setting the node id in the influx cli program
The string `node <n>` can be used to specify which data node the data
should be retrieved from. This uses the `node_id=X` query parameter that
is supported, but wasn't exposed anywhere in the client library.

We use this feature enough internally when attempting to find
inconsistencies or network errors that it is easier if this is just
supported. Otherwise, I continue having to recompile the CLI program
every time I need to do this.

To clear a previously set node, you can use `node 0` or `node clear`.
2018-01-02 11:15:19 -06:00
Ben Johnson d8b1d208c0
rebase 2017-12-20 15:13:34 -07:00
Edd Robinson 289d1f8d44 Allow iterators to return if shard is closing 2017-12-15 00:46:43 +00:00
Ben Johnson 288c5217e8
Fix tsi1 tools. 2017-12-08 16:12:33 -07:00
Edd Robinson 7d13bf3262 merge master 2017-12-08 17:21:58 +00:00