Commit Graph

1773 Commits (2e470974ecf876d9da8a310909e68b717d0fb53c)

Author SHA1 Message Date
Ben Johnson 2e470974ec
Merge pull request from influxdata/bj-delete-nonprintable-unicode
Add key sanitization to deletetsm
2018-04-30 15:38:08 -06:00
Ben Johnson 4cb36d5301
Add key sanitization to deletetsm
This commit adds the `-sanitize` flag to `influx_inspect deletetsm`
which will delete all keys that contain invalid, non-printable, or
replacement character unicode.

Usage:

```sh
$ influx_inspect deletetsm -sanitize PATH
```
2018-04-30 13:06:33 -06:00
Ben Johnson 108fa09439
Rename corrupt tsm files on load. 2018-04-27 14:27:44 -06:00
Tanya Gordeeva 0badc89b11 influx-tools/importer: Clean up shards with errors 2018-04-25 09:33:43 -07:00
Tanya Gordeeva 1ea9397353 influx-tools/importer: Binary format import command
Import command using the binary format reader to directly write TSM data,
optionally replacing existing retention policies and shards.
2018-04-25 09:33:43 -07:00
Tanya Gordeeva 15b46c8ebe influx-tools/importer: Binary format reader unit tests 2018-04-25 09:33:43 -07:00
Tanya Gordeeva 6f99a3cfad influx-tools/importer: Binary format reader
Binary format reader for output from `import-tools export -format binary`.
2018-04-25 09:33:43 -07:00
Adam b3cd949c3f
Update backup flags to have long/short versions ()
Updated flags, help text, removed documentation for deprecated legacy options.  Updated documentation to describe the  syntax and options for the newer -portable format. Legacy support remains, but is only referenced in the online documentation.
2018-04-25 10:49:46 -04:00
Jonathan A. Sternberg dff34939f8
Merge pull request from cbushko/lint-the-c-dirs
Fix linting on cmd files
2018-04-23 10:41:05 -05:00
Ben Johnson 9a58835b82
Merge pull request from influxdata/bj-delete-tsm
Add 'influx_inspect deletetsm'
2018-04-20 07:41:25 -06:00
Stuart Carnie 14dcc5d6e7 PR feedback 2018-04-19 18:05:55 -07:00
Stuart Carnie 29ce031fc2 influx-tools/export: driver for reshaping and exporting data
The only file require in Enterprise is a reimplementation of main.go,
the server.Interface and server.MetaClient interfaces.
2018-04-19 18:05:55 -07:00
Stuart Carnie 9e79b5cbaf influx-tools/format: implementation of various series and value writers
A format.Writer is an abstraction for reading data from
storage.ResultSet and writing to various formats. Those included are

* binary: efficient binary format using protocol buffers. This is the
  expected format for the import tooling. The data is written in the
  desired shard group shape so that it can be read and streams to
  TSM files without further transformation.

* line: line protocol use for exporting field type conflicts or as an
  alternative, lossless export format

* text: two debugging modes for outputting series or series and values
  in a more efficient format that line protocol.

* discard: reads and discards the source data. This can be useful for
  benchmarking and profiling the read and decode performance.
2018-04-19 18:05:55 -07:00
Stuart Carnie 338d6b463d influx-tools/server: abstractions to simplify usage in Enterprise
This package is the only component that must be implemented to
use the influx-tools in the Enterprise edition.
2018-04-19 18:05:55 -07:00
Stuart Carnie a8f3c8ed5b import-tools: minimal storage API to read shard data efficiently
* this implementation produces a cursor per shard to permit the export
  tool to distinguish field type conflicts
2018-04-19 18:05:55 -07:00
Ben Johnson 8c9d179a94
Add 'influx_inspect deletetsm' 2018-04-19 10:36:08 -06:00
Adam a00e4dc70b
Changed file download retry rate to have exponential wait times ()
* Changed file download retry rate to have exponential wait times

* Fixes from PR review
2018-04-12 15:21:02 -04:00
Ben Johnson fd491b28a6
Check for root user when running buildtsi.
This commit also adds the `max-log-file-size` flag.
2018-04-04 09:58:02 -06:00
Mark Rushakoff b3c2d9290f Log error encountered when reading WAL files
Inspired by .
2018-03-30 09:40:58 -07:00
Ben Johnson eb6215acbf
Fix buildtsi partition key. 2018-03-29 14:27:28 -06:00
Adam ca9f2510b8
check for failure case where backup dir has no manifest files () 2018-03-27 12:05:29 -04:00
Stuart Carnie ee3e2ad67f rename Tenant -> OrgID 2018-03-23 12:26:55 -07:00
Stuart Carnie f69681c7c2 store command: add support for multi-tenant queries; fix validation 2018-03-23 12:26:55 -07:00
Stuart Carnie aa61359cc7 Storage RPC API improvements. See PR for details
* reduce # allocations (115M -> 22M)
* reduce size allocations (53GB -> 1.3GB)
* reduce RPC query time (45s -> 12.9s)
2018-03-21 13:46:09 -07:00
Mark Rushakoff fbddcf7cad Move ApplyEnvOverrides to toml package
This code has been duplicated to other projects and its implementations
have grown out of sync. Now the code can live as a package-level
function rather than a method coupled with particular structs.
2018-03-14 15:23:46 -07:00
Adam b213b31cd4
Bugfix for shard collisions resulting from attempted _internal import () 2018-03-14 11:51:57 -04:00
Jonathan A. Sternberg 9cafd4b1f5
Merge pull request from influxdata/js-9297-support-http-proxy-in-clients
Support proxy environment variables in the influx client
2018-03-12 11:40:09 -05:00
Jonathan A. Sternberg 5d8d460034 Support setting the log level through the environment variable 2018-03-12 11:10:32 -05:00
Jonathan A. Sternberg ed27a00255
Merge pull request from influxdata/js-remove-query-prefix
Rename some of the structs and interfaces in the query package
2018-03-07 18:00:37 -06:00
Stuart Carnie 044ca2ac89 ensure ENV overrides are applied when configuring logging
Additionally, the config file is only parsed once.
2018-03-03 07:38:15 -07:00
Jonathan A. Sternberg de4390ae83 Rename some of the structs and interfaces in the query package
Remove the `Query` prefix from some structs and interfaces. They were
there so when the query engine was in the same package as influxql,
these would be differentiated. Now that the package name is query, the
extra prefix seems redundant.
2018-03-02 09:44:12 -06:00
Ben Johnson b3fcc63a78
Add dumptsi path error handling. 2018-02-27 08:30:12 -07:00
Stuart Carnie d135aecf02 Generate trace logs for a number of significant influx operations
* tsdb Store.Open traces all events related to opening files
    * op.name : tsdb.open
* retention policy shard deletions
    * op.name : retention.delete_check
* all TSM compaction strategies
    * op.name : tsm1.compact_group
* series file compactions
    * op.name : series_partition.compaction
* continuous query execution (if logging enabled)
    * op.name : continuous_querier.execute
* TSI log file compaction
    * op_name: index.tsi.compact_log_file
* TSI level compaction
    * op.name: index.tsi.compact_to_level
2018-02-21 15:08:49 -07:00
Jonathan A. Sternberg 2bbd96768d Update logging calls to take advantage of structured logging
Includes a style guide that details the basics of how to log.
2018-02-20 10:04:19 -06:00
Stuart Carnie 1303030184 Suppress the InfluxDB banner if the log output is not a TTY 2018-02-14 15:23:58 -07:00
Jonathan A. Sternberg 2bb5b68833 Add an auto choice that will choose the most appropriate logger
This restricts the logging format options so console cannot be chosen.
This way, logs written to a non-TTY will never be the console format.
2018-02-14 11:56:29 -05:00
Jonathan A. Sternberg 2efc9ddbac Add an option to suppress the initial logo
For any systems that want to read the log file in the specific format,
the logo being printed on restart may not be good for those parsers
since the log parser would have to be aware of the logos existance or
capable of just ignoring lines it couldn't parse.

This gives an option to disable the printed logo if required.

Like other logging options, this will fail if the configuration file
itself is invalid.
2018-02-14 10:11:02 -05:00
Jonathan A. Sternberg ccf2e549b8 Add logging configuration to the main configuration file
Make logfmt the default log output format with json and a more user
friendly console into possible options.
2018-02-14 10:11:02 -05:00
Jonathan A. Sternberg c67a0548a0 Support proxy environment variables in the influx client
This also adds support for both the v1 and v2 APIs to set the Proxy
function on the underlying http.Transport.

The functionality for proxy environment variables is provided by the
[http.ProxyFromEnvironment](https://godoc.org/net/http#ProxyFromEnvironment).
2018-02-11 21:51:58 -06:00
Stuart Carnie 41dc96ca91 restore `MetaClient`, which is needed by store
* Switch from an anonymous type to avoid false positives with
  `megacheck`
2018-02-08 12:13:13 -07:00
Ben Johnson 0af6fb8e10
Merge pull request from influxdata/er-series-sketches
Sketches, Cont'd
2018-02-08 11:41:39 -07:00
kun 7bc52b7a81 Fix windows history file location 2018-02-09 00:20:52 +08:00
Edd Robinson 7a55735562
Add option to set LogFile compaction size 2018-02-07 14:52:13 -07:00
Adam 43fdb866dd
update to logs to print start/end values when relevant () 2018-02-02 13:32:43 -05:00
Adam f11d6fcd29
Removed tabs from help text () 2018-01-31 09:28:15 -05:00
Ben Johnson b42245acbe
Refactoring 'buildtsi' command. 2018-01-30 11:01:35 -07:00
Edd Robinson 13a1505ba2 Cleanup cmd package 2018-01-21 12:08:25 -08:00
Edd Robinson 6a66b5faf0 Cleanup services package 2018-01-21 10:52:37 -08:00
Adam 28f3912491
Renamed references for "enterprise" backup/restore mode to "portable" () 2018-01-19 16:53:43 -05:00
Adam 938db68198
Update restore functionality to run in online mode, consume Enterprise backup files. ()
* Live Restore + Enterprise data format compatability

* Extended ImportData to import all DB's if no db name given

* Added a new enterprise data test, and backup command now prints the backup file paths at conclusion

* Added whole-system backup test

* Update to use protobuf in all enterprise data cases

* Update to test to do cross-testing with enterprise version

* incremental enterprise backup format support
2018-01-10 13:59:18 -05:00