This upgrades the flux version to v0.50.2.
The secret service, which is used for alerts, is not included. The
`to()` function is also still not included.
The flux in influxdb has been upgraded to use v0.33.2. A lot of
interfaces for the storage engine were changed during this so code had
to change to accomodate the new interfaces and remove the old ones.
Included in this commit is a patch file for the changes that were made.
A patch was generated for the following packages:
* `flux/stdlib/influxdata/influxdb`
* `storage/reads`
* `tsdb/cursors`
These are the three packages that are in common with version 2 of the
database and the first of these packages contains the specific
implementations that are used for version 1.
It is very possible that the next time we upgrade this, the patch will
not apply cleanly just like it wouldn't have applied cleanly to this
update. The patch is mostly meant to document exactly what changed
during the copy over to help ensure we don't forget things when adapting
the interfaces.
Add a patch file to hopefully make this easier in the future
There is no need to print such messages in the CLI output of the version
flag. Besides being unnecessary, it makes harder to automate some tests
for those installing influxDB CE by themselves and/or automating
dockerfiles.
Fix influxdata/influxdb/issues/10451
The series index looks at a set of tombstones when querying the id for
a given key, but it does not look when asking for the offset for some
id, even if that id is deleted.
Update the verify tooling to check that the index agrees with the
deleted status of the id, but skip doing the extra checks if the
id is deleted.
This integrates the influxdb 1.x series to the latest version of Flux
and updates the code to use it. It also removes the dependency on
platform and copies the necessary code from storage into the 1.x series
so the dependency is unneeded.
The flux functions specific to 1.x have been moved to the same structure
that flux changed to with having a `stdlib` directory instead of a
`functions` directory. It also adds a `databases()` function that
returns the databases from the meta client.
Specifically:
* renamed files for consistency between versions
* added `time-interval` schema option
* updated schema example documentation
Back port of improvements from #12710
* Add AuthorizeDatabase API to QueryAuthorizer to verify a user has
appropriate access to the specified database
* Update serverFluxQuery handler to require a meta.User when auth is
enabled
* update Flux createFromSource and createBucketsSource dependencies to
require Authorizer when auth is enabled in configuration
* update createFromSource to verify read permissions for each bucket
specified in a Flux query
* update BucketsDecoder, which implements the buckets() Flux function,
to return buckets that the user has read or write permissions to
* add unit tests to verify authentication is required for Flux HTTP
requests when auth is enabled in configuration
* gen-init initializes a database based on the provided CLI spec
* gen-exec generates the data for the target database based on the same
CLI spec as gen-init
* `-type` reports error for invalid values
* Implemented decision tree such that appropriate message is printed
when Flux version of command is executed
* all variations for invoking influx identified in #10407 work as
expected
This commit deletes most of the code to service reads from influxdb
and pulls it in from platform instead.
Of note, the models.Tag and models.Tags types are now aliases to the
platform models.Tag and models.Tags types. Additionally, many types
in the tsdb package relating to cursors are also aliases to the same
types in the platform cursors package.
This updates the platform and flux repos to the current master in the
Gopkg.lock.
If there is a significant amount of data in the WAL, then building the
TSI index can be problematic without being able to set the max cache
size to something larger.
This commit adds an option to se the maximum cache size.
This commit fixes an issue with the series file compaction process
where tombstones are lost after compaction and series existence
checks are not correct. This commit also fixes some smaller flushing
issues within the series file that mainly related to testing.
* the protocol service definition, ReadRequest and ReadResponse is
reused across projects, rather than requiring redefinition.
* the ReadRequest protocol buffer definition removes the concept of a
database and retention policy, replacing it with a field named
ReadSource of type google.protobuf.Any. OSS requests will use the
ReadSource message structure defined in local to this package, which
defines fields to represent a Database and RetentionPolicy. Other
implementations can provide their own data structure allowing the
remainder of the ReadRequest to be reused.
* The RPC service and Store are expected to be redefined to handle their
specific requirements for resolving a ReadSource
* ResultSet and GroupResultSet are interfaces representing non-grouping
and grouping read behavior respectively. Calling NewResultSet or
NewGroupResultSet will construct instances of these types
* The ResponseWriter type is exported to deal with serialization of
the ResultSet and GroupResultSet types
When adding many series using offline tooling, it's likely that every
series involves an entry being appended to a LogFile. Typically an entry
is 11 or 12 bytes, but the default bufio.Writer buffer size is only 4K.
This means by default a write of 10,000 new series would involve ~30
buffer flushes.
This commit makes the buffer configurable, and sets the value in
`buildtsi` such that it reflects the number of series being written to
the LogFile.
* utilizes `tsm1.Compactor#CompactFull` to fully compact the specified
shard
* the WAL is unmodified
* added `-verbose` option to show progress as TSM files are opened
This should not have caused correctness issues, but is an unintended
side effect that exporting data may cause compactions to run. It is
possible that a compaction would not run to completion, leaving .tmp
files around after an export.
* Update Prometheus remote write to use metric name as measurement name and value as the field name.
* Update Prometheus remote read to use the storage.Read method to bypass the InfluxQL query engine.
This commit adds `debug-pprof-enabled` which will start the default
`net/http/pprof` endpoint and bind against `localhost:6060`. This
will help to debug startup performance issues.
- Expose io for testing
- Initialize logging only when present
- Fix nil cases when replacing retention policies
- Use meta client when getting shard groups
- Disallow updating retention policies
- Delete shard files, not shard groups, when replacing shards
- Add duration and replication options for retention policy
This commit adds the `-sanitize` flag to `influx_inspect deletetsm`
which will delete all keys that contain invalid, non-printable, or
replacement character unicode.
Usage:
```sh
$ influx_inspect deletetsm -sanitize PATH
```
does some basic sanity checks. it's hard to be more exhaustive without
either taking a crazy amount of time, or being non-deterministic,
but at least this makes sure we barf in some cases.
Updated flags, help text, removed documentation for deprecated legacy options. Updated documentation to describe the syntax and options for the newer -portable format. Legacy support remains, but is only referenced in the online documentation.
A format.Writer is an abstraction for reading data from
storage.ResultSet and writing to various formats. Those included are
* binary: efficient binary format using protocol buffers. This is the
expected format for the import tooling. The data is written in the
desired shard group shape so that it can be read and streams to
TSM files without further transformation.
* line: line protocol use for exporting field type conflicts or as an
alternative, lossless export format
* text: two debugging modes for outputting series or series and values
in a more efficient format that line protocol.
* discard: reads and discards the source data. This can be useful for
benchmarking and profiling the read and decode performance.
This code has been duplicated to other projects and its implementations
have grown out of sync. Now the code can live as a package-level
function rather than a method coupled with particular structs.
Remove the `Query` prefix from some structs and interfaces. They were
there so when the query engine was in the same package as influxql,
these would be differentiated. Now that the package name is query, the
extra prefix seems redundant.
For any systems that want to read the log file in the specific format,
the logo being printed on restart may not be good for those parsers
since the log parser would have to be aware of the logos existance or
capable of just ignoring lines it couldn't parse.
This gives an option to disable the printed logo if required.
Like other logging options, this will fail if the configuration file
itself is invalid.
This also adds support for both the v1 and v2 APIs to set the Proxy
function on the underlying http.Transport.
The functionality for proxy environment variables is provided by the
[http.ProxyFromEnvironment](https://godoc.org/net/http#ProxyFromEnvironment).
Edd Robinson
Jonathan A. Sternberg
Adam Perlin
Diego Rabatone Oliveira
Jeff Wendling
Stuart Carnie
Jeff Wendling
Ben Johnson
Warren Harper
Manuel Schoenlaub
Adam
Sven Mäder
Tim Brooks
Jacob Marble
Paul Dix
Stuart Carnie
Peng Liu
Tanya Gordeeva
Jeff Wendling
Mark Rushakoff
Jonathan A. Sternberg
kun