* test(storage): ensure multiple engines can run concurrently
* feat(storage): expose control over retention run
Fixes#15134.
This commit adds the ability to inject a functional option into a
storage.Engine for controlling when the retention enforcer can run.
Previously, retention enforcers ran on an interval; if you ran multiple
storage engines (as we do in some environments) then it was not possible
to coordinate when engines ran retention. Often they would synchronise
because they started at the same time.
This change will let you specify a blocking function to control when the
retention enforcer can run.
A simple function for serialising retention enforcement across multiple
storage engines could look like:
```go
var mu sync.Mutex
func f() (done func()) {
mu.Lock()
return func() { mu.Unlock() }
}
```
The storage engine will now drop any points that contain invalid tag
data. Special tag keys for the measurement and field key will be
excepted from this validation.
When the WAL was moved up, the validation that happened at the cache
was skipped. This moves the field type validation for a batch of
points up ahead of the WAL again.
I did this with a dumb editor macro, so some comments changed too.
Also rename root package from platform to influxdb.
In interest of minimizing risk, anyone importing the root package has
now aliased it to "platform" so that no changes beyond imports were
necessary in those files.
Lastly, replace the old platform module to local path /dev/null so that
nobody can accidentally reintroduce a platform dependency while
migrating platform code to influxdb.
We were passing a non-nil tsm1.Log containing a nil *tsm1.WAL which
would cause a panic when it was attempted to be used. Instead, always
pass a non-nil WAL.
We change the storage engine code to not pass in a nil WAL, and
additionally add a defensive check to change any nil WALs into a
NopWAL.
This pulls in the code that allows doing reads with flux into the
platform repo, and removes extra.go.
The reusable portion is under storage/reads, where the concrete
implementation for one of the platform's engines is in
storage/readservice.
In order to make this more reusable, the cursors had to move into
their own package, decoupling it from all of the other code in the
tsdb package. tsdb/cursors is this new package, and type/function
aliases have been added to the tsdb package to point at it.
The models package already is very light on transitive dependencies
and so it was allowed to be depended on in a concrete way in the
cursors package.
Finally, the protobuf definitions for issuing GRPC reads has been
moved into its own package for two reasons:
1. It's a clean separation, and helps keep it that way.
2. Many/most consumers will not be using GRPC. We just
use the datatypes to express the API which helps making
a GRPC server easier.
It is left up to future refactorings (specifically ones that involve
GPRC) to determine if these types should remain, or if there is a
cleaner way.
There's still some dependencies on both github.com/influxdata/influxql
and github.com/influxdata/influxdb/logger that we can hopefully remove
in future refactorings.
It may become a uint64 in the future, for example. This does mean
that we have to call Decode on some data that we just Encoded, but
we can fix that later.
elbehery
Edd Robinson
Edd Robinson
Tanya Gordeeva
Jeff Wendling
Jeff Wendling
Ben Johnson
Jacob Marble
Michael Desa
Mark Rushakoff