This is the initial prototype of the WriteBuffer and WAL. This does the following:
* accepts a slice of ParsedLine into the DB
* writes those into an in memory structure with tags represented as u32 dictionaries and all field types supported
* persists those writes into the WAL as Flatbuffer blobs (one WAL entry per slice of lines written, or WriteBatch)
* has a method to return a table from the buffer as an Arrow RecordBatch
* recovers the WAL after the database is closed and opened back up again
* has a single test that covers the end-to-end from the DB side
* It doesn't include partitioning yet. Although the write_lines method does actually try to do partitions on time. That'll get changed to be something more general defined by a per database configuration.
* hooked up to the v2 HTTP write API
* hooked up to a read API which will execute a SQL query against the data in the buffer
This includes a refactor of the WAL:
Refactors the WAL to remove async and threading so that it can be moved higher up. This simplifies the API while keeping just about the same amount of code in ParitionStore to handle the asynchronous writes.
This also modifies the WAL to remove the SideFile implementation, which was causing significant performance problems and write amplification. The downside is that WAL writes are no longer guarranteed atomic.
Further, this modifies the WAL to keep the active segement file handle open. Appends now don't have to list the directory contents and look for the latest file and open the file handle to do appends, which should also improve performance and reduce iops.
In the data generator, we're going to have potentially 2 different
clients for the data and for telemetry, and I'd like to be able to log
how they're configured, for example.
I don't see any reason for URL to
be private, because the user of the client passed in the URL and we
haven't changed anything about it.
Connects to influxdata/fusion#60
As far as I can tell, `/api/v2/write` uses `org` and `bucket` params
that can take org and bucket names, so just call those `org` and
`bucket` throughout and clarify in examples they can be any string.
However, `/api/v2/buckets` uses only `orgID` that must be the
organization's 16-digit hex ID, so clarify that case.
This means in examples where we create a bucket and push points into it,
we'll need both a hex ID and name for the organization.
Connects to influxdata/fusion#59. Delorean currently ignores this
header.
Also add an example of using this to connect to an InfluxDB 2 instance;
I tested this out with a locally running Influx DB 2 and I was able to
write points!
The `/api/v2/create_bucket` API was delorean-specific for testing
purposes. This change makes it match the [Influx 2.0 API][influx] and
adds a method to the client for creating buckets.
The client will always send an empty array of `retentionRules` because
that is a required parameter for the Influx API. Delorean always ignores
`retentionRules`. The `description` and `rp` parameters are optional and
are never sent.
[influx]: https://v2.docs.influxdata.com/v2.0/api/#operation/PostBuckets
I believe the gRPC create bucket is also delorean-specific and perhaps
not needed, but I'm leaving it in for now with a note.
Paul Dix
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
Andrew Lamb
Edd Robinson
alamb
Jake Goulding