Restarting the test server and testing for WAL restoration on startup
(and possible error scenarios) was disabled in 70ff426, which also noted
that significant refactoring will likely be needed when loading info on
startup is re-enabled. With that in mind, remove this code creating an
invalid dir in the WAL dir.
This adds functionality to the server to load database rules on startup. Follow on work will update the rules to store additional data (the catalog) and ensure that updates to the catalog can occur as outlined in #651. This work also updated the configuration to not require a database directory so the server can run entirely in memory. I needed this to get the end-to-end test passing since the file object store API doesn't yet have the functionality needed. I've logged #688 to track adding that in.
* refactor: consolidate line protocol schema creation into data_types, and port code to use it
refactor: Port mutable buffer to use SchemaBuilder
* fix: doctest
* refactor: remove unecessary clippyisms
* docs: Improve comments via suggestions from code review
Co-authored-by: Edd Robinson <me@edd.io>
* refactor: use more idomatic try_ naming and TryInto trait
* docs: Change from line protocol data model to InfluxDB data model
* refactor: rename LP --> Influx in code
* feat: add support for UInteger type
Co-authored-by: Edd Robinson <me@edd.io>
Replaces the hand-rolled config system with a StructOpt managed config struct.
I've got most of it ported across, but the interaction between all the logging
config bits is complex! I've left what is there and hooked in the value from
the config struct (which directly replaces the env var in usage, as it also
sources from the env).
* feat: add create and get database to API
This commit is start of the IOx specific API. It puts everything under /iox/api/v1 as this is the beginning of the IOx API. Creating a database is done with a PUT and a GET request can retrieve the DatabaseRules details.
* feat: add defaults for DatabaseRules for create_database
* feat: add create and get database to API
This commit is start of the IOx specific API. It puts everything under /iox/api/v1 as this is the beginning of the IOx API. Creating a database is done with a PUT and a GET request can retrieve the DatabaseRules details.
* feat: Create configuration system, port IOx to use it
* docs: Apply suggestions from code review
Co-authored-by: Paul Dix <paul@influxdata.com>
* fix: fix test for setting values
Co-authored-by: Paul Dix <paul@influxdata.com>
* chore: Connect server crate to http routes and server command
This updates the http_routes and main server to use the Server crate. This is the first step in a larger effort to start hooking up the initial IOx API and get things running end to end with in-memory database, WAL buffer, and object storage.
For the time being, this disables the previous disk based WAL. Or rather, it uses the WriteBufferDb without it. That means that this IOx server has no persistence until later. Because of this, the restart in the end-to-end was removed.
Later PRs will add the WAL buffer and restart logic that loads from object store. We can opt to bring the local disk based WAL back later, but it will likely require some refactoring to work with how the WAL Buffer will operate.
* feat: Update storage protobuf definitions, add stubs for read_window_aggregate
* refactor: Extract the features field in a clearer way
* docs: Add provenance information to service.proto
Connects to influxdata/fusion#59. Delorean currently ignores this
header.
Also add an example of using this to connect to an InfluxDB 2 instance;
I tested this out with a locally running Influx DB 2 and I was able to
write points!
The `/api/v2/create_bucket` API was delorean-specific for testing
purposes. This change makes it match the [Influx 2.0 API][influx] and
adds a method to the client for creating buckets.
The client will always send an empty array of `retentionRules` because
that is a required parameter for the Influx API. Delorean always ignores
`retentionRules`. The `description` and `rp` parameters are optional and
are never sent.
[influx]: https://v2.docs.influxdata.com/v2.0/api/#operation/PostBuckets
I believe the gRPC create bucket is also delorean-specific and perhaps
not needed, but I'm leaving it in for now with a note.
This commit extends the ingest crate to support converting multiple TSM
files to a single Parquet file by merging identical measurements across
the TSM files.
This does not yet support merging blocks that overlap.
This gets us built-in help text and error messages, and does less work
before failing because of an unsupported value.
Before this change, the help text was:
```
OPTIONS:
--compression-level <compression_level>
Compression level: max or compatibility (default). [default: compatibility]
```
After this change, the help text is:
```
OPTIONS:
--compression-level <compression_level>
How much to compress the output data. 'max' compresses the most; 'compatibility' compresses in a manner more
likely to be readable by other tools. [default: compatibility] [possible values: max, compatibility]
```
Before this change, if you supplied an unsupported value, the error was:
```
[2020-06-29T14:47:42Z INFO delorean::commands::convert] convert starting
[2020-06-29T14:47:42Z INFO delorean::commands::convert] Preparing to convert 591 bytes from tests/fixtures/lineproto/temperature.lp
Conversion failed: Error creating a parquet table writer Unknown compression level 'foo'. Valid options 'max' or 'compatibility'
```
After this change, the error is:
```
error: 'foo' isn't a valid value for '--compression-level <compression_level>'
[possible values: compatibility, max]
```
* feat: benchmark for lp->parquet performance
* feat: improve parser performance by storing contiguous EscapedStr
* fix: remove all string copies during LP-Parquet conversion
* refactor: Implement from_str as From<&str> only
* refactor: implement Deref instead of as_str
* refactor: Remove ends_with because Deref now makes it work
* refactor: Eq can be derived
* refactor: Remove unused From implementation
* refactor: Replace single-character strings with chars as requested by clippy
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@integer32.com>
* refactor: move all dstool code into delorean binary
* fix: Move code/mods to make it compile and run
* fix: warn if db dir does not exist
* refactor: Match argument subcommands w/ more idomatic rust
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
fix: restore hyper logging
fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: update expected code
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* chore: Add a test that decodes the entire tsm index
* fix: update test and change example to not use hard coded len
* fix: comment cleanup
* fix: clippy
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
* fix: fmt/clippy after code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
* refactor: rename the module containing generated types
The nested `delorean` was confusing anyway, and this will make more
sense when we extract a new crate.
* refactor: Move the generated types to their own crate
This allows us to have more lax warnings in that crate alone, keeping
the main crate more strict.
* style: Re-enable elided lifetimes lint in the main crate
Carol (Nichols || Goulding)
Paul Dix
Andrew Lamb
Dom
Paul Dix
alamb
Edd Robinson
Jake Goulding
Carol (Nichols || Goulding)