* feat: Add database rules configuration for mutable buffer
* refactor: change all database rules usage to use mutable_buffer_config rather than store_locally
* feat: Change table_names to return either Some(set) or None, rather than a plan
* docs: improve comments
* docs: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: merge conflict
* fix: don't clone a string unless needed
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: test_helpers crate should only be a dev-dep
* fix: object_store no longer has a build script, so no longer needs a build dep
* chore: Alphabetize all Cargo.tomls
* feat: implement schema merging
* docs: Update data_types/src/schema.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* test: add test for merging schemas with no measurement names
* test: add test for merging and different measurements
* docs: clarify comments about merging
* fix: fmt
* fix: update comments to retrigger semantic pull request
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
This adds functionality to the server to load database rules on startup. Follow on work will update the rules to store additional data (the catalog) and ensure that updates to the catalog can occur as outlined in #651. This work also updated the configuration to not require a database directory so the server can run entirely in memory. I needed this to get the end-to-end test passing since the file object store API doesn't yet have the functionality needed. I've logged #688 to track adding that in.
* refactor: Remove import of unimplemented macro that's in the prelude
* refactor: Remove allowing of dead code that isn't dead anymore
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
This pulls the server configuration into its own module. It updates the config so that databases can be created and saved without holding a write lock across an async call. Follow on work will add reading of these rules from object store on server initialization.
* refactor: consolidate line protocol schema creation into data_types, and port code to use it
refactor: Port mutable buffer to use SchemaBuilder
* fix: doctest
* refactor: remove unecessary clippyisms
* docs: Improve comments via suggestions from code review
Co-authored-by: Edd Robinson <me@edd.io>
* refactor: use more idomatic try_ naming and TryInto trait
* docs: Change from line protocol data model to InfluxDB data model
* refactor: rename LP --> Influx in code
* feat: add support for UInteger type
Co-authored-by: Edd Robinson <me@edd.io>
This hooks the wal buffer up to the server. When segments go over their max size, they will be persisted if that configuration was set. A follow on commit will add persistence based on time (how long the segment has been open).
Adds serialization with compression and checksum for WAL buffer segments.
This required a weird structure where the flatbuffer bytes of ReplicatedWrite were kept as a raw payload. I did this because otherwise each of the replicated writes would have been rebuilt in the segment.
The other thing that isn't ideal is that deserializing a segment actually marshals it into a Rust struct as opposed to keeping the entire thing as raw flatbuffers. We could update this later to have a concept of an open segment (regular rust stuct) and closed segments that are just the flatbuffers.
* feat: add create and get database to API
This commit is start of the IOx specific API. It puts everything under /iox/api/v1 as this is the beginning of the IOx API. Creating a database is done with a PUT and a GET request can retrieve the DatabaseRules details.
* feat: add defaults for DatabaseRules for create_database
* feat: add create and get database to API
This commit is start of the IOx specific API. It puts everything under /iox/api/v1 as this is the beginning of the IOx API. Creating a database is done with a PUT and a GET request can retrieve the DatabaseRules details.
This adds benchmarks to the data_types crate for ReplicatedWrite. This is the first in a series to test benchmarking Flatbuffers vs. JSON for the WAL Segment format.
* feat: implement partition chunk rollover + ids and timestamps
* feat: add last_write_timestamp
* refactor: Use DateTime<Utc> rather than Instant
* refactor: avoid use of structure to generate ids
* feat: Implement write buffer to Parquet snapshotting
This introduces snapshot to the server packages to manage snapshotting. It also introduces a new trait for representing a Partition. There is a very crude API wired up in http_routes for testing purposes. Follow on work will bring the server package into http_routes and rework the snapshot API.
Previosuly the $ORG and $BUCKET was joined as:
$ORG + "_" + $BUCKET
Which is fine unless either $ORG or $BUCKET includes a "_", such as:
$ORG = "org_a"
$BUCKET = "bucket"
and
$ORG = "org"
$BUCKET = "a_bucket"
This change continues to join $ORG and $BUCKET with an underscore, but
disallows underscores in either $ORG or $BUCKET. It appears these values
are non-zero u64s in the gRPC protocol converted to their base-10 string
representations for the DB name, so this seems safe to enforce.
In addition, this change introduces a `DatabaseName` type to avoid
passing bare strings around, and allow consuming code to ensure only
valid database names are provided at compile type. This type works with
both owned & borrowed content so doesn't force a string copy where we
can avoid it, and derefs to `str` to make it easier to use with existing
code.
I've been minimally invasive in pushing the `DatabaseName` through the
existing code and figured I'd see what the sentement is first.
Candidates for conversion from `str` to `DatabaseName` that seem to make
sense to me include:
- `DatabaseStore` trait
- `RemoteServer` trait
- Others? Basically anywhere other than the "edge" API inputs
Fixes#436 (thanks @zeebo)
This splits the cluster package out into server and buffer modules. The WAL buffer is in-memory and split into segments. Follow on commits will implement it in the server and add persistence to object storage.
Raphael Taylor-Davies
Edd Robinson
Andrew Lamb
Paul Dix
Marko Mikulicic
Carol (Nichols || Goulding)
Paul Dix
Andrew Lamb
Dom
huming