Closes#528
This patch adds support for Microsfot Azure Blob storage. The
implementations requires an account, a key and container name. They can
be configured via the environment variables `AZURE_STORAGE_ACCOUNT`,
`AZURE_STORAGE_MASTER_KEY` and `AZURE_STORAGE_CONTAINER`.
* feat: add create and get database to API
This commit is start of the IOx specific API. It puts everything under /iox/api/v1 as this is the beginning of the IOx API. Creating a database is done with a PUT and a GET request can retrieve the DatabaseRules details.
* feat: add defaults for DatabaseRules for create_database
* feat: add create and get database to API
This commit is start of the IOx specific API. It puts everything under /iox/api/v1 as this is the beginning of the IOx API. Creating a database is done with a PUT and a GET request can retrieve the DatabaseRules details.
I had stuff in my Google storage from when I was manually testing out
how paths are handled; it was hard to see that was the problem without
this extra failure text.
Now that the code is separated into modules, we don't need the modules
inside the test modules. So before this commit, the test names looked
like this:
```
test aws::tests::amazon_s3::s3_test_put_nonexistent_bucket ... ok
test gcp::test::google_cloud_storage::gcs_test ... ok
test disk::tests::file::length_mismatch_is_an_error ... ok
test memory::tests::in_memory::length_mismatch_is_an_error ... ok
```
and after this commit, the test names look like this:
```
test aws::tests::s3_test_put_nonexistent_bucket ... ok
test gcp::test::gcs_test ... ok
test disk::tests::length_mismatch_is_an_error ... ok
test memory::tests::length_mismatch_is_an_error ... ok
```
This adds the list_with_delimiter function to the in-memory object store. It also updates the function signature to require a prefix since it will always only want to list either the objects in the dir or the common prefixes.
This adds a new function list_with_delimiter to the object store. This commit contains just the implementation for S3, leaving the others to be completed in follow on commits.
This has a fixed delimiter to ensure a directory structure is created. This delimiter should be dependent on platform and which object store is used. For any of the cloud object stores or in memory, the delimiter should be /. For the future disk based implementation it should be dependendent on if you're running on Windows or Linux.
I didn't use Stream for the return type because I found it difficult to work with and I don't think it actually added anything useful. The return ListResult struct has the next token and I prefer that the caller explicitly makes calls that go over the network so they're more aware of what's going on, where a Stream abstracts that away so it's hidden behind the scenes. We can easilsy add a Stream based version on top of this existing API if we want.
This pulls the different backing implmenetations into their own modules. They're about to get more complex so it felt like it was time to separate them out rather than building towards a single multi-thousand line lib.rs. The error type is only defined in lib and imported by the individual modules, which I think makes it easier to work with.
* refactor: Create database with mutable buffer, read buffer and parquet files
* docs: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: rename planners to clarify what they are
* refactor: simplify traits
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* feat: Script to scrape benchmark results into line protocol
* feat: script to run benchmark on a checkout
* fix: rename
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* docs: update comments
* fix: use a single jq command invocation rather than 4
* fix: use one jq command rather than 2
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: check for directory existence too
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* feat: Create configuration system, port IOx to use it
* docs: Apply suggestions from code review
Co-authored-by: Paul Dix <paul@influxdata.com>
* fix: fix test for setting values
Co-authored-by: Paul Dix <paul@influxdata.com>
Edd Robinson
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
Andrew Lamb
Dom
Karsten Jeschkies
Paul Dix
Paul Dix