I had stuff in my Google storage from when I was manually testing out
how paths are handled; it was hard to see that was the problem without
this extra failure text.
Now that the code is separated into modules, we don't need the modules
inside the test modules. So before this commit, the test names looked
like this:
```
test aws::tests::amazon_s3::s3_test_put_nonexistent_bucket ... ok
test gcp::test::google_cloud_storage::gcs_test ... ok
test disk::tests::file::length_mismatch_is_an_error ... ok
test memory::tests::in_memory::length_mismatch_is_an_error ... ok
```
and after this commit, the test names look like this:
```
test aws::tests::s3_test_put_nonexistent_bucket ... ok
test gcp::test::gcs_test ... ok
test disk::tests::length_mismatch_is_an_error ... ok
test memory::tests::length_mismatch_is_an_error ... ok
```
This adds the list_with_delimiter function to the in-memory object store. It also updates the function signature to require a prefix since it will always only want to list either the objects in the dir or the common prefixes.
This adds a new function list_with_delimiter to the object store. This commit contains just the implementation for S3, leaving the others to be completed in follow on commits.
This has a fixed delimiter to ensure a directory structure is created. This delimiter should be dependent on platform and which object store is used. For any of the cloud object stores or in memory, the delimiter should be /. For the future disk based implementation it should be dependendent on if you're running on Windows or Linux.
I didn't use Stream for the return type because I found it difficult to work with and I don't think it actually added anything useful. The return ListResult struct has the next token and I prefer that the caller explicitly makes calls that go over the network so they're more aware of what's going on, where a Stream abstracts that away so it's hidden behind the scenes. We can easilsy add a Stream based version on top of this existing API if we want.
This pulls the different backing implmenetations into their own modules. They're about to get more complex so it felt like it was time to separate them out rather than building towards a single multi-thousand line lib.rs. The error type is only defined in lib and imported by the individual modules, which I think makes it easier to work with.
* feat: Implement write buffer to Parquet snapshotting
This introduces snapshot to the server packages to manage snapshotting. It also introduces a new trait for representing a Partition. There is a very crude API wired up in http_routes for testing purposes. Follow on work will bring the server package into http_routes and rework the snapshot API.
Carol (Nichols || Goulding)
Paul Dix
Carol (Nichols || Goulding)
Dom
Brandon Sov
huming
Paul Dix
Andrew Lamb