Empty directory names are silently ignored and can lead to very surprising effects
such as directory layouts missing a level. This makes it hard to reason about directory structures.
A sane object store path API should either disallow empty names or deal with them gracefully.
Since we already have to escape file/directory names using the minimum common denominator valid character
set for known cloud providers, it feels quite natural to treat this empty dir/file name problem as encoding problem.
Addresses the API aspect of #818
Adds a utility module that helps computing the length of a stream while buffering it
for later replay (in-memory or spilling it in a temporary file).
* fix: test_helpers crate should only be a dev-dep
* fix: object_store no longer has a build script, so no longer needs a build dep
* chore: Alphabetize all Cargo.tomls
I didn't want the object store lib Error to have to know about walkdir,
but I feel better about it now that this error type is scoped to the
disk module. The walkdir errors might have a bit more information.
This test was invalid because there are cases in which we use the
assumption that all file names in object storage should end with
`.json`, `.parquet`, or `.segment`.
This is the promised cleanup. This structure gets rid of a lot of
intermediate structures and encodes through associated types how the
object stores and path types are related.
The enums are still necessary to avoid having generics leak all over
the place, but the object store variants and path variants should always
match because they'll always come from the object store trait
implementations that use the associated types.
This is the start of using the type system to enforce that only
CloudPaths will be used with S3, GCS, and Azure.
Still some mess in here, cleanup coming.
* chore: Update arrow + tokio deps
* chore: Use bleeding edge azure
* chore: Update aws + other deps
* fix: fmt
* fix: Switch to in-house version of routerify
* fix: Upgrade to hyper 0.14
The hyper::error module is now private; hyper::Error is the public
re-export
* fix: Upgrade cloud storage to get tokio upgrade
* fix: Upgrade open_telemetry
* fix: Do not call `panic::set_hook` during another panic
Doing so leads to a double panic which aborts the process.
* fix: new h2 error who dis
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@integer32.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
* refactor: Remove import of unimplemented macro that's in the prelude
* refactor: Remove allowing of dead code that isn't dead anymore
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Carol (Nichols || Goulding)
Marko Mikulicic
Jake Goulding
Edd Robinson
Raphael Taylor-Davies
Carol (Nichols || Goulding)
Andrew Lamb