Commit Graph

102 Commits (37746173d94aa0fb14f6850c718274b778845e06)

Author SHA1 Message Date
Carol (Nichols || Goulding) 37746173d9 feat: Change azure object store to only get config from args, not env 2021-03-04 10:14:42 -05:00
Carol (Nichols || Goulding) 02d981451d feat: Implement Google Cloud Storage-related CLI arguments 2021-03-04 10:14:42 -05:00
Carol (Nichols || Goulding) ef13c1023e feat: Change google cloud object store to get config from args, not env 2021-03-04 10:14:42 -05:00
Carol (Nichols || Goulding) 06236e796b feat: Change aws object store to get config from args, not env 2021-03-04 10:14:42 -05:00
Carol (Nichols || Goulding) c7ef18337c feat: Consolidate all bucket config into one option/env var
Fixes #869.
2021-02-25 15:53:20 -05:00
Marko Mikulicic 12b768b8f1 fix: Escape empty string PathPart
Empty directory names are silently ignored and can lead to very surprising effects
such as directory layouts missing a level. This makes it hard to reason about directory structures.

A sane object store path API should either disallow empty names or deal with them gracefully.

Since we already have to escape file/directory names using the minimum common denominator valid character
set for known cloud providers, it feels quite natural to treat this empty dir/file name problem as encoding problem.
2021-02-24 21:13:56 +00:00
Marko Mikulicic 9e521a2ea1 feat: Plug GCS list_with_delimiter impl
And fix env.example. Now GCS can be used to persist snapshots.
2021-02-23 21:06:53 +00:00
Marko Mikulicic 9860def4b2 feat: Add S3 plumbing to iodx config 2021-02-23 14:55:55 +00:00
Carol (Nichols || Goulding) cff12da3a1 fix: Upgrade to released version of cloud_storage
Fixes #801.
2021-02-22 13:01:06 -05:00
Carol (Nichols || Goulding) a42103f436 Merge remote-tracking branch 'origin/main' into cn/google-list-with-delimiter 2021-02-22 12:53:46 -05:00
Marko Mikulicic 81739bf486 docs: Fix typo 2021-02-22 10:35:23 +00:00
Carol (Nichols || Goulding) cc6738c6f3 fix: Check for AZURE_STORAGE_MASTER_KEY in the test macro too 2021-02-18 16:53:06 -05:00
Carol (Nichols || Goulding) fcd4f91909 feat: Implement list_with_delimiter for Azure storage 2021-02-18 16:37:23 -05:00
Carol (Nichols || Goulding) 57942b51b7 feat: Update to latest Azure sdk to get delimiter support
Needed these PRs:
  - https://github.com/Azure/azure-sdk-for-rust/pull/176
  - https://github.com/Azure/azure-sdk-for-rust/pull/179

Also needed to enable the queue feature to get the azure_storage crate
compiling; at the moment, the code is still being reorganized and the
features aren't independent yet:
https://github.com/Azure/azure-sdk-for-rust/issues/177
2021-02-18 14:59:06 -05:00
Jake Goulding 484adcc257 chore: fix typo in an error message 2021-02-18 14:57:38 -05:00
Marko Mikulicic 536c1724bd feat: Allow to put streams of unknown length to objectstore
Addresses the API aspect of #818

Adds a utility module that helps computing the length of a stream while buffering it
for later replay (in-memory or spilling it in a temporary file).
2021-02-18 16:49:18 +00:00
Carol (Nichols || Goulding) f934a21efe test: Update tests to match new cloud storage error behavior 2021-02-17 14:23:34 -05:00
Carol (Nichols || Goulding) ef54131afb feat: Gets google cloud list_with_delimiter tests passing 2021-02-17 14:23:33 -05:00
Edd Robinson 2b642a8da6 refactor: add arc clone lint 2021-02-15 12:38:19 +00:00
Jake Goulding dad426d02e
fix: Report a failure to parse an AWS datetime (#794)
* fix: Report a failure to parse an AWS datetime

* refactor: use SNAFU context selectors instead of enum variants
2021-02-12 15:10:49 +00:00
Raphael Taylor-Davies c7e8a68fbe
fix: enable tokio::fs for object_store crate (#788)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-11 12:12:57 +00:00
Carol (Nichols || Goulding) c17feb998a feat: Implement display on FilePath 2021-02-08 15:13:25 -05:00
Carol (Nichols || Goulding) 076f67285d
Merge branch 'main' into cn+jg/file-delimiter 2021-02-05 09:45:34 -05:00
Carol (Nichols || Goulding) fbf776c6b3
chore: Clean up Cargo.tomls (#754)
* fix: test_helpers crate should only be a dev-dep

* fix: object_store no longer has a build script, so no longer needs a build dep

* chore: Alphabetize all Cargo.tomls
2021-02-04 18:56:02 -05:00
Carol (Nichols || Goulding) 8b18003e19 test: Don't check file metadata because SystemTime is not monotonic
See https://doc.rust-lang.org/std/time/struct.SystemTime.html
2021-02-04 15:46:11 -05:00
Carol (Nichols || Goulding) fa8594327d test: Add a better failure message to aid debugging 2021-02-04 15:12:33 -05:00
Carol (Nichols || Goulding) abbd29aeeb fix: Use Self in From impls 2021-02-04 13:40:33 -05:00
Carol (Nichols || Goulding) 3f1434e0e4 refactor: Remove redundant parts of error variant names
This info is now conveyed by the module each error comes from.
2021-02-04 13:04:52 -05:00
Carol (Nichols || Goulding) af7a5fa952 fix: Use walkdir::Errors within disk::Errors
I didn't want the object store lib Error to have to know about walkdir,
but I feel better about it now that this error type is scoped to the
disk module. The walkdir errors might have a bit more information.
2021-02-04 13:04:52 -05:00
Carol (Nichols || Goulding) 80581c9084 fix: Remove vestigial error types 2021-02-04 13:04:52 -05:00
Carol (Nichols || Goulding) f8fb24b88c refactor: Extract In-memory memory::Error 2021-02-04 13:04:52 -05:00
Carol (Nichols || Goulding) 8e6a06ebb2 refactor: Extract Azure azure::Error 2021-02-04 13:04:52 -05:00
Carol (Nichols || Goulding) 841f4ee314 refactor: Extract AWS S3 aws::Error 2021-02-04 13:04:52 -05:00
Carol (Nichols || Goulding) 1492e52e57 refactor: Extract Google Cloud Storage gcp::Error 2021-02-04 13:04:52 -05:00
Carol (Nichols || Goulding) f795c56c8d refactor: Start splitting up the object store error type; extract disk::Error
It's starting to get out of control. Time to fix that.
2021-02-04 13:04:52 -05:00
Carol (Nichols || Goulding) f9454fb57f feat: Implement list_with_delimiter for File object store
Fixes #688.
2021-02-04 13:04:52 -05:00
Carol (Nichols || Goulding) 5b18f7dbea feat: Hook DirsAndFileName push_part_as_dir to FilePath 2021-02-04 13:04:52 -05:00
Carol (Nichols || Goulding) a7cd8a2796 feat: Add a way to unset a file name in an object store path 2021-02-04 13:04:52 -05:00
Carol (Nichols || Goulding) c722188c5a feat: Connect parts_after_prefix from DirsAndFileName to FilePath
This will be useful in the File object store's list_with_delimiter.
2021-02-04 13:04:52 -05:00
Carol (Nichols || Goulding) c66efa80d1 feat: Implement PartialOrd and Ord for FilePath
This allows storing FilePaths in a BTreeSet and ordering FilePaths.
2021-02-04 13:04:52 -05:00
Carol (Nichols || Goulding) fd39315388 refactor: Improve error name
I thought this only had to do with the InMemory Put request, but it's a
bit more general than that. Hopefully clarifying the purpose of this
error.
2021-02-04 13:04:52 -05:00
Jake Goulding 678044e08a fix: test isn't a special extension recognized by object storage
This test was invalid because there are cases in which we use the
assumption that all file names in object storage should end with
`.json`, `.parquet`, or `.segment`.
2021-02-04 13:04:52 -05:00
Carol (Nichols || Goulding) 5d1c7dfe82 docs: Improve descriptive code comments as suggested in review 2021-02-01 14:56:49 -05:00
Carol (Nichols || Goulding) 5c8b351f57 fix: Address clippy suggestions 2021-02-01 14:56:49 -05:00
Carol (Nichols || Goulding) f9539f2b74 fix: Remove blanket trait impl now causing a stack overflow 2021-02-01 14:56:49 -05:00
Carol (Nichols || Goulding) ff6955a433 refactor: Extract a trait for ObjectStoreApi with associated path
This is the promised cleanup. This structure gets rid of a lot of
intermediate structures and encodes through associated types how the
object stores and path types are related.

The enums are still necessary to avoid having generics leak all over
the place, but the object store variants and path variants should always
match because they'll always come from the object store trait
implementations that use the associated types.
2021-02-01 14:56:47 -05:00
Carol (Nichols || Goulding) c40205b37e test: Move DirsAndFileName functionality tests with the definition 2021-02-01 14:39:18 -05:00
Carol (Nichols || Goulding) 596a73f56a refactor: Extract a FilePath type for use in file storage
Enforces that on-disk storage will only ever use file paths.

More cleanup coming!
2021-02-01 14:39:18 -05:00
Carol (Nichols || Goulding) d39131ab49 refactor: Extract a CloudPath type for use in cloud storage
This is the start of using the type system to enforce that only
CloudPaths will be used with S3, GCS, and Azure.

Still some mess in here, cleanup coming.
2021-02-01 14:39:16 -05:00
Carol (Nichols || Goulding) 7d3b4db234 fix: InMemory doesn't need pagination 2021-02-01 14:35:47 -05:00