8bd7a39607
fix: Use InstanceMetadataProvider directly
...
Rusoto's ChainProvider swallows the error message produced by the underlying InstanceMetadataProvider
(see d59d716f09/rusoto/credential/src/lib.rs (L397)
2021-03-15 13:55:43 +00:00
3c8a58266a
feat: Make S3 auth parameters optional
...
so that if not present, the aws client library can use builtin auth providers,
such as the InstanceMetadataProvider, which is commonly used to get the credentials
granted to the AWS VM via cloud native mechanism.
2021-03-15 10:11:01 +01:00
d67a03f616
fix: Improved tests found another unimplemented hiding
2021-03-04 16:18:31 -05:00
ca2f74063e
fix: Test using the object store wrapper interface
2021-03-04 16:18:11 -05:00
16df4c542b
fix: Rename AZURE_STORAGE_MASTER_KEY to AZURE_STORAGE_ACCESS_KEY
2021-03-04 10:15:46 -05:00
e9fedfae17
fix: Use AWS_DEFAULT_REGION instead of AWS_REGION
2021-03-04 10:15:30 -05:00
37746173d9
feat: Change azure object store to only get config from args, not env
2021-03-04 10:14:42 -05:00
02d981451d
feat: Implement Google Cloud Storage-related CLI arguments
2021-03-04 10:14:42 -05:00
ef13c1023e
feat: Change google cloud object store to get config from args, not env
2021-03-04 10:14:42 -05:00
06236e796b
feat: Change aws object store to get config from args, not env
2021-03-04 10:14:42 -05:00
c7ef18337c
feat: Consolidate all bucket config into one option/env var
...
Fixes #869 .
2021-02-25 15:53:20 -05:00
12b768b8f1
fix: Escape empty string PathPart
...
Empty directory names are silently ignored and can lead to very surprising effects
such as directory layouts missing a level. This makes it hard to reason about directory structures.
A sane object store path API should either disallow empty names or deal with them gracefully.
Since we already have to escape file/directory names using the minimum common denominator valid character
set for known cloud providers, it feels quite natural to treat this empty dir/file name problem as encoding problem.
2021-02-24 21:13:56 +00:00
9e521a2ea1
feat: Plug GCS list_with_delimiter impl
...
And fix env.example. Now GCS can be used to persist snapshots.
2021-02-23 21:06:53 +00:00
9860def4b2
feat: Add S3 plumbing to iodx config
2021-02-23 14:55:55 +00:00
cff12da3a1
fix: Upgrade to released version of cloud_storage
...
Fixes #801 .
2021-02-22 13:01:06 -05:00
a42103f436
Merge remote-tracking branch 'origin/main' into cn/google-list-with-delimiter
2021-02-22 12:53:46 -05:00
81739bf486
docs: Fix typo
2021-02-22 10:35:23 +00:00
cc6738c6f3
fix: Check for AZURE_STORAGE_MASTER_KEY in the test macro too
2021-02-18 16:53:06 -05:00
fcd4f91909
feat: Implement list_with_delimiter for Azure storage
2021-02-18 16:37:23 -05:00
57942b51b7
feat: Update to latest Azure sdk to get delimiter support
...
Needed these PRs:
- https://github.com/Azure/azure-sdk-for-rust/pull/176
- https://github.com/Azure/azure-sdk-for-rust/pull/179
Also needed to enable the queue feature to get the azure_storage crate
compiling; at the moment, the code is still being reorganized and the
features aren't independent yet:
https://github.com/Azure/azure-sdk-for-rust/issues/177
2021-02-18 14:59:06 -05:00
484adcc257
chore: fix typo in an error message
2021-02-18 14:57:38 -05:00
536c1724bd
feat: Allow to put streams of unknown length to objectstore
...
Addresses the API aspect of #818
Adds a utility module that helps computing the length of a stream while buffering it
for later replay (in-memory or spilling it in a temporary file).
2021-02-18 16:49:18 +00:00
f934a21efe
test: Update tests to match new cloud storage error behavior
2021-02-17 14:23:34 -05:00
ef54131afb
feat: Gets google cloud list_with_delimiter tests passing
2021-02-17 14:23:33 -05:00
2b642a8da6
refactor: add arc clone lint
2021-02-15 12:38:19 +00:00
dad426d02e
fix: Report a failure to parse an AWS datetime ( #794 )
...
* fix: Report a failure to parse an AWS datetime
* refactor: use SNAFU context selectors instead of enum variants
2021-02-12 15:10:49 +00:00
c7e8a68fbe
fix: enable tokio::fs for object_store crate ( #788 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-11 12:12:57 +00:00
c17feb998a
feat: Implement display on FilePath
2021-02-08 15:13:25 -05:00
076f67285d
Merge branch 'main' into cn+jg/file-delimiter
2021-02-05 09:45:34 -05:00
fbf776c6b3
chore: Clean up Cargo.tomls ( #754 )
...
* fix: test_helpers crate should only be a dev-dep
* fix: object_store no longer has a build script, so no longer needs a build dep
* chore: Alphabetize all Cargo.tomls
2021-02-04 18:56:02 -05:00
8b18003e19
test: Don't check file metadata because SystemTime is not monotonic
...
See https://doc.rust-lang.org/std/time/struct.SystemTime.html
2021-02-04 15:46:11 -05:00
fa8594327d
test: Add a better failure message to aid debugging
2021-02-04 15:12:33 -05:00
abbd29aeeb
fix: Use Self in From impls
2021-02-04 13:40:33 -05:00
3f1434e0e4
refactor: Remove redundant parts of error variant names
...
This info is now conveyed by the module each error comes from.
2021-02-04 13:04:52 -05:00
af7a5fa952
fix: Use walkdir::Errors within disk::Errors
...
I didn't want the object store lib Error to have to know about walkdir,
but I feel better about it now that this error type is scoped to the
disk module. The walkdir errors might have a bit more information.
2021-02-04 13:04:52 -05:00
80581c9084
fix: Remove vestigial error types
2021-02-04 13:04:52 -05:00
f8fb24b88c
refactor: Extract In-memory memory::Error
2021-02-04 13:04:52 -05:00
8e6a06ebb2
refactor: Extract Azure azure::Error
2021-02-04 13:04:52 -05:00
841f4ee314
refactor: Extract AWS S3 aws::Error
2021-02-04 13:04:52 -05:00
1492e52e57
refactor: Extract Google Cloud Storage gcp::Error
2021-02-04 13:04:52 -05:00
f795c56c8d
refactor: Start splitting up the object store error type; extract disk::Error
...
It's starting to get out of control. Time to fix that.
2021-02-04 13:04:52 -05:00
f9454fb57f
feat: Implement list_with_delimiter for File object store
...
Fixes #688 .
2021-02-04 13:04:52 -05:00
5b18f7dbea
feat: Hook DirsAndFileName push_part_as_dir to FilePath
2021-02-04 13:04:52 -05:00
a7cd8a2796
feat: Add a way to unset a file name in an object store path
2021-02-04 13:04:52 -05:00
c722188c5a
feat: Connect parts_after_prefix from DirsAndFileName to FilePath
...
This will be useful in the File object store's list_with_delimiter.
2021-02-04 13:04:52 -05:00
c66efa80d1
feat: Implement PartialOrd and Ord for FilePath
...
This allows storing FilePaths in a BTreeSet and ordering FilePaths.
2021-02-04 13:04:52 -05:00
fd39315388
refactor: Improve error name
...
I thought this only had to do with the InMemory Put request, but it's a
bit more general than that. Hopefully clarifying the purpose of this
error.
2021-02-04 13:04:52 -05:00
678044e08a
fix: test isn't a special extension recognized by object storage
...
This test was invalid because there are cases in which we use the
assumption that all file names in object storage should end with
`.json`, `.parquet`, or `.segment`.
2021-02-04 13:04:52 -05:00
5d1c7dfe82
docs: Improve descriptive code comments as suggested in review
2021-02-01 14:56:49 -05:00
5c8b351f57
fix: Address clippy suggestions
2021-02-01 14:56:49 -05:00
Marko Mikulicic
Carol (Nichols || Goulding)
Jake Goulding
Edd Robinson
Raphael Taylor-Davies
Carol (Nichols || Goulding)