fadf0176d4
feat: impl Hash for DirsAndFileName and PathParts
2021-05-07 09:51:44 +02:00
d726cf4829
refactor: make `object_store::path::parts` public
2021-05-07 09:51:44 +02:00
9eb658eac5
feat: s3 compatibility storages support
2021-05-06 23:27:37 +08:00
1f5a9b5ebf
feat: make DirsAndFileName members public
...
Helps with #1253 .
2021-05-03 09:51:00 +02:00
ca5351088a
feat: add Path->DirsAndFileName conversion
...
This allows for easier parsing of received paths.
Helps with #1253 .
2021-05-03 09:51:00 +02:00
140d96dbea
feat: tests ffor loading data to object store and make sure twe still query read buffer
2021-04-28 15:59:17 -04:00
eddc9319ff
docs: deny broken intradoc links
2021-04-27 13:22:28 +02:00
53e83f8385
feat: add throttled store wrapper
2021-04-27 09:31:48 +02:00
d8a92a8570
test: Don't run any integration tests unless TEST_INTEGRATION is set
2021-04-22 14:21:50 -04:00
c7e6a0a797
fix: Use wb rather than wal in object storage test path names for consistency
2021-04-21 17:43:03 +00:00
f136931225
fix: Inconsistent ordering lints
2021-04-19 08:48:11 -04:00
fd0da7e74a
chore: upgrade arrow and Rust
...
See https://github.com/apache/arrow/pull/10082 for upstream PR.
2021-04-19 14:00:04 +02:00
657bfa1b20
refactor: address Andrew's comments
2021-04-16 17:44:46 -04:00
4c23ca8888
feat: full implementation of parquet's read_filter for review
2021-04-16 16:03:24 -04:00
1997324344
feat: mutable buffer snapshotting ( #1179 )
...
* feat: mutable buffer snapshotting
* chore: review feedback
2021-04-13 12:14:54 +00:00
16cc37e7f3
chore: update cloud-storage to 0.9 ( #1165 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-09 14:37:51 +00:00
82588d5c72
fix: Don't return Result from test functions
2021-04-07 12:40:00 -04:00
8bd7a39607
fix: Use InstanceMetadataProvider directly
...
Rusoto's ChainProvider swallows the error message produced by the underlying InstanceMetadataProvider
(see d59d716f09/rusoto/credential/src/lib.rs (L397)
2021-03-15 13:55:43 +00:00
3c8a58266a
feat: Make S3 auth parameters optional
...
so that if not present, the aws client library can use builtin auth providers,
such as the InstanceMetadataProvider, which is commonly used to get the credentials
granted to the AWS VM via cloud native mechanism.
2021-03-15 10:11:01 +01:00
d67a03f616
fix: Improved tests found another unimplemented hiding
2021-03-04 16:18:31 -05:00
ca2f74063e
fix: Test using the object store wrapper interface
2021-03-04 16:18:11 -05:00
16df4c542b
fix: Rename AZURE_STORAGE_MASTER_KEY to AZURE_STORAGE_ACCESS_KEY
2021-03-04 10:15:46 -05:00
e9fedfae17
fix: Use AWS_DEFAULT_REGION instead of AWS_REGION
2021-03-04 10:15:30 -05:00
37746173d9
feat: Change azure object store to only get config from args, not env
2021-03-04 10:14:42 -05:00
02d981451d
feat: Implement Google Cloud Storage-related CLI arguments
2021-03-04 10:14:42 -05:00
ef13c1023e
feat: Change google cloud object store to get config from args, not env
2021-03-04 10:14:42 -05:00
06236e796b
feat: Change aws object store to get config from args, not env
2021-03-04 10:14:42 -05:00
c7ef18337c
feat: Consolidate all bucket config into one option/env var
...
Fixes #869 .
2021-02-25 15:53:20 -05:00
12b768b8f1
fix: Escape empty string PathPart
...
Empty directory names are silently ignored and can lead to very surprising effects
such as directory layouts missing a level. This makes it hard to reason about directory structures.
A sane object store path API should either disallow empty names or deal with them gracefully.
Since we already have to escape file/directory names using the minimum common denominator valid character
set for known cloud providers, it feels quite natural to treat this empty dir/file name problem as encoding problem.
2021-02-24 21:13:56 +00:00
9e521a2ea1
feat: Plug GCS list_with_delimiter impl
...
And fix env.example. Now GCS can be used to persist snapshots.
2021-02-23 21:06:53 +00:00
9860def4b2
feat: Add S3 plumbing to iodx config
2021-02-23 14:55:55 +00:00
cff12da3a1
fix: Upgrade to released version of cloud_storage
...
Fixes #801 .
2021-02-22 13:01:06 -05:00
a42103f436
Merge remote-tracking branch 'origin/main' into cn/google-list-with-delimiter
2021-02-22 12:53:46 -05:00
81739bf486
docs: Fix typo
2021-02-22 10:35:23 +00:00
cc6738c6f3
fix: Check for AZURE_STORAGE_MASTER_KEY in the test macro too
2021-02-18 16:53:06 -05:00
fcd4f91909
feat: Implement list_with_delimiter for Azure storage
2021-02-18 16:37:23 -05:00
57942b51b7
feat: Update to latest Azure sdk to get delimiter support
...
Needed these PRs:
- https://github.com/Azure/azure-sdk-for-rust/pull/176
- https://github.com/Azure/azure-sdk-for-rust/pull/179
Also needed to enable the queue feature to get the azure_storage crate
compiling; at the moment, the code is still being reorganized and the
features aren't independent yet:
https://github.com/Azure/azure-sdk-for-rust/issues/177
2021-02-18 14:59:06 -05:00
484adcc257
chore: fix typo in an error message
2021-02-18 14:57:38 -05:00
536c1724bd
feat: Allow to put streams of unknown length to objectstore
...
Addresses the API aspect of #818
Adds a utility module that helps computing the length of a stream while buffering it
for later replay (in-memory or spilling it in a temporary file).
2021-02-18 16:49:18 +00:00
f934a21efe
test: Update tests to match new cloud storage error behavior
2021-02-17 14:23:34 -05:00
ef54131afb
feat: Gets google cloud list_with_delimiter tests passing
2021-02-17 14:23:33 -05:00
2b642a8da6
refactor: add arc clone lint
2021-02-15 12:38:19 +00:00
dad426d02e
fix: Report a failure to parse an AWS datetime ( #794 )
...
* fix: Report a failure to parse an AWS datetime
* refactor: use SNAFU context selectors instead of enum variants
2021-02-12 15:10:49 +00:00
c7e8a68fbe
fix: enable tokio::fs for object_store crate ( #788 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-11 12:12:57 +00:00
c17feb998a
feat: Implement display on FilePath
2021-02-08 15:13:25 -05:00
076f67285d
Merge branch 'main' into cn+jg/file-delimiter
2021-02-05 09:45:34 -05:00
fbf776c6b3
chore: Clean up Cargo.tomls ( #754 )
...
* fix: test_helpers crate should only be a dev-dep
* fix: object_store no longer has a build script, so no longer needs a build dep
* chore: Alphabetize all Cargo.tomls
2021-02-04 18:56:02 -05:00
8b18003e19
test: Don't check file metadata because SystemTime is not monotonic
...
See https://doc.rust-lang.org/std/time/struct.SystemTime.html
2021-02-04 15:46:11 -05:00
fa8594327d
test: Add a better failure message to aid debugging
2021-02-04 15:12:33 -05:00
abbd29aeeb
fix: Use Self in From impls
2021-02-04 13:40:33 -05:00
3f1434e0e4
refactor: Remove redundant parts of error variant names
...
This info is now conveyed by the module each error comes from.
2021-02-04 13:04:52 -05:00
af7a5fa952
fix: Use walkdir::Errors within disk::Errors
...
I didn't want the object store lib Error to have to know about walkdir,
but I feel better about it now that this error type is scoped to the
disk module. The walkdir errors might have a bit more information.
2021-02-04 13:04:52 -05:00
80581c9084
fix: Remove vestigial error types
2021-02-04 13:04:52 -05:00
f8fb24b88c
refactor: Extract In-memory memory::Error
2021-02-04 13:04:52 -05:00
8e6a06ebb2
refactor: Extract Azure azure::Error
2021-02-04 13:04:52 -05:00
841f4ee314
refactor: Extract AWS S3 aws::Error
2021-02-04 13:04:52 -05:00
1492e52e57
refactor: Extract Google Cloud Storage gcp::Error
2021-02-04 13:04:52 -05:00
f795c56c8d
refactor: Start splitting up the object store error type; extract disk::Error
...
It's starting to get out of control. Time to fix that.
2021-02-04 13:04:52 -05:00
f9454fb57f
feat: Implement list_with_delimiter for File object store
...
Fixes #688 .
2021-02-04 13:04:52 -05:00
5b18f7dbea
feat: Hook DirsAndFileName push_part_as_dir to FilePath
2021-02-04 13:04:52 -05:00
a7cd8a2796
feat: Add a way to unset a file name in an object store path
2021-02-04 13:04:52 -05:00
c722188c5a
feat: Connect parts_after_prefix from DirsAndFileName to FilePath
...
This will be useful in the File object store's list_with_delimiter.
2021-02-04 13:04:52 -05:00
c66efa80d1
feat: Implement PartialOrd and Ord for FilePath
...
This allows storing FilePaths in a BTreeSet and ordering FilePaths.
2021-02-04 13:04:52 -05:00
fd39315388
refactor: Improve error name
...
I thought this only had to do with the InMemory Put request, but it's a
bit more general than that. Hopefully clarifying the purpose of this
error.
2021-02-04 13:04:52 -05:00
678044e08a
fix: test isn't a special extension recognized by object storage
...
This test was invalid because there are cases in which we use the
assumption that all file names in object storage should end with
`.json`, `.parquet`, or `.segment`.
2021-02-04 13:04:52 -05:00
5d1c7dfe82
docs: Improve descriptive code comments as suggested in review
2021-02-01 14:56:49 -05:00
5c8b351f57
fix: Address clippy suggestions
2021-02-01 14:56:49 -05:00
f9539f2b74
fix: Remove blanket trait impl now causing a stack overflow
2021-02-01 14:56:49 -05:00
ff6955a433
refactor: Extract a trait for ObjectStoreApi with associated path
...
This is the promised cleanup. This structure gets rid of a lot of
intermediate structures and encodes through associated types how the
object stores and path types are related.
The enums are still necessary to avoid having generics leak all over
the place, but the object store variants and path variants should always
match because they'll always come from the object store trait
implementations that use the associated types.
2021-02-01 14:56:47 -05:00
c40205b37e
test: Move DirsAndFileName functionality tests with the definition
2021-02-01 14:39:18 -05:00
596a73f56a
refactor: Extract a FilePath type for use in file storage
...
Enforces that on-disk storage will only ever use file paths.
More cleanup coming!
2021-02-01 14:39:18 -05:00
d39131ab49
refactor: Extract a CloudPath type for use in cloud storage
...
This is the start of using the type system to enforce that only
CloudPaths will be used with S3, GCS, and Azure.
Still some mess in here, cleanup coming.
2021-02-01 14:39:16 -05:00
7d3b4db234
fix: InMemory doesn't need pagination
2021-02-01 14:35:47 -05:00
fdbe602e57
refactor: Always get a path to build from the object store
2021-02-01 14:30:21 -05:00
f3bd8bd0e3
chore: update deps (tokio 1.0 and ecosystem) ( #707 )
...
* chore: Update arrow + tokio deps
* chore: Use bleeding edge azure
* chore: Update aws + other deps
* fix: fmt
* fix: Switch to in-house version of routerify
* fix: Upgrade to hyper 0.14
The hyper::error module is now private; hyper::Error is the public
re-export
* fix: Upgrade cloud storage to get tokio upgrade
* fix: Upgrade open_telemetry
* fix: Do not call `panic::set_hook` during another panic
Doing so leads to a double panic which aborts the process.
* fix: new h2 error who dis
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@integer32.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2021-01-29 16:11:55 -05:00
6bb91653c1
refactor: Some tiny cleanups ( #680 )
...
* refactor: Remove import of unimplemented macro that's in the prelude
* refactor: Remove allowing of dead code that isn't dead anymore
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-01-21 07:27:17 -05:00
979458034a
refactor: Extract PathPart to its own module
...
And be more explicit about the privacy of the inner String
2021-01-15 15:29:01 -05:00
004fb8f803
refactor: Extract a file module for file-based paths
2021-01-15 15:26:35 -05:00
ac22d2bb44
refactor: Extract a cloud module to deal with cloud paths
2021-01-15 15:24:49 -05:00
f8956dfbe8
refactor: Extract a module for the parsed DirsAndFileName structure
2021-01-15 15:24:48 -05:00
813092649d
fix: Make file behave the same as other object stores with paths
2021-01-15 10:25:05 -05:00
0415d4a186
fix: Only treat PathBuf ending parts that match one of our extensions as file names
2021-01-15 10:15:00 -05:00
25cc396f5e
test: Handle macOS temp directories that start with . better
2021-01-15 10:15:00 -05:00
4acf0f6ea9
fix: Take prefix's file name into account in
...
This makes the in memory object store behave consistently with the cloud
object stores.
2021-01-13 09:57:13 -05:00
ea719724e9
test: Add another filename to clarify behavior in this test
2021-01-13 09:56:26 -05:00
383e3abfce
fix: Remove dbgs
2021-01-11 16:57:38 -05:00
8570c88689
test: Add a partial file name scenario to list_with_delimiter tests
2021-01-11 16:57:38 -05:00
f0ab0e25a0
fix: Match partial directory names with prefix
2021-01-11 16:57:38 -05:00
7c457710ee
fix: Implement parts_after_prefix; InMemory now passes
2021-01-11 16:57:38 -05:00
06f1358e2d
feat: Change ObjectStorePath API to be more explicit
...
Now you have to designate whether you're adding a directory or a file
name, with some assumptions based on paths coming from a cloud object
storage or the file system.
A notable difference: checking to see if "apple/b" is a prefix of
"apple/bear/cow.json" will now say no; only whole directories are
matched.
2021-01-11 16:57:37 -05:00
2cd383af6f
feat: Azure support for object store
...
Closes #528
This patch adds support for Microsfot Azure Blob storage. The
implementations requires an account, a key and container name. They can
be configured via the environment variables `AZURE_STORAGE_ACCOUNT`,
`AZURE_STORAGE_MASTER_KEY` and `AZURE_STORAGE_CONTAINER`.
2021-01-08 16:27:17 +01:00
cef0bb7c98
feat: Implement starts_with on ObjectStorePath
2021-01-07 16:51:32 -05:00
535e65c02a
refactor: Use itertools' extend with the iter instead of collecting
2021-01-07 16:02:23 -05:00
91c4e26628
feat: Disallow parts of paths to be only one or two dots
2021-01-07 16:02:20 -05:00
23782dc9b7
test: Add some tests around path building and encoding
2021-01-07 16:02:10 -05:00
164c0e7357
fix: Use DELIMITER to create DELIMITER_BYTE
2021-01-07 15:23:51 -05:00
37056a1753
feat: Decode PathPart's values when Displaying
2021-01-07 15:23:32 -05:00
b421de77c4
feat: Encode characters GCS recommends avoiding
2021-01-07 14:58:58 -05:00
18ee1b561b
feat: Use ObjectStorePath everywhere to feel out the API needed
2021-01-07 10:48:22 -05:00
e58607f015
fix: Add more info to a test failure message
...
I had stuff in my Google storage from when I was manually testing out
how paths are handled; it was hard to see that was the problem without
this extra failure text.
2021-01-07 09:21:00 -05:00
Marco Neumann
DCjanus
Nga Tran
Carol (Nichols || Goulding)
Raphael Taylor-Davies
Jake Goulding
Marko Mikulicic
Edd Robinson
Carol (Nichols || Goulding)
Andrew Lamb
Karsten Jeschkies