1dda568d28
test: Add a case for a path without a filename
2021-10-18 08:46:00 -04:00
d5ab29711e
fix: Serialize relative db object store paths to the server config
...
So that they can be deserialized, without parsing, to create a new
iox object store from the location listed in the server config.
Notably, the locations serialized don't start with the object storage's
prefix like "s3:" or "file:". The location is the same object storage as
the server configuration that was just read from object storage. Having
the server config on one type of object storage and the database files
on another type is not supported.
2021-10-18 08:37:36 -04:00
44d7479db1
chore(deps): bump cloud-storage from 0.10.2 to 0.10.3
...
Bumps [cloud-storage](https://github.com/ThouCheese/cloud-storage-rs ) from 0.10.2 to 0.10.3.
- [Release notes](https://github.com/ThouCheese/cloud-storage-rs/releases )
- [Changelog](https://github.com/ThouCheese/cloud-storage-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ThouCheese/cloud-storage-rs/commits )
---
updated-dependencies:
- dependency-name: cloud-storage
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-04 01:43:09 +00:00
fb0e6ad5b1
docs: Document the s3_request generics and how they fit together
2021-09-29 08:19:33 -04:00
2b9612a033
fix: Remove some of the futures in futures layers
...
I think the FuturesRetry crate needed this, but we're no longer using
the crate so we can get rid of some of the complexity.
2021-09-29 08:19:33 -04:00
ad1405dcdf
fix: Reduce number of AWS S3 request retries from 10 to 3
2021-09-29 08:19:32 -04:00
5857edb19a
fix: cloud_storage crate returns a different error type in this case
2021-09-29 08:19:32 -04:00
92583aee82
fix: Remove streaming API since we're not streaming anyway
2021-09-29 08:19:32 -04:00
edd6c12e93
fix: Add logging when retrying AWS requests
2021-09-29 08:19:32 -04:00
ea4cb9a6c1
refactor: Extract the maximum number of retries into a documented public constant
2021-09-29 08:19:32 -04:00
9cf343db08
refactor: Use loop instead of futures_retry
2021-09-29 08:19:32 -04:00
9cdccae49d
fix: Actually do the exponential part of exponential backoff
2021-09-29 08:19:32 -04:00
d69472e114
refactor: Use s3_request in s3 delete
2021-09-29 08:19:32 -04:00
d05528bcfd
refactor: Use s3_request for put requests
...
Which meant we also needed to change the byte stream to be a closure
that can generate a byte stream
2021-09-29 08:19:32 -04:00
dd3b405727
fix: Only retry s3 requests if error is 500
2021-09-29 08:19:32 -04:00
936fba24b9
refactor: Make s3_request generic over the return type
2021-09-29 08:19:32 -04:00
5c78b7d5ae
refactor: Extract an s3_request method that handles retries
2021-09-29 08:19:32 -04:00
fd3a027ca8
refactor: Use the extracted aws list function in list_with_delimiter
2021-09-29 08:19:32 -04:00
a1a8bc01bf
refactor: Extract a method just to stream aws list requests
2021-09-29 08:19:32 -04:00
9cdd4799c7
chore: `object_store` needs `reqwest` only for azure
2021-09-22 11:16:00 +02:00
ef5ab67c77
chore: upgrade azure dependencies
2021-09-21 13:35:11 +02:00
4382bc0b71
feat: disable Snafu futures feature ( #2573 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-19 11:36:24 +00:00
b67610d9b9
chore(deps): bump tokio from 1.10.1 to 1.11.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.10.1 to 1.11.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.10.1...tokio-1.11.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 09:11:38 +00:00
fbf18813ac
fix: Correct a mistaken assumption in in-memory object storage
...
The implementation of list_with_delimiter for the in-memory object
storage assumed that paths returned from the BTreeMap keys that sorted
greater than the prefix given to list_with_delimiter and for whom
prefix_matches returned true would also have parts after the prefix.
This didn't account for paths that started with the prefix but didn't
immediately have the delimiter after the prefix: that is,
prefix = 1/database_name
would match the in-memory paths:
1/database_name/0/rules.pb
1/database_name_and_another_thing/0/rules.pb
The first path here *would* return some parts_after_prefix, but the
second path would not and the previously existing code would panic for
the added path in the list_with_delimiter test case.
2021-09-02 14:57:27 -04:00
37c81fb483
chore(deps): bump cloud-storage from 0.9.0 to 0.10.2
...
Bumps [cloud-storage](https://github.com/ThouCheese/cloud-storage-rs ) from 0.9.0 to 0.10.2.
- [Release notes](https://github.com/ThouCheese/cloud-storage-rs/releases )
- [Changelog](https://github.com/ThouCheese/cloud-storage-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ThouCheese/cloud-storage-rs/commits )
---
updated-dependencies:
- dependency-name: cloud-storage
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-23 11:26:28 +00:00
063f6da7d6
docs: remove outdated comment ( #2302 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-17 12:57:07 +00:00
44f9f220cd
Merge branch 'main' into dependabot/cargo/rusoto_credential-0.47.0
2021-08-17 11:38:54 +00:00
bda4458ebd
fix: Update all the rusoto crates together
2021-08-16 19:59:26 -04:00
c012e996ab
refactor: remove display methods, use fmt::Display instead. ( #2272 )
...
* refactor: remove display methods, use fmt::Display instead.
Signed-off-by: Ning Sun <sunng@protonmail.com>
* refactor: update a few calls from .display to .to_string()
* fix: consistently use `Path` rather than occasionally `DirsAndFileName`
* fix: fixup for merge conflicts
* fix: update test
* fix: Catch another case or two
* fix: fmt
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-16 18:00:22 +00:00
97ed42c42f
build(deps): bump rusoto_credential from 0.46.0 to 0.47.0
...
Bumps [rusoto_credential](https://github.com/rusoto/rusoto ) from 0.46.0 to 0.47.0.
- [Release notes](https://github.com/rusoto/rusoto/releases )
- [Changelog](https://github.com/rusoto/rusoto/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rusoto/rusoto/compare/mock-v0.46.0...rusoto-v0.47.0 )
---
updated-dependencies:
- dependency-name: rusoto_credential
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-16 16:35:09 +00:00
ae6b0e669b
refactor: Extract a database persister type that wraps object store
...
Connects to #2193 .
2021-08-12 15:05:32 -04:00
3de6b44e23
build: use new rustdoc lint name ( #2261 )
...
* fix: nocache feature code rot
The MBChunk::snapshot code when using the "nocache" option no longer
compiles - this commit updates it to match the not(nocache) code.
* build: use updated broken_intra_doc_links name
The broken_intra_doc_links lint was renamed
rustdoc::broken_intra_doc_links
https://doc.rust-lang.org/rustdoc/lints.html
2021-08-11 19:48:51 +00:00
4c4bb1a93e
feat: add fmt::Display implementation for path data structs ( #2223 )
...
Signed-off-by: Ning Sun <sunng@protonmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-09 18:09:09 +00:00
9d15798288
fix: Address or allow Clippy warnings new with Rust 1.54
2021-07-30 09:59:59 -04:00
d95b5df03e
refactor: move cache to ObjectStore
...
Since the consumers of ObjectStore always use the concrete type rather than the ObjectStoreApi trait, it makes more sense to just change the concrete type to have a pointer to the cache. This removes the cache from the ObjectStoreApi trait and changes the ObjectStore to be a regular struct rather than a tuple around the ObjectStoreIntegration. Future work will have the server configure the cache on the ObjectStore struct when its options are set.
2021-07-21 18:27:56 -04:00
47044d537c
chore: make cache a type in object store trait
2021-07-21 18:27:56 -04:00
d0ea812041
feat: add skeleton for object store file cache
2021-07-21 18:27:56 -04:00
939d15a21f
perf: Avoid clone when an error doesn't occur
2021-07-16 09:59:54 -04:00
d35b74c226
fix: Fix doc build warnings ( #1945 )
...
* fix: Fix doc build warnings
* refactor: add deny bare_urls to crates
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-13 08:03:42 +00:00
670826daf9
refactor: make object_store construction interface consistent ( #1944 )
...
* refactor: make object_store construction interface consistent
* fix: benchmarks
* fix: doc build
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-12 12:56:36 +00:00
1a79bf7e99
refactor: Make aws/azure/gcs optional features and stop compiling 100 dependencies during dev ( #1933 )
...
* feat: make aws, gcp, azure dependencies optional
* fix: only run object store tests if the features are enabled
* fix: clean up testing
* fix: rename step
* fix: add to list of jobs
* fix: remove test with object store
* fix: review comments
2021-07-09 11:38:30 +00:00
3cb8f297b1
refactor: encapsulate the ObjectStore implementations in the object store crate ( #1932 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-09 10:38:32 +00:00
24b249ad7b
fix: do not silence all IO errors in disk-based LIST
2021-06-22 18:20:36 +02:00
ec43a87909
chore: Update itertools deps ( #1750 )
2021-06-17 17:56:44 +00:00
0e2a291683
chore: enforce `clippy::future_not_send` for `object_store`
2021-06-09 17:56:43 +02:00
bd39954d73
fix: do not error on `list_with_delimiter` w/ non-existing prefix
...
Disk-driven object stores were reporting `UnableToProcessEntry` when
calling `list_with_delimiter` and a non-existing prefix.
To prevent regressions add a test to the test suite shared by all
store implementations.
2021-06-03 10:25:20 +02:00
1cbe4031d3
feat: allow passing a session token to AWS stores
2021-05-27 11:53:12 +02:00
d5df7c381a
test: accept optional `AWS_ENDPOINT` for OS tests
2021-05-26 18:14:58 +02:00
464667d8b8
docs: clarify the FilePath directory handling
2021-05-20 14:20:56 +02:00
ccd094dfcf
refactor: streamline directory handling in path parsing
2021-05-20 10:22:20 +02:00
ff004c1a51
feat: add `ObjectStore.path_from_dirs_and_filename`
2021-05-17 11:33:22 +02:00
4d105713a5
feat: add convenience `parsed_path!` marco
2021-05-07 09:51:44 +02:00
fadf0176d4
feat: impl Hash for DirsAndFileName and PathParts
2021-05-07 09:51:44 +02:00
d726cf4829
refactor: make `object_store::path::parts` public
2021-05-07 09:51:44 +02:00
9eb658eac5
feat: s3 compatibility storages support
2021-05-06 23:27:37 +08:00
1f5a9b5ebf
feat: make DirsAndFileName members public
...
Helps with #1253 .
2021-05-03 09:51:00 +02:00
ca5351088a
feat: add Path->DirsAndFileName conversion
...
This allows for easier parsing of received paths.
Helps with #1253 .
2021-05-03 09:51:00 +02:00
140d96dbea
feat: tests ffor loading data to object store and make sure twe still query read buffer
2021-04-28 15:59:17 -04:00
eddc9319ff
docs: deny broken intradoc links
2021-04-27 13:22:28 +02:00
53e83f8385
feat: add throttled store wrapper
2021-04-27 09:31:48 +02:00
d8a92a8570
test: Don't run any integration tests unless TEST_INTEGRATION is set
2021-04-22 14:21:50 -04:00
c7e6a0a797
fix: Use wb rather than wal in object storage test path names for consistency
2021-04-21 17:43:03 +00:00
f136931225
fix: Inconsistent ordering lints
2021-04-19 08:48:11 -04:00
fd0da7e74a
chore: upgrade arrow and Rust
...
See https://github.com/apache/arrow/pull/10082 for upstream PR.
2021-04-19 14:00:04 +02:00
657bfa1b20
refactor: address Andrew's comments
2021-04-16 17:44:46 -04:00
4c23ca8888
feat: full implementation of parquet's read_filter for review
2021-04-16 16:03:24 -04:00
1997324344
feat: mutable buffer snapshotting ( #1179 )
...
* feat: mutable buffer snapshotting
* chore: review feedback
2021-04-13 12:14:54 +00:00
16cc37e7f3
chore: update cloud-storage to 0.9 ( #1165 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-09 14:37:51 +00:00
82588d5c72
fix: Don't return Result from test functions
2021-04-07 12:40:00 -04:00
8bd7a39607
fix: Use InstanceMetadataProvider directly
...
Rusoto's ChainProvider swallows the error message produced by the underlying InstanceMetadataProvider
(see d59d716f09/rusoto/credential/src/lib.rs (L397)
2021-03-15 13:55:43 +00:00
3c8a58266a
feat: Make S3 auth parameters optional
...
so that if not present, the aws client library can use builtin auth providers,
such as the InstanceMetadataProvider, which is commonly used to get the credentials
granted to the AWS VM via cloud native mechanism.
2021-03-15 10:11:01 +01:00
d67a03f616
fix: Improved tests found another unimplemented hiding
2021-03-04 16:18:31 -05:00
ca2f74063e
fix: Test using the object store wrapper interface
2021-03-04 16:18:11 -05:00
16df4c542b
fix: Rename AZURE_STORAGE_MASTER_KEY to AZURE_STORAGE_ACCESS_KEY
2021-03-04 10:15:46 -05:00
e9fedfae17
fix: Use AWS_DEFAULT_REGION instead of AWS_REGION
2021-03-04 10:15:30 -05:00
37746173d9
feat: Change azure object store to only get config from args, not env
2021-03-04 10:14:42 -05:00
02d981451d
feat: Implement Google Cloud Storage-related CLI arguments
2021-03-04 10:14:42 -05:00
ef13c1023e
feat: Change google cloud object store to get config from args, not env
2021-03-04 10:14:42 -05:00
06236e796b
feat: Change aws object store to get config from args, not env
2021-03-04 10:14:42 -05:00
c7ef18337c
feat: Consolidate all bucket config into one option/env var
...
Fixes #869 .
2021-02-25 15:53:20 -05:00
12b768b8f1
fix: Escape empty string PathPart
...
Empty directory names are silently ignored and can lead to very surprising effects
such as directory layouts missing a level. This makes it hard to reason about directory structures.
A sane object store path API should either disallow empty names or deal with them gracefully.
Since we already have to escape file/directory names using the minimum common denominator valid character
set for known cloud providers, it feels quite natural to treat this empty dir/file name problem as encoding problem.
2021-02-24 21:13:56 +00:00
9e521a2ea1
feat: Plug GCS list_with_delimiter impl
...
And fix env.example. Now GCS can be used to persist snapshots.
2021-02-23 21:06:53 +00:00
9860def4b2
feat: Add S3 plumbing to iodx config
2021-02-23 14:55:55 +00:00
cff12da3a1
fix: Upgrade to released version of cloud_storage
...
Fixes #801 .
2021-02-22 13:01:06 -05:00
a42103f436
Merge remote-tracking branch 'origin/main' into cn/google-list-with-delimiter
2021-02-22 12:53:46 -05:00
81739bf486
docs: Fix typo
2021-02-22 10:35:23 +00:00
cc6738c6f3
fix: Check for AZURE_STORAGE_MASTER_KEY in the test macro too
2021-02-18 16:53:06 -05:00
fcd4f91909
feat: Implement list_with_delimiter for Azure storage
2021-02-18 16:37:23 -05:00
57942b51b7
feat: Update to latest Azure sdk to get delimiter support
...
Needed these PRs:
- https://github.com/Azure/azure-sdk-for-rust/pull/176
- https://github.com/Azure/azure-sdk-for-rust/pull/179
Also needed to enable the queue feature to get the azure_storage crate
compiling; at the moment, the code is still being reorganized and the
features aren't independent yet:
https://github.com/Azure/azure-sdk-for-rust/issues/177
2021-02-18 14:59:06 -05:00
484adcc257
chore: fix typo in an error message
2021-02-18 14:57:38 -05:00
536c1724bd
feat: Allow to put streams of unknown length to objectstore
...
Addresses the API aspect of #818
Adds a utility module that helps computing the length of a stream while buffering it
for later replay (in-memory or spilling it in a temporary file).
2021-02-18 16:49:18 +00:00
f934a21efe
test: Update tests to match new cloud storage error behavior
2021-02-17 14:23:34 -05:00
ef54131afb
feat: Gets google cloud list_with_delimiter tests passing
2021-02-17 14:23:33 -05:00
2b642a8da6
refactor: add arc clone lint
2021-02-15 12:38:19 +00:00
dad426d02e
fix: Report a failure to parse an AWS datetime ( #794 )
...
* fix: Report a failure to parse an AWS datetime
* refactor: use SNAFU context selectors instead of enum variants
2021-02-12 15:10:49 +00:00
c7e8a68fbe
fix: enable tokio::fs for object_store crate ( #788 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-11 12:12:57 +00:00
c17feb998a
feat: Implement display on FilePath
2021-02-08 15:13:25 -05:00
076f67285d
Merge branch 'main' into cn+jg/file-delimiter
2021-02-05 09:45:34 -05:00
fbf776c6b3
chore: Clean up Cargo.tomls ( #754 )
...
* fix: test_helpers crate should only be a dev-dep
* fix: object_store no longer has a build script, so no longer needs a build dep
* chore: Alphabetize all Cargo.tomls
2021-02-04 18:56:02 -05:00
8b18003e19
test: Don't check file metadata because SystemTime is not monotonic
...
See https://doc.rust-lang.org/std/time/struct.SystemTime.html
2021-02-04 15:46:11 -05:00
Carol (Nichols || Goulding)
dependabot[bot]
Marco Neumann
Raphael Taylor-Davies
Andrew Lamb
kodiakhq[bot]
Ning Sun
Dom
Ning Sun
Paul Dix
Jake Goulding
DCjanus
Nga Tran
Marko Mikulicic
Edd Robinson
Carol (Nichols || Goulding)