44d7479db1
chore(deps): bump cloud-storage from 0.10.2 to 0.10.3
...
Bumps [cloud-storage](https://github.com/ThouCheese/cloud-storage-rs ) from 0.10.2 to 0.10.3.
- [Release notes](https://github.com/ThouCheese/cloud-storage-rs/releases )
- [Changelog](https://github.com/ThouCheese/cloud-storage-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ThouCheese/cloud-storage-rs/commits )
---
updated-dependencies:
- dependency-name: cloud-storage
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-04 01:43:09 +00:00
fb0e6ad5b1
docs: Document the s3_request generics and how they fit together
2021-09-29 08:19:33 -04:00
2b9612a033
fix: Remove some of the futures in futures layers
...
I think the FuturesRetry crate needed this, but we're no longer using
the crate so we can get rid of some of the complexity.
2021-09-29 08:19:33 -04:00
ad1405dcdf
fix: Reduce number of AWS S3 request retries from 10 to 3
2021-09-29 08:19:32 -04:00
5857edb19a
fix: cloud_storage crate returns a different error type in this case
2021-09-29 08:19:32 -04:00
92583aee82
fix: Remove streaming API since we're not streaming anyway
2021-09-29 08:19:32 -04:00
edd6c12e93
fix: Add logging when retrying AWS requests
2021-09-29 08:19:32 -04:00
ea4cb9a6c1
refactor: Extract the maximum number of retries into a documented public constant
2021-09-29 08:19:32 -04:00
9cf343db08
refactor: Use loop instead of futures_retry
2021-09-29 08:19:32 -04:00
9cdccae49d
fix: Actually do the exponential part of exponential backoff
2021-09-29 08:19:32 -04:00
d69472e114
refactor: Use s3_request in s3 delete
2021-09-29 08:19:32 -04:00
d05528bcfd
refactor: Use s3_request for put requests
...
Which meant we also needed to change the byte stream to be a closure
that can generate a byte stream
2021-09-29 08:19:32 -04:00
dd3b405727
fix: Only retry s3 requests if error is 500
2021-09-29 08:19:32 -04:00
936fba24b9
refactor: Make s3_request generic over the return type
2021-09-29 08:19:32 -04:00
5c78b7d5ae
refactor: Extract an s3_request method that handles retries
2021-09-29 08:19:32 -04:00
fd3a027ca8
refactor: Use the extracted aws list function in list_with_delimiter
2021-09-29 08:19:32 -04:00
a1a8bc01bf
refactor: Extract a method just to stream aws list requests
2021-09-29 08:19:32 -04:00
9cdd4799c7
chore: `object_store` needs `reqwest` only for azure
2021-09-22 11:16:00 +02:00
ef5ab67c77
chore: upgrade azure dependencies
2021-09-21 13:35:11 +02:00
4382bc0b71
feat: disable Snafu futures feature ( #2573 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-19 11:36:24 +00:00
b67610d9b9
chore(deps): bump tokio from 1.10.1 to 1.11.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.10.1 to 1.11.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.10.1...tokio-1.11.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 09:11:38 +00:00
fbf18813ac
fix: Correct a mistaken assumption in in-memory object storage
...
The implementation of list_with_delimiter for the in-memory object
storage assumed that paths returned from the BTreeMap keys that sorted
greater than the prefix given to list_with_delimiter and for whom
prefix_matches returned true would also have parts after the prefix.
This didn't account for paths that started with the prefix but didn't
immediately have the delimiter after the prefix: that is,
prefix = 1/database_name
would match the in-memory paths:
1/database_name/0/rules.pb
1/database_name_and_another_thing/0/rules.pb
The first path here *would* return some parts_after_prefix, but the
second path would not and the previously existing code would panic for
the added path in the list_with_delimiter test case.
2021-09-02 14:57:27 -04:00
37c81fb483
chore(deps): bump cloud-storage from 0.9.0 to 0.10.2
...
Bumps [cloud-storage](https://github.com/ThouCheese/cloud-storage-rs ) from 0.9.0 to 0.10.2.
- [Release notes](https://github.com/ThouCheese/cloud-storage-rs/releases )
- [Changelog](https://github.com/ThouCheese/cloud-storage-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ThouCheese/cloud-storage-rs/commits )
---
updated-dependencies:
- dependency-name: cloud-storage
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-23 11:26:28 +00:00
063f6da7d6
docs: remove outdated comment ( #2302 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-17 12:57:07 +00:00
44f9f220cd
Merge branch 'main' into dependabot/cargo/rusoto_credential-0.47.0
2021-08-17 11:38:54 +00:00
bda4458ebd
fix: Update all the rusoto crates together
2021-08-16 19:59:26 -04:00
c012e996ab
refactor: remove display methods, use fmt::Display instead. ( #2272 )
...
* refactor: remove display methods, use fmt::Display instead.
Signed-off-by: Ning Sun <sunng@protonmail.com>
* refactor: update a few calls from .display to .to_string()
* fix: consistently use `Path` rather than occasionally `DirsAndFileName`
* fix: fixup for merge conflicts
* fix: update test
* fix: Catch another case or two
* fix: fmt
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-16 18:00:22 +00:00
97ed42c42f
build(deps): bump rusoto_credential from 0.46.0 to 0.47.0
...
Bumps [rusoto_credential](https://github.com/rusoto/rusoto ) from 0.46.0 to 0.47.0.
- [Release notes](https://github.com/rusoto/rusoto/releases )
- [Changelog](https://github.com/rusoto/rusoto/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rusoto/rusoto/compare/mock-v0.46.0...rusoto-v0.47.0 )
---
updated-dependencies:
- dependency-name: rusoto_credential
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-16 16:35:09 +00:00
ae6b0e669b
refactor: Extract a database persister type that wraps object store
...
Connects to #2193 .
2021-08-12 15:05:32 -04:00
3de6b44e23
build: use new rustdoc lint name ( #2261 )
...
* fix: nocache feature code rot
The MBChunk::snapshot code when using the "nocache" option no longer
compiles - this commit updates it to match the not(nocache) code.
* build: use updated broken_intra_doc_links name
The broken_intra_doc_links lint was renamed
rustdoc::broken_intra_doc_links
https://doc.rust-lang.org/rustdoc/lints.html
2021-08-11 19:48:51 +00:00
4c4bb1a93e
feat: add fmt::Display implementation for path data structs ( #2223 )
...
Signed-off-by: Ning Sun <sunng@protonmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-09 18:09:09 +00:00
9d15798288
fix: Address or allow Clippy warnings new with Rust 1.54
2021-07-30 09:59:59 -04:00
d95b5df03e
refactor: move cache to ObjectStore
...
Since the consumers of ObjectStore always use the concrete type rather than the ObjectStoreApi trait, it makes more sense to just change the concrete type to have a pointer to the cache. This removes the cache from the ObjectStoreApi trait and changes the ObjectStore to be a regular struct rather than a tuple around the ObjectStoreIntegration. Future work will have the server configure the cache on the ObjectStore struct when its options are set.
2021-07-21 18:27:56 -04:00
47044d537c
chore: make cache a type in object store trait
2021-07-21 18:27:56 -04:00
d0ea812041
feat: add skeleton for object store file cache
2021-07-21 18:27:56 -04:00
939d15a21f
perf: Avoid clone when an error doesn't occur
2021-07-16 09:59:54 -04:00
d35b74c226
fix: Fix doc build warnings ( #1945 )
...
* fix: Fix doc build warnings
* refactor: add deny bare_urls to crates
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-13 08:03:42 +00:00
670826daf9
refactor: make object_store construction interface consistent ( #1944 )
...
* refactor: make object_store construction interface consistent
* fix: benchmarks
* fix: doc build
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-12 12:56:36 +00:00
1a79bf7e99
refactor: Make aws/azure/gcs optional features and stop compiling 100 dependencies during dev ( #1933 )
...
* feat: make aws, gcp, azure dependencies optional
* fix: only run object store tests if the features are enabled
* fix: clean up testing
* fix: rename step
* fix: add to list of jobs
* fix: remove test with object store
* fix: review comments
2021-07-09 11:38:30 +00:00
3cb8f297b1
refactor: encapsulate the ObjectStore implementations in the object store crate ( #1932 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-09 10:38:32 +00:00
24b249ad7b
fix: do not silence all IO errors in disk-based LIST
2021-06-22 18:20:36 +02:00
ec43a87909
chore: Update itertools deps ( #1750 )
2021-06-17 17:56:44 +00:00
0e2a291683
chore: enforce `clippy::future_not_send` for `object_store`
2021-06-09 17:56:43 +02:00
bd39954d73
fix: do not error on `list_with_delimiter` w/ non-existing prefix
...
Disk-driven object stores were reporting `UnableToProcessEntry` when
calling `list_with_delimiter` and a non-existing prefix.
To prevent regressions add a test to the test suite shared by all
store implementations.
2021-06-03 10:25:20 +02:00
1cbe4031d3
feat: allow passing a session token to AWS stores
2021-05-27 11:53:12 +02:00
d5df7c381a
test: accept optional `AWS_ENDPOINT` for OS tests
2021-05-26 18:14:58 +02:00
464667d8b8
docs: clarify the FilePath directory handling
2021-05-20 14:20:56 +02:00
ccd094dfcf
refactor: streamline directory handling in path parsing
2021-05-20 10:22:20 +02:00
ff004c1a51
feat: add `ObjectStore.path_from_dirs_and_filename`
2021-05-17 11:33:22 +02:00
4d105713a5
feat: add convenience `parsed_path!` marco
2021-05-07 09:51:44 +02:00
dependabot[bot]
Carol (Nichols || Goulding)
Marco Neumann
Raphael Taylor-Davies
Andrew Lamb
kodiakhq[bot]
Ning Sun
Dom
Ning Sun
Paul Dix
Jake Goulding