ae6b0e669b
refactor: Extract a database persister type that wraps object store
...
Connects to #2193 .
2021-08-12 15:05:32 -04:00
3de6b44e23
build: use new rustdoc lint name ( #2261 )
...
* fix: nocache feature code rot
The MBChunk::snapshot code when using the "nocache" option no longer
compiles - this commit updates it to match the not(nocache) code.
* build: use updated broken_intra_doc_links name
The broken_intra_doc_links lint was renamed
rustdoc::broken_intra_doc_links
https://doc.rust-lang.org/rustdoc/lints.html
2021-08-11 19:48:51 +00:00
4c4bb1a93e
feat: add fmt::Display implementation for path data structs ( #2223 )
...
Signed-off-by: Ning Sun <sunng@protonmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-09 18:09:09 +00:00
9d15798288
fix: Address or allow Clippy warnings new with Rust 1.54
2021-07-30 09:59:59 -04:00
d95b5df03e
refactor: move cache to ObjectStore
...
Since the consumers of ObjectStore always use the concrete type rather than the ObjectStoreApi trait, it makes more sense to just change the concrete type to have a pointer to the cache. This removes the cache from the ObjectStoreApi trait and changes the ObjectStore to be a regular struct rather than a tuple around the ObjectStoreIntegration. Future work will have the server configure the cache on the ObjectStore struct when its options are set.
2021-07-21 18:27:56 -04:00
47044d537c
chore: make cache a type in object store trait
2021-07-21 18:27:56 -04:00
d0ea812041
feat: add skeleton for object store file cache
2021-07-21 18:27:56 -04:00
939d15a21f
perf: Avoid clone when an error doesn't occur
2021-07-16 09:59:54 -04:00
d35b74c226
fix: Fix doc build warnings ( #1945 )
...
* fix: Fix doc build warnings
* refactor: add deny bare_urls to crates
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-13 08:03:42 +00:00
670826daf9
refactor: make object_store construction interface consistent ( #1944 )
...
* refactor: make object_store construction interface consistent
* fix: benchmarks
* fix: doc build
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-12 12:56:36 +00:00
1a79bf7e99
refactor: Make aws/azure/gcs optional features and stop compiling 100 dependencies during dev ( #1933 )
...
* feat: make aws, gcp, azure dependencies optional
* fix: only run object store tests if the features are enabled
* fix: clean up testing
* fix: rename step
* fix: add to list of jobs
* fix: remove test with object store
* fix: review comments
2021-07-09 11:38:30 +00:00
3cb8f297b1
refactor: encapsulate the ObjectStore implementations in the object store crate ( #1932 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-09 10:38:32 +00:00
24b249ad7b
fix: do not silence all IO errors in disk-based LIST
2021-06-22 18:20:36 +02:00
ec43a87909
chore: Update itertools deps ( #1750 )
2021-06-17 17:56:44 +00:00
0e2a291683
chore: enforce `clippy::future_not_send` for `object_store`
2021-06-09 17:56:43 +02:00
bd39954d73
fix: do not error on `list_with_delimiter` w/ non-existing prefix
...
Disk-driven object stores were reporting `UnableToProcessEntry` when
calling `list_with_delimiter` and a non-existing prefix.
To prevent regressions add a test to the test suite shared by all
store implementations.
2021-06-03 10:25:20 +02:00
1cbe4031d3
feat: allow passing a session token to AWS stores
2021-05-27 11:53:12 +02:00
d5df7c381a
test: accept optional `AWS_ENDPOINT` for OS tests
2021-05-26 18:14:58 +02:00
464667d8b8
docs: clarify the FilePath directory handling
2021-05-20 14:20:56 +02:00
ccd094dfcf
refactor: streamline directory handling in path parsing
2021-05-20 10:22:20 +02:00
ff004c1a51
feat: add `ObjectStore.path_from_dirs_and_filename`
2021-05-17 11:33:22 +02:00
4d105713a5
feat: add convenience `parsed_path!` marco
2021-05-07 09:51:44 +02:00
fadf0176d4
feat: impl Hash for DirsAndFileName and PathParts
2021-05-07 09:51:44 +02:00
d726cf4829
refactor: make `object_store::path::parts` public
2021-05-07 09:51:44 +02:00
9eb658eac5
feat: s3 compatibility storages support
2021-05-06 23:27:37 +08:00
1f5a9b5ebf
feat: make DirsAndFileName members public
...
Helps with #1253 .
2021-05-03 09:51:00 +02:00
ca5351088a
feat: add Path->DirsAndFileName conversion
...
This allows for easier parsing of received paths.
Helps with #1253 .
2021-05-03 09:51:00 +02:00
140d96dbea
feat: tests ffor loading data to object store and make sure twe still query read buffer
2021-04-28 15:59:17 -04:00
eddc9319ff
docs: deny broken intradoc links
2021-04-27 13:22:28 +02:00
53e83f8385
feat: add throttled store wrapper
2021-04-27 09:31:48 +02:00
d8a92a8570
test: Don't run any integration tests unless TEST_INTEGRATION is set
2021-04-22 14:21:50 -04:00
c7e6a0a797
fix: Use wb rather than wal in object storage test path names for consistency
2021-04-21 17:43:03 +00:00
f136931225
fix: Inconsistent ordering lints
2021-04-19 08:48:11 -04:00
fd0da7e74a
chore: upgrade arrow and Rust
...
See https://github.com/apache/arrow/pull/10082 for upstream PR.
2021-04-19 14:00:04 +02:00
657bfa1b20
refactor: address Andrew's comments
2021-04-16 17:44:46 -04:00
4c23ca8888
feat: full implementation of parquet's read_filter for review
2021-04-16 16:03:24 -04:00
1997324344
feat: mutable buffer snapshotting ( #1179 )
...
* feat: mutable buffer snapshotting
* chore: review feedback
2021-04-13 12:14:54 +00:00
16cc37e7f3
chore: update cloud-storage to 0.9 ( #1165 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-09 14:37:51 +00:00
82588d5c72
fix: Don't return Result from test functions
2021-04-07 12:40:00 -04:00
8bd7a39607
fix: Use InstanceMetadataProvider directly
...
Rusoto's ChainProvider swallows the error message produced by the underlying InstanceMetadataProvider
(see d59d716f09/rusoto/credential/src/lib.rs (L397)
2021-03-15 13:55:43 +00:00
3c8a58266a
feat: Make S3 auth parameters optional
...
so that if not present, the aws client library can use builtin auth providers,
such as the InstanceMetadataProvider, which is commonly used to get the credentials
granted to the AWS VM via cloud native mechanism.
2021-03-15 10:11:01 +01:00
d67a03f616
fix: Improved tests found another unimplemented hiding
2021-03-04 16:18:31 -05:00
ca2f74063e
fix: Test using the object store wrapper interface
2021-03-04 16:18:11 -05:00
16df4c542b
fix: Rename AZURE_STORAGE_MASTER_KEY to AZURE_STORAGE_ACCESS_KEY
2021-03-04 10:15:46 -05:00
e9fedfae17
fix: Use AWS_DEFAULT_REGION instead of AWS_REGION
2021-03-04 10:15:30 -05:00
37746173d9
feat: Change azure object store to only get config from args, not env
2021-03-04 10:14:42 -05:00
02d981451d
feat: Implement Google Cloud Storage-related CLI arguments
2021-03-04 10:14:42 -05:00
ef13c1023e
feat: Change google cloud object store to get config from args, not env
2021-03-04 10:14:42 -05:00
06236e796b
feat: Change aws object store to get config from args, not env
2021-03-04 10:14:42 -05:00
c7ef18337c
feat: Consolidate all bucket config into one option/env var
...
Fixes #869 .
2021-02-25 15:53:20 -05:00
Carol (Nichols || Goulding)
Dom
Ning Sun
Paul Dix
Jake Goulding
Andrew Lamb
Marco Neumann
DCjanus
Nga Tran
Raphael Taylor-Davies
Marko Mikulicic