b870b9340b
chore: remove uneeded dependencies ( #3929 )
...
* chore: remove unused deps in compactor
* chore: remove unused deps in influxdb_ioxd
* chore: remove unused deps in object_store
* chore: remove unused deps in server
* fix: object_store needs observability deps when compiled with aws
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-04 17:39:46 +00:00
cf0c238ae4
feat: allow unencrypted HTTP connections to AWS via flag ( #3916 )
...
This is required for the test bench.
2022-03-03 17:01:03 +00:00
46bb107be4
refactor: use rustls
...
Removes openssl as a dependency, switching to rustls[1] as the TLS
implementation throughout.
It is important to note that this change brings with it a significant
behavioural difference - rustls does not currently support IP SANs in
certificates (instead only supporting fully-qualified names / DNS) and
this will manifest as a failure to connect to IP endpoints over TLS.
This might be a blocker that prevents us using rustls exclusively, but
there's noe asy way to know without trying it. Fortunately the rustls
project has received funding to work on IP SAN support[2].
[1]: https://github.com/rustls/rustls
[2]: https://www.abetterinternet.org/post/preparing-rustls-for-wider-adoption/
2022-03-03 11:05:20 +00:00
4785a7d028
refactor: bump Azure SDK
...
Update to the latest Azure SDK to pick up rustls support.
2022-03-03 11:02:20 +00:00
ad3868ed7c
chore(deps): Bump tokio from 1.16.1 to 1.17.0 ( #3814 )
...
* chore(deps): Bump tokio from 1.16.1 to 1.17.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.16.1 to 1.17.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.16.1...tokio-1.17.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build: update workspace-hack
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:27:43 +00:00
89105ccfab
chore(deps): Bump tokio-util from 0.6.9 to 0.7.0 ( #3743 )
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.6.9 to 0.7.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/commits )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-15 11:33:41 +00:00
85d44552fa
chore(deps): bump cloud-storage from 0.10.3 to 0.11.0 ( #3697 )
...
Bumps [cloud-storage](https://github.com/ThouCheese/cloud-storage-rs ) from 0.10.3 to 0.11.0.
- [Release notes](https://github.com/ThouCheese/cloud-storage-rs/releases )
- [Changelog](https://github.com/ThouCheese/cloud-storage-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ThouCheese/cloud-storage-rs/commits )
---
updated-dependencies:
- dependency-name: cloud-storage
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-02-09 20:03:44 +00:00
f3f6f335a9
chore: upgrade to snafu 0.7 ( #3440 )
2022-01-11 19:22:36 +00:00
099d9dc6e1
Merge branch 'main' into crepererum/issue3226
2021-12-17 16:37:16 +00:00
24fd2e549b
docs: Fix some typos and outdated comments
2021-12-17 11:33:36 -05:00
9f2694bf1b
test: `objest_store` azure support via Azurite
2021-12-17 09:30:21 +01:00
61ee70d222
refactor: make object store prefixes segment based
...
Don't use string prefixes, e.g. `foo/bar/` is a prefix of `foo/bar/x`
but NOT of `foo/bar_baz/y`.
This also removes some heuristics during the cloud storage parsing that
assumed that file names always contain a dot but directories don't.
Technically we should now always be able to know whether a path points
to a file or a directory:
- Rust (manually constructed): we use `DirsAndFileName` which knows the
difference (i.e. if `file_name` is set)
- in-mem store: we also use `DirsAndFileName`
- file system: this was fixed by #1523
(ccd094dfcf464667d8b8Fixes #3226 .
2021-12-16 14:40:45 +01:00
f02d846389
test: `object_store` aws test using localstack
...
This removes 3 "nonexisting region" tests that where testing very
specific error behavior that no local emulator (minio and localstack)
replicate and that don't add much value. It's better to test our AWS
code at all than being to picky.
2021-12-16 10:10:06 +01:00
876a9af35a
fix: limit number of S3 connections
...
Otherwise the whole thing blows up when starting a server that has many
DBs registerd, because we potentially create 1 connection per DB (e.g.
to read out the preserved catalog).
Fixes #3336 .
2021-12-08 19:06:02 +01:00
02c297e850
fix: Always specify the parking_lot feature of tokio to get potential perf boost
2021-12-06 09:37:15 -05:00
39a7077cd9
fix: Turn on std for indexmap to avoid a build-time target check
...
Connects to #3117 .
2021-12-06 09:37:14 -05:00
7a241882c2
fix: Make version reqs on futures-related crates consistent
2021-12-06 09:37:14 -05:00
39862e3886
fix: Disable default chrono features
...
Connects to #3117 . Won't affect the workspace-hack crate until changes
to some other crates can be upstreamed.
2021-12-06 09:36:49 -05:00
bca561366b
feat: don't copy parquet files out of disk object store ( #3282 ) ( #3293 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-05 16:31:40 +00:00
5d0fd1c603
fix: Allow dead code on fields that are now detected as never read
2021-12-02 11:52:01 -05:00
9fd4a560f5
feat: Results of running cargo hakari manage-deps
2021-11-19 09:21:57 -05:00
f0136e9791
ci: run clippy for all features
...
This indeed fixes some issues in our object store implementation.
Closes #537 .
2021-11-11 10:48:21 +01:00
c540b40f05
chore(deps): bump tokio from 1.12.0 to 1.13.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.12.0...tokio-1.13.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 11:21:59 +00:00
a1f978e280
chore(deps): bump tokio-util from 0.6.8 to 0.6.9
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.6.8 to 0.6.9.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.6.8...tokio-util-0.6.9 )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 01:28:41 +00:00
eeb017990d
fix: Add missing features to object_store crate to enable running tests separately
2021-10-28 13:20:28 -04:00
04b4eeb6ec
Merge branch 'main' into cn/only-read-server
2021-10-25 15:15:17 +00:00
bc7244c48e
chore: use Rust edition 2021
2021-10-25 10:58:20 +02:00
4de7dce8ba
feat: Add a 404 NotFound error to the object store errors for GET
...
With the object store implementations that support figuring out if a GET
request returns 404 not found. Azure doesn't have great support for
this.
2021-10-22 15:55:59 -04:00
6ae42bb774
fix: Put both Debug and Display of AWS errors in error messages
2021-10-22 13:59:48 -04:00
a2b81cdb20
fix: Put Debug format of rusoto errors in error messages
...
Fixes #2942
2021-10-22 13:16:37 -04:00
f7ca80e29f
test: ensure query cancellation (somewhat) works ( #2931 )
...
* feat: enable reconfiguration of in-use throttled store
This is handy for tests for which a part should run "normal" and another
one should be throttled/blocked.
* feat: keep track of the number of tasks within a `DedicatedExecutor`
* test: ensure query cancellation (somewhat) works
We cannot really test that query cancellation finishes all subtasks
because _tokio_ doesn't provide sufficient stats / inspection, at least
as long we don't want to rely heavily on _tokio_ tracing. So let's at
least check that tasks from the dedicated executors are pruned properly.
For all other regressions we need to add unit tests to the affected
components. See for example:
- https://github.com/apache/arrow-datafusion/issues/1103
- https://github.com/apache/arrow-datafusion/pull/1105
- https://github.com/apache/arrow-datafusion/pull/1112
- https://github.com/apache/arrow-datafusion/pull/1121
Closes #2027 .
2021-10-21 19:10:58 +00:00
4cc87fb184
chore: misc updates ( #2886 )
...
* chore: misc updates
* fix: don't downgrade nix
* chore: remove orphaned comment
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-19 11:47:27 +00:00
1dda568d28
test: Add a case for a path without a filename
2021-10-18 08:46:00 -04:00
d5ab29711e
fix: Serialize relative db object store paths to the server config
...
So that they can be deserialized, without parsing, to create a new
iox object store from the location listed in the server config.
Notably, the locations serialized don't start with the object storage's
prefix like "s3:" or "file:". The location is the same object storage as
the server configuration that was just read from object storage. Having
the server config on one type of object storage and the database files
on another type is not supported.
2021-10-18 08:37:36 -04:00
44d7479db1
chore(deps): bump cloud-storage from 0.10.2 to 0.10.3
...
Bumps [cloud-storage](https://github.com/ThouCheese/cloud-storage-rs ) from 0.10.2 to 0.10.3.
- [Release notes](https://github.com/ThouCheese/cloud-storage-rs/releases )
- [Changelog](https://github.com/ThouCheese/cloud-storage-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ThouCheese/cloud-storage-rs/commits )
---
updated-dependencies:
- dependency-name: cloud-storage
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-04 01:43:09 +00:00
fb0e6ad5b1
docs: Document the s3_request generics and how they fit together
2021-09-29 08:19:33 -04:00
2b9612a033
fix: Remove some of the futures in futures layers
...
I think the FuturesRetry crate needed this, but we're no longer using
the crate so we can get rid of some of the complexity.
2021-09-29 08:19:33 -04:00
ad1405dcdf
fix: Reduce number of AWS S3 request retries from 10 to 3
2021-09-29 08:19:32 -04:00
5857edb19a
fix: cloud_storage crate returns a different error type in this case
2021-09-29 08:19:32 -04:00
92583aee82
fix: Remove streaming API since we're not streaming anyway
2021-09-29 08:19:32 -04:00
edd6c12e93
fix: Add logging when retrying AWS requests
2021-09-29 08:19:32 -04:00
ea4cb9a6c1
refactor: Extract the maximum number of retries into a documented public constant
2021-09-29 08:19:32 -04:00
9cf343db08
refactor: Use loop instead of futures_retry
2021-09-29 08:19:32 -04:00
9cdccae49d
fix: Actually do the exponential part of exponential backoff
2021-09-29 08:19:32 -04:00
d69472e114
refactor: Use s3_request in s3 delete
2021-09-29 08:19:32 -04:00
d05528bcfd
refactor: Use s3_request for put requests
...
Which meant we also needed to change the byte stream to be a closure
that can generate a byte stream
2021-09-29 08:19:32 -04:00
dd3b405727
fix: Only retry s3 requests if error is 500
2021-09-29 08:19:32 -04:00
936fba24b9
refactor: Make s3_request generic over the return type
2021-09-29 08:19:32 -04:00
5c78b7d5ae
refactor: Extract an s3_request method that handles retries
2021-09-29 08:19:32 -04:00
fd3a027ca8
refactor: Use the extracted aws list function in list_with_delimiter
2021-09-29 08:19:32 -04:00
Andrew Lamb
Marco Neumann
Dom Dwyer
dependabot[bot]
kodiakhq[bot]
Carol (Nichols || Goulding)
Raphael Taylor-Davies
Carol (Nichols || Goulding)