420c306caa
chore(deps): Bump tokio from 1.17.0 to 1.18.0 ( #4453 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.17.0 to 1.18.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.17.0...tokio-1.18.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-28 08:21:17 +00:00
663944a2a9
chore(deps): Bump all rusoto crates from 0.47.0 to 0.48.0 ( #4439 )
...
* chore(deps): Bump rusoto_credential from 0.47.0 to 0.48.0
Bumps [rusoto_credential](https://github.com/rusoto/rusoto ) from 0.47.0 to 0.48.0.
- [Release notes](https://github.com/rusoto/rusoto/releases )
- [Changelog](https://github.com/rusoto/rusoto/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rusoto/rusoto/compare/rusoto-v0.47.0...mock-v0.48.0 )
---
updated-dependencies:
- dependency-name: rusoto_credential
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): Bump rusoto_core from 0.47.0 to 0.48.0
Bumps [rusoto_core](https://github.com/rusoto/rusoto ) from 0.47.0 to 0.48.0.
- [Release notes](https://github.com/rusoto/rusoto/releases )
- [Changelog](https://github.com/rusoto/rusoto/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rusoto/rusoto/compare/rusoto-v0.47.0...mock-v0.48.0 )
---
updated-dependencies:
- dependency-name: rusoto_core
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): Bump rusoto_s3 from 0.47.0 to 0.48.0
Bumps [rusoto_s3](https://github.com/rusoto/rusoto ) from 0.47.0 to 0.48.0.
- [Release notes](https://github.com/rusoto/rusoto/releases )
- [Changelog](https://github.com/rusoto/rusoto/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rusoto/rusoto/compare/rusoto-v0.47.0...mock-v0.48.0 )
---
updated-dependencies:
- dependency-name: rusoto_s3
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-27 14:58:40 +00:00
4b47d723b1
refactor: Rename time to iox_time ( #4416 )
...
Signed-off-by: hi-rustin <rustin.liu@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-26 00:19:59 +00:00
9950bcee27
chore(deps): Bump indexmap from 1.8.0 to 1.8.1 ( #4166 )
...
Bumps [indexmap](https://github.com/bluss/indexmap ) from 1.8.0 to 1.8.1.
- [Release notes](https://github.com/bluss/indexmap/releases )
- [Changelog](https://github.com/bluss/indexmap/blob/master/RELEASES.md )
- [Commits](https://github.com/bluss/indexmap/compare/1.8.0...1.8.1 )
---
updated-dependencies:
- dependency-name: indexmap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-30 08:49:07 +00:00
a373c90415
refactor: Extract the list_all function to object store
...
I'm about to use this in a third file, so time to extract this.
Make it clear that this is appropriate for tests only.
2022-03-29 08:15:24 -04:00
17af5fcbd1
chore(deps): Bump tokio-util from 0.7.0 to 0.7.1 ( #4154 )
...
* chore(deps): Bump tokio-util from 0.7.0 to 0.7.1
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.7.0 to 0.7.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.7.0...tokio-util-0.7.1 )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-29 08:39:02 +00:00
4f9515ffba
chore(deps): Bump async-trait from 0.1.52 to 0.1.53 ( #4141 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.52 to 0.1.53.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.52...0.1.53 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-28 08:55:24 +00:00
5585dd3c21
refactor: switch to using DynObjectStore
...
Changes all consumers of the object store to use the dynamically
dispatched DynObjectStore type, instead of using a hardcoded concrete
implementation type.
2022-03-15 16:32:52 +00:00
b727d26dab
refactor: path_from_dirs_and_filename trait method
...
Moves the path_from_dirs_and_filename from an ObjectStoreImpl method to
a trait method, completing the abstraction over all object store
backends.
2022-03-15 16:29:43 +00:00
1d5066c421
refactor: rename ObjectStore -> ObjectStoreImpl
...
Frees up the name for so we can use `dyn ObjectStore` throughout the
code instead of `ObjectStoreApi`.
2022-03-15 16:29:43 +00:00
7a7dfade94
feat(catalog): instrument list() ops
...
List (but not list_with_delimiter?!) returns a stream of ListResult
which previously wasn't instrumented - this commit uses the
StreamMetricRecorder to record the wall clock duration of the entire
list operation.
2022-03-15 14:23:11 +00:00
ee01225fec
refactor: polymorphic stream recorder impl
...
Changes the StreamMetricRecorder to be generic over the Ok types in the
result stream, invoking a T-specific delegate when Ok(T) is observed.
This enables the stream instrumentation to be reused across different
stream types while keeping the hairy state checks DRY.
This will allow the StreamMetricRecorder to decorate the streams
returned by both the get() and list() operations, but this commit causes
no functional change.
2022-03-15 14:13:17 +00:00
1aaf193299
refactor: take last_yielded_at
2022-03-14 11:54:02 +00:00
3802401b4f
docs: explicit stream duration semantics
2022-03-14 11:54:02 +00:00
1aee8402bf
refactor: suffix duration vars with _ms
2022-03-14 11:54:02 +00:00
f01382f578
docs: fix typos
...
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2022-03-14 11:36:49 +00:00
dc8a708fb7
refactor: use DummyObjectStore for error tests
2022-03-11 14:36:34 +00:00
bfc473b8cd
feat: object store instrumentation
...
Implements a decorator of the ObjectStoreAPI trait, recording:
* Bytes uploaded / downloaded through the instrumented API
* Call latencies, broken down by operation & success / error state
All the current implementations that return a Stream from the get
operation actually return a "fake" stream containing all the data in one
go rather than streaming chunks from the upstream. I've instrumented the
Stream to future-proof any actual streaming impls in the future.
2022-03-11 13:38:48 +00:00
b870b9340b
chore: remove uneeded dependencies ( #3929 )
...
* chore: remove unused deps in compactor
* chore: remove unused deps in influxdb_ioxd
* chore: remove unused deps in object_store
* chore: remove unused deps in server
* fix: object_store needs observability deps when compiled with aws
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-04 17:39:46 +00:00
cf0c238ae4
feat: allow unencrypted HTTP connections to AWS via flag ( #3916 )
...
This is required for the test bench.
2022-03-03 17:01:03 +00:00
46bb107be4
refactor: use rustls
...
Removes openssl as a dependency, switching to rustls[1] as the TLS
implementation throughout.
It is important to note that this change brings with it a significant
behavioural difference - rustls does not currently support IP SANs in
certificates (instead only supporting fully-qualified names / DNS) and
this will manifest as a failure to connect to IP endpoints over TLS.
This might be a blocker that prevents us using rustls exclusively, but
there's noe asy way to know without trying it. Fortunately the rustls
project has received funding to work on IP SAN support[2].
[1]: https://github.com/rustls/rustls
[2]: https://www.abetterinternet.org/post/preparing-rustls-for-wider-adoption/
2022-03-03 11:05:20 +00:00
4785a7d028
refactor: bump Azure SDK
...
Update to the latest Azure SDK to pick up rustls support.
2022-03-03 11:02:20 +00:00
ad3868ed7c
chore(deps): Bump tokio from 1.16.1 to 1.17.0 ( #3814 )
...
* chore(deps): Bump tokio from 1.16.1 to 1.17.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.16.1 to 1.17.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.16.1...tokio-1.17.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build: update workspace-hack
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:27:43 +00:00
89105ccfab
chore(deps): Bump tokio-util from 0.6.9 to 0.7.0 ( #3743 )
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.6.9 to 0.7.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/commits )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-15 11:33:41 +00:00
85d44552fa
chore(deps): bump cloud-storage from 0.10.3 to 0.11.0 ( #3697 )
...
Bumps [cloud-storage](https://github.com/ThouCheese/cloud-storage-rs ) from 0.10.3 to 0.11.0.
- [Release notes](https://github.com/ThouCheese/cloud-storage-rs/releases )
- [Changelog](https://github.com/ThouCheese/cloud-storage-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ThouCheese/cloud-storage-rs/commits )
---
updated-dependencies:
- dependency-name: cloud-storage
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-02-09 20:03:44 +00:00
f3f6f335a9
chore: upgrade to snafu 0.7 ( #3440 )
2022-01-11 19:22:36 +00:00
099d9dc6e1
Merge branch 'main' into crepererum/issue3226
2021-12-17 16:37:16 +00:00
24fd2e549b
docs: Fix some typos and outdated comments
2021-12-17 11:33:36 -05:00
9f2694bf1b
test: `objest_store` azure support via Azurite
2021-12-17 09:30:21 +01:00
61ee70d222
refactor: make object store prefixes segment based
...
Don't use string prefixes, e.g. `foo/bar/` is a prefix of `foo/bar/x`
but NOT of `foo/bar_baz/y`.
This also removes some heuristics during the cloud storage parsing that
assumed that file names always contain a dot but directories don't.
Technically we should now always be able to know whether a path points
to a file or a directory:
- Rust (manually constructed): we use `DirsAndFileName` which knows the
difference (i.e. if `file_name` is set)
- in-mem store: we also use `DirsAndFileName`
- file system: this was fixed by #1523
(ccd094dfcf464667d8b8Fixes #3226 .
2021-12-16 14:40:45 +01:00
f02d846389
test: `object_store` aws test using localstack
...
This removes 3 "nonexisting region" tests that where testing very
specific error behavior that no local emulator (minio and localstack)
replicate and that don't add much value. It's better to test our AWS
code at all than being to picky.
2021-12-16 10:10:06 +01:00
876a9af35a
fix: limit number of S3 connections
...
Otherwise the whole thing blows up when starting a server that has many
DBs registerd, because we potentially create 1 connection per DB (e.g.
to read out the preserved catalog).
Fixes #3336 .
2021-12-08 19:06:02 +01:00
02c297e850
fix: Always specify the parking_lot feature of tokio to get potential perf boost
2021-12-06 09:37:15 -05:00
39a7077cd9
fix: Turn on std for indexmap to avoid a build-time target check
...
Connects to #3117 .
2021-12-06 09:37:14 -05:00
7a241882c2
fix: Make version reqs on futures-related crates consistent
2021-12-06 09:37:14 -05:00
39862e3886
fix: Disable default chrono features
...
Connects to #3117 . Won't affect the workspace-hack crate until changes
to some other crates can be upstreamed.
2021-12-06 09:36:49 -05:00
bca561366b
feat: don't copy parquet files out of disk object store ( #3282 ) ( #3293 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-05 16:31:40 +00:00
5d0fd1c603
fix: Allow dead code on fields that are now detected as never read
2021-12-02 11:52:01 -05:00
9fd4a560f5
feat: Results of running cargo hakari manage-deps
2021-11-19 09:21:57 -05:00
f0136e9791
ci: run clippy for all features
...
This indeed fixes some issues in our object store implementation.
Closes #537 .
2021-11-11 10:48:21 +01:00
c540b40f05
chore(deps): bump tokio from 1.12.0 to 1.13.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.12.0...tokio-1.13.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 11:21:59 +00:00
a1f978e280
chore(deps): bump tokio-util from 0.6.8 to 0.6.9
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.6.8 to 0.6.9.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.6.8...tokio-util-0.6.9 )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 01:28:41 +00:00
eeb017990d
fix: Add missing features to object_store crate to enable running tests separately
2021-10-28 13:20:28 -04:00
04b4eeb6ec
Merge branch 'main' into cn/only-read-server
2021-10-25 15:15:17 +00:00
bc7244c48e
chore: use Rust edition 2021
2021-10-25 10:58:20 +02:00
4de7dce8ba
feat: Add a 404 NotFound error to the object store errors for GET
...
With the object store implementations that support figuring out if a GET
request returns 404 not found. Azure doesn't have great support for
this.
2021-10-22 15:55:59 -04:00
6ae42bb774
fix: Put both Debug and Display of AWS errors in error messages
2021-10-22 13:59:48 -04:00
a2b81cdb20
fix: Put Debug format of rusoto errors in error messages
...
Fixes #2942
2021-10-22 13:16:37 -04:00
f7ca80e29f
test: ensure query cancellation (somewhat) works ( #2931 )
...
* feat: enable reconfiguration of in-use throttled store
This is handy for tests for which a part should run "normal" and another
one should be throttled/blocked.
* feat: keep track of the number of tasks within a `DedicatedExecutor`
* test: ensure query cancellation (somewhat) works
We cannot really test that query cancellation finishes all subtasks
because _tokio_ doesn't provide sufficient stats / inspection, at least
as long we don't want to rely heavily on _tokio_ tracing. So let's at
least check that tasks from the dedicated executors are pruned properly.
For all other regressions we need to add unit tests to the affected
components. See for example:
- https://github.com/apache/arrow-datafusion/issues/1103
- https://github.com/apache/arrow-datafusion/pull/1105
- https://github.com/apache/arrow-datafusion/pull/1112
- https://github.com/apache/arrow-datafusion/pull/1121
Closes #2027 .
2021-10-21 19:10:58 +00:00
4cc87fb184
chore: misc updates ( #2886 )
...
* chore: misc updates
* fix: don't downgrade nix
* chore: remove orphaned comment
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-19 11:47:27 +00:00
dependabot[bot]
Carol (Nichols || Goulding)
二手掉包工程师
Carol (Nichols || Goulding)
Dom Dwyer
Andrew Lamb
Marco Neumann
kodiakhq[bot]
Raphael Taylor-Davies
Carol (Nichols || Goulding)