613cec4513
chore: remove azure git pin ( #4564 )
...
* chore: remove azure git pin
* chore: fix hakari
* chore: Run cargo hakari tasks
* chore: remove outdated comment
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-11 10:21:17 +00:00
1c3a3f25e6
feat: make ObjectStore::list return stream of ObjectMeta ( #4549 )
...
* feat: make ObjectStore::list return stream of ObjectMeta
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-10 16:39:25 +00:00
9befcd5b3d
chore: rename INFLUXDB_IOX_BUCKET to OBJECT_STORE_BUCKET in object_store ( #4548 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-10 16:31:46 +00:00
99b1a9b83f
refactor: split out ObjectStoreMetrics ( #4547 )
...
* refactor: split out ObjectStoreMetrics
* chore: add workspace hack
* fix: compile
2022-05-10 10:56:28 +00:00
8b379c83cc
refactor: simplify object_store path handling ( #4534 )
...
* refactor: simplify object_store path handling
* fix: aws integration tests
* chore: lint
* fix: update gcs tests
* refactor: move errors into submodules
* chore: lint
* chore: review feedback
* refactor: replace provider with Display
* fix: failing tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-09 18:43:22 +00:00
a12172dfe8
Merge branch 'main' into cn/delete-more
2022-05-09 08:47:03 +00:00
65189f56f4
chore(deps): Bump cloud-storage from 0.11.0 to 0.11.1 ( #4538 )
...
Bumps [cloud-storage](https://github.com/ThouCheese/cloud-storage-rs ) from 0.11.0 to 0.11.1.
- [Release notes](https://github.com/ThouCheese/cloud-storage-rs/releases )
- [Changelog](https://github.com/ThouCheese/cloud-storage-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ThouCheese/cloud-storage-rs/commits )
---
updated-dependencies:
- dependency-name: cloud-storage
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-05-09 08:19:08 +00:00
e07bcd40c2
refactor: Remove unused dependencies
...
These were found by iterating over all of the dependencies of each
Cargo.toml, then grepping that crate for the dependency's name. If it
didn't show up, I attempted to remove it.
I left a few dependencies that this process flagged:
* generated_types
- `pbjson`,`serde`. Apparently used by the generated code.
* grpc-router-test-gen
- `prost`. Apparently used by the generated code.
* influxdb_iox
- `heappy`. Doesn't appear used, but is behind enough feature
flags that I don't care to reason about and it's already optional.
- `tikv_jemalloc_sys`. Appears to be setting a feature flag of an
indirect dependency.
* iox_gitops_adapter
- `k8s_openapi`. Appears to be setting a feature flag of an indirect
dependency.
2022-05-06 15:57:58 -04:00
420c306caa
chore(deps): Bump tokio from 1.17.0 to 1.18.0 ( #4453 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.17.0 to 1.18.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.17.0...tokio-1.18.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-28 08:21:17 +00:00
663944a2a9
chore(deps): Bump all rusoto crates from 0.47.0 to 0.48.0 ( #4439 )
...
* chore(deps): Bump rusoto_credential from 0.47.0 to 0.48.0
Bumps [rusoto_credential](https://github.com/rusoto/rusoto ) from 0.47.0 to 0.48.0.
- [Release notes](https://github.com/rusoto/rusoto/releases )
- [Changelog](https://github.com/rusoto/rusoto/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rusoto/rusoto/compare/rusoto-v0.47.0...mock-v0.48.0 )
---
updated-dependencies:
- dependency-name: rusoto_credential
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): Bump rusoto_core from 0.47.0 to 0.48.0
Bumps [rusoto_core](https://github.com/rusoto/rusoto ) from 0.47.0 to 0.48.0.
- [Release notes](https://github.com/rusoto/rusoto/releases )
- [Changelog](https://github.com/rusoto/rusoto/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rusoto/rusoto/compare/rusoto-v0.47.0...mock-v0.48.0 )
---
updated-dependencies:
- dependency-name: rusoto_core
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): Bump rusoto_s3 from 0.47.0 to 0.48.0
Bumps [rusoto_s3](https://github.com/rusoto/rusoto ) from 0.47.0 to 0.48.0.
- [Release notes](https://github.com/rusoto/rusoto/releases )
- [Changelog](https://github.com/rusoto/rusoto/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rusoto/rusoto/compare/rusoto-v0.47.0...mock-v0.48.0 )
---
updated-dependencies:
- dependency-name: rusoto_s3
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-27 14:58:40 +00:00
4b47d723b1
refactor: Rename time to iox_time ( #4416 )
...
Signed-off-by: hi-rustin <rustin.liu@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-26 00:19:59 +00:00
9950bcee27
chore(deps): Bump indexmap from 1.8.0 to 1.8.1 ( #4166 )
...
Bumps [indexmap](https://github.com/bluss/indexmap ) from 1.8.0 to 1.8.1.
- [Release notes](https://github.com/bluss/indexmap/releases )
- [Changelog](https://github.com/bluss/indexmap/blob/master/RELEASES.md )
- [Commits](https://github.com/bluss/indexmap/compare/1.8.0...1.8.1 )
---
updated-dependencies:
- dependency-name: indexmap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-30 08:49:07 +00:00
a373c90415
refactor: Extract the list_all function to object store
...
I'm about to use this in a third file, so time to extract this.
Make it clear that this is appropriate for tests only.
2022-03-29 08:15:24 -04:00
17af5fcbd1
chore(deps): Bump tokio-util from 0.7.0 to 0.7.1 ( #4154 )
...
* chore(deps): Bump tokio-util from 0.7.0 to 0.7.1
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.7.0 to 0.7.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.7.0...tokio-util-0.7.1 )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-29 08:39:02 +00:00
4f9515ffba
chore(deps): Bump async-trait from 0.1.52 to 0.1.53 ( #4141 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.52 to 0.1.53.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.52...0.1.53 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-28 08:55:24 +00:00
5585dd3c21
refactor: switch to using DynObjectStore
...
Changes all consumers of the object store to use the dynamically
dispatched DynObjectStore type, instead of using a hardcoded concrete
implementation type.
2022-03-15 16:32:52 +00:00
b727d26dab
refactor: path_from_dirs_and_filename trait method
...
Moves the path_from_dirs_and_filename from an ObjectStoreImpl method to
a trait method, completing the abstraction over all object store
backends.
2022-03-15 16:29:43 +00:00
1d5066c421
refactor: rename ObjectStore -> ObjectStoreImpl
...
Frees up the name for so we can use `dyn ObjectStore` throughout the
code instead of `ObjectStoreApi`.
2022-03-15 16:29:43 +00:00
7a7dfade94
feat(catalog): instrument list() ops
...
List (but not list_with_delimiter?!) returns a stream of ListResult
which previously wasn't instrumented - this commit uses the
StreamMetricRecorder to record the wall clock duration of the entire
list operation.
2022-03-15 14:23:11 +00:00
ee01225fec
refactor: polymorphic stream recorder impl
...
Changes the StreamMetricRecorder to be generic over the Ok types in the
result stream, invoking a T-specific delegate when Ok(T) is observed.
This enables the stream instrumentation to be reused across different
stream types while keeping the hairy state checks DRY.
This will allow the StreamMetricRecorder to decorate the streams
returned by both the get() and list() operations, but this commit causes
no functional change.
2022-03-15 14:13:17 +00:00
1aaf193299
refactor: take last_yielded_at
2022-03-14 11:54:02 +00:00
3802401b4f
docs: explicit stream duration semantics
2022-03-14 11:54:02 +00:00
1aee8402bf
refactor: suffix duration vars with _ms
2022-03-14 11:54:02 +00:00
f01382f578
docs: fix typos
...
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2022-03-14 11:36:49 +00:00
dc8a708fb7
refactor: use DummyObjectStore for error tests
2022-03-11 14:36:34 +00:00
bfc473b8cd
feat: object store instrumentation
...
Implements a decorator of the ObjectStoreAPI trait, recording:
* Bytes uploaded / downloaded through the instrumented API
* Call latencies, broken down by operation & success / error state
All the current implementations that return a Stream from the get
operation actually return a "fake" stream containing all the data in one
go rather than streaming chunks from the upstream. I've instrumented the
Stream to future-proof any actual streaming impls in the future.
2022-03-11 13:38:48 +00:00
b870b9340b
chore: remove uneeded dependencies ( #3929 )
...
* chore: remove unused deps in compactor
* chore: remove unused deps in influxdb_ioxd
* chore: remove unused deps in object_store
* chore: remove unused deps in server
* fix: object_store needs observability deps when compiled with aws
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-04 17:39:46 +00:00
cf0c238ae4
feat: allow unencrypted HTTP connections to AWS via flag ( #3916 )
...
This is required for the test bench.
2022-03-03 17:01:03 +00:00
46bb107be4
refactor: use rustls
...
Removes openssl as a dependency, switching to rustls[1] as the TLS
implementation throughout.
It is important to note that this change brings with it a significant
behavioural difference - rustls does not currently support IP SANs in
certificates (instead only supporting fully-qualified names / DNS) and
this will manifest as a failure to connect to IP endpoints over TLS.
This might be a blocker that prevents us using rustls exclusively, but
there's noe asy way to know without trying it. Fortunately the rustls
project has received funding to work on IP SAN support[2].
[1]: https://github.com/rustls/rustls
[2]: https://www.abetterinternet.org/post/preparing-rustls-for-wider-adoption/
2022-03-03 11:05:20 +00:00
4785a7d028
refactor: bump Azure SDK
...
Update to the latest Azure SDK to pick up rustls support.
2022-03-03 11:02:20 +00:00
ad3868ed7c
chore(deps): Bump tokio from 1.16.1 to 1.17.0 ( #3814 )
...
* chore(deps): Bump tokio from 1.16.1 to 1.17.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.16.1 to 1.17.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.16.1...tokio-1.17.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build: update workspace-hack
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:27:43 +00:00
89105ccfab
chore(deps): Bump tokio-util from 0.6.9 to 0.7.0 ( #3743 )
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.6.9 to 0.7.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/commits )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-15 11:33:41 +00:00
85d44552fa
chore(deps): bump cloud-storage from 0.10.3 to 0.11.0 ( #3697 )
...
Bumps [cloud-storage](https://github.com/ThouCheese/cloud-storage-rs ) from 0.10.3 to 0.11.0.
- [Release notes](https://github.com/ThouCheese/cloud-storage-rs/releases )
- [Changelog](https://github.com/ThouCheese/cloud-storage-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ThouCheese/cloud-storage-rs/commits )
---
updated-dependencies:
- dependency-name: cloud-storage
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-02-09 20:03:44 +00:00
f3f6f335a9
chore: upgrade to snafu 0.7 ( #3440 )
2022-01-11 19:22:36 +00:00
099d9dc6e1
Merge branch 'main' into crepererum/issue3226
2021-12-17 16:37:16 +00:00
24fd2e549b
docs: Fix some typos and outdated comments
2021-12-17 11:33:36 -05:00
9f2694bf1b
test: `objest_store` azure support via Azurite
2021-12-17 09:30:21 +01:00
61ee70d222
refactor: make object store prefixes segment based
...
Don't use string prefixes, e.g. `foo/bar/` is a prefix of `foo/bar/x`
but NOT of `foo/bar_baz/y`.
This also removes some heuristics during the cloud storage parsing that
assumed that file names always contain a dot but directories don't.
Technically we should now always be able to know whether a path points
to a file or a directory:
- Rust (manually constructed): we use `DirsAndFileName` which knows the
difference (i.e. if `file_name` is set)
- in-mem store: we also use `DirsAndFileName`
- file system: this was fixed by #1523
(ccd094dfcf464667d8b8Fixes #3226 .
2021-12-16 14:40:45 +01:00
f02d846389
test: `object_store` aws test using localstack
...
This removes 3 "nonexisting region" tests that where testing very
specific error behavior that no local emulator (minio and localstack)
replicate and that don't add much value. It's better to test our AWS
code at all than being to picky.
2021-12-16 10:10:06 +01:00
876a9af35a
fix: limit number of S3 connections
...
Otherwise the whole thing blows up when starting a server that has many
DBs registerd, because we potentially create 1 connection per DB (e.g.
to read out the preserved catalog).
Fixes #3336 .
2021-12-08 19:06:02 +01:00
02c297e850
fix: Always specify the parking_lot feature of tokio to get potential perf boost
2021-12-06 09:37:15 -05:00
39a7077cd9
fix: Turn on std for indexmap to avoid a build-time target check
...
Connects to #3117 .
2021-12-06 09:37:14 -05:00
7a241882c2
fix: Make version reqs on futures-related crates consistent
2021-12-06 09:37:14 -05:00
39862e3886
fix: Disable default chrono features
...
Connects to #3117 . Won't affect the workspace-hack crate until changes
to some other crates can be upstreamed.
2021-12-06 09:36:49 -05:00
bca561366b
feat: don't copy parquet files out of disk object store ( #3282 ) ( #3293 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-05 16:31:40 +00:00
5d0fd1c603
fix: Allow dead code on fields that are now detected as never read
2021-12-02 11:52:01 -05:00
9fd4a560f5
feat: Results of running cargo hakari manage-deps
2021-11-19 09:21:57 -05:00
f0136e9791
ci: run clippy for all features
...
This indeed fixes some issues in our object store implementation.
Closes #537 .
2021-11-11 10:48:21 +01:00
c540b40f05
chore(deps): bump tokio from 1.12.0 to 1.13.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.12.0...tokio-1.13.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 11:21:59 +00:00
a1f978e280
chore(deps): bump tokio-util from 0.6.8 to 0.6.9
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.6.8 to 0.6.9.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.6.8...tokio-util-0.6.9 )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 01:28:41 +00:00
Raphael Taylor-Davies
kodiakhq[bot]
dependabot[bot]
Jake Goulding
Carol (Nichols || Goulding)
二手掉包工程师
Carol (Nichols || Goulding)
Dom Dwyer
Andrew Lamb
Marco Neumann