f2bb0fdf77
feat: update to crates.io object_store version ( #4595 )
...
* feat: update to crates.io object_store version
* chore: Run cargo hakari tasks
* fix: tests
* chore: remove object store integration test plumbing
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-05-13 16:26:07 +00:00
35d80fc512
refactor: rename `iox_catalog_service` to `service_grpc_catalog` for consistency ( #4581 )
...
* refactor: rename `iox_catalog_service` to `service_grpc_catalog`
* fix: fmt
* fix: alphabetize crate
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-13 14:07:58 +00:00
7eff52d04e
refactor: rename `iox_object_store_service` to `service_grpc_object_store` for consisntency ( #4572 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-12 14:22:18 +00:00
b8cb4c3f2b
feat: Interrogate schema from querier (as well as router) ( #4557 )
...
* refactor: move SchemaService into `service_grpc_schema`
* feat: implement schema gRPC for querier
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-05-10 20:55:58 +00:00
776b0e0778
docs: finish incomplete comments
2022-05-09 14:46:32 +01:00
31f805faa5
docs: partial sharded write semantics
...
Documents the partial write semantics and expected resolution for the
ShardedWriteBuffer in the router.
2022-05-09 14:46:32 +01:00
5a33749c98
chore: fix test comments
...
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-05-09 14:46:32 +01:00
5069e37bd7
feat: shard delete ops with no table predicate
...
Changes the JumpHash sharder and modifies the ShardedWriteBuffer's
DmlHandler::delete() impl in order to enqueue delete ops across all
shards if no table name is specified.
If a table name is specified, it is sharded as before: a delete for a
given table & namespace always maps to the same shard as a write to the
same table & namespace.
2022-05-09 14:46:30 +01:00
e07bcd40c2
refactor: Remove unused dependencies
...
These were found by iterating over all of the dependencies of each
Cargo.toml, then grepping that crate for the dependency's name. If it
didn't show up, I attempted to remove it.
I left a few dependencies that this process flagged:
* generated_types
- `pbjson`,`serde`. Apparently used by the generated code.
* grpc-router-test-gen
- `prost`. Apparently used by the generated code.
* influxdb_iox
- `heappy`. Doesn't appear used, but is behind enough feature
flags that I don't care to reason about and it's already optional.
- `tikv_jemalloc_sys`. Appears to be setting a feature flag of an
indirect dependency.
* iox_gitops_adapter
- `k8s_openapi`. Appears to be setting a feature flag of an indirect
dependency.
2022-05-06 15:57:58 -04:00
fcd4815645
fix: Rename router2 to router
2022-05-06 14:51:52 -04:00
e0bc1801ac
fix: Remove router
2022-05-06 11:30:36 -04:00
420c306caa
chore(deps): Bump tokio from 1.17.0 to 1.18.0 ( #4453 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.17.0 to 1.18.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.17.0...tokio-1.18.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-28 08:21:17 +00:00
4b47d723b1
refactor: Rename time to iox_time ( #4416 )
...
Signed-off-by: hi-rustin <rustin.liu@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-26 00:19:59 +00:00
e2ebeff39c
feat: Add write tokens to the gRPC write endpoint response
...
As a header in the same way the HTTP write endpoint does.
Fixes #4208 .
2022-04-25 09:46:13 -04:00
ad3868ed7c
chore(deps): Bump tokio from 1.16.1 to 1.17.0 ( #3814 )
...
* chore(deps): Bump tokio from 1.16.1 to 1.17.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.16.1 to 1.17.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.16.1...tokio-1.17.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build: update workspace-hack
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:27:43 +00:00
6c3b208438
chore(deps): bump cache_loader_async from 0.1.2 to 0.2.0 ( #3665 )
...
* chore(deps): bump cache_loader_async from 0.1.2 to 0.2.0
Bumps [cache_loader_async](https://github.com/ZeroTwo-Bot/cache-loader-async-rs ) from 0.1.2 to 0.2.0.
- [Release notes](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/releases )
- [Changelog](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/commits/v0.2.0 )
---
updated-dependencies:
- dependency-name: cache_loader_async
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: breaking changes from `cache_loader_async`
* fix: Update workspace hack: tokio full is no longer needed
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
2022-02-10 17:01:11 +00:00
ca331503a5
feat: add WriteBufferErrorKind ( #3664 )
...
* feat: add WriteBufferErrorKind
* fix: test_offset_after_broken_message
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 15:34:05 +00:00
50cff27b01
chore: remove rdkafka dependency ( #3625 )
...
All features are now covered by rskafka. This also removes the need to
specify a server ID for write buffer consumers. This was only used for
rdkafka since there we needed to specify a consumer group, even though
we did not use any transactions.
2022-02-03 13:33:56 +00:00
22778a3a80
chore: upgrade rskafka and parking_lot ( #3592 )
2022-02-01 11:50:42 +00:00
b326b62b44
feat: buffer writes when writing to RSKafka ( #3520 )
2022-02-01 10:07:52 +00:00
2062267d0f
chore: Update hashbrown ( #3551 )
...
* chore: Update hashbrown
* fix: hakari
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 15:34:10 +00:00
16d584b2ff
feat: Add db_name/namespace to DmlWrite and DmlDelete ( #3531 )
...
* feat: Add db_name/namespace to DmlWrite and DmlDelete
This is required for the new ingester to be able to work with the write buffer. The protobuf that gets serialized over Kafka already includes the database name, it just wasn't getting carried through to the marshaled Dml operation.
* fix: database != namespace, propagation through write buffer
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 14:12:20 +00:00
9c19cd6cc4
fix: clamp start/end of TimestampRange to min/max valid timestamp values ( #3487 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-20 16:08:00 +00:00
f3f6f335a9
chore: upgrade to snafu 0.7 ( #3440 )
2022-01-11 19:22:36 +00:00
471c3181bb
feat: Make a top-level influxdb_iox feature for kafka functionality
...
Thread the feature through router and server to the write buffer crate.
Move an end-to-end test that uses Kafka behind the feature flag.
2021-12-09 08:49:34 -05:00
16d8ae5e04
fix: Match tokio features to what's actually in use in each crate
...
Some crates listed features they don't use; other crates ware relying on
feature flags enabled by something else. I tested these changes by
disabling the workspace hack crate and testing each crate.
2021-12-06 09:37:16 -05:00
02c297e850
fix: Always specify the parking_lot feature of tokio to get potential perf boost
2021-12-06 09:37:15 -05:00
30c4da7ca7
fix: Be consistent with regex version range specification
2021-12-06 09:37:15 -05:00
f17b670fe9
refactor: cleanup gRPC client error handling ( #3251 ) ( #3277 )
...
* refactor: cleanup gRPC client error handling (#3251 )
* chore: review feedback
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-02 20:56:58 +00:00
ad65687cba
refactor: consolidate sharding implementation
...
Move sharding implementation from router to `dml` (where the types can
be consumed w/o cloning the data) and only support the new sharding
configs.
The old sharding configs will be removed soon.
2021-11-26 13:00:36 +01:00
7f2e4f4342
refactor: remove write buffer direction
...
The direction was required when a database could read or write from/to a
write buffer. Now it is clear from the usage context of a write buffer
context which of the two applications is meant (databases read, routers
write) so the direction flag is no longer required.
2021-11-26 12:38:40 +01:00
5ecaa6d8a9
refactor: use serialized predicates instead of strings for gRPC deletes
...
IOx is the only consumer of this API so we might just use the serialized
form. Cloud2 uses the HTTP API which supports SQL-like predicates.
Fixes #3192 .
2021-11-24 09:30:32 +01:00
93cc79875a
refactor: simplify code
2021-11-23 15:39:53 +01:00
18cab1545c
test: improve test naming
...
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2021-11-23 15:39:53 +01:00
df19d67532
test: allow to specify `WriteBufferConfigFactory` when creating a test `RouterServer`
2021-11-23 15:39:53 +01:00
964411bdbb
feat: add delete support to router
2021-11-23 15:39:53 +01:00
3ed9dfc4db
fix: Make all usages of cache_loader_async use the same features
2021-11-19 14:33:55 -05:00
9fd4a560f5
feat: Results of running cargo hakari manage-deps
2021-11-19 09:21:57 -05:00
8155747735
feat: add write buffer delete encoding ( #2731 ) ( #3127 )
...
* feat: add write buffer delete encoding (#2731 )
* chore: fix doc
* chore: review feedback
* chore: review feedback
* chore: fmt
* chore: review feedback
2021-11-17 16:12:19 +00:00
e6fdd79a0f
feat: emit Kafka stats as metrics instead of logs
...
This maps a subset of Kafka stats as metrics. The set can -- of course
-- be changed in the future depending on our needs.
Fixes #3100 .
2021-11-16 17:18:41 +01:00
553e412226
refactor: DMLOperation write path ( #2731 ) ( #3121 )
...
* refactor: DMLOperation write path (#2731 )
* chore: fmt
* chore: review feedback
2021-11-16 12:42:19 +00:00
6f268f8260
refactor: extract DML types ( #2731 ) ( #3084 )
...
* refactor: extract DML types (#2731 )
* chore: fmt
2021-11-11 12:34:07 +00:00
09239c4943
feat: improve testing interface for the `router` crate
2021-11-10 17:51:58 +01:00
ea2604b7b6
docs: explain write sink test
2021-11-08 14:09:33 +01:00
4130d508c0
docs: document write buffer variants
2021-11-08 14:09:33 +01:00
1d34ba0183
docs: explain router name argument
2021-11-08 14:09:33 +01:00
12c684b546
test: extend `test_write` test
2021-11-08 14:09:33 +01:00
dcc376456a
docs: explain iteration order in `Router`
2021-11-08 14:09:33 +01:00
6a090b785a
fix: use new `WriteMeta` constructor
2021-11-08 14:09:33 +01:00
b226d629c1
feat: router write logic
2021-11-08 14:09:33 +01:00
Raphael Taylor-Davies
Andrew Lamb
Dom Dwyer
Jake Goulding
Carol (Nichols || Goulding)
dependabot[bot]
二手掉包工程师
Marco Neumann
Andrew Lamb
Paul Dix