22778a3a80
chore: upgrade rskafka and parking_lot ( #3592 )
2022-02-01 11:50:42 +00:00
b326b62b44
feat: buffer writes when writing to RSKafka ( #3520 )
2022-02-01 10:07:52 +00:00
2062267d0f
chore: Update hashbrown ( #3551 )
...
* chore: Update hashbrown
* fix: hakari
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 15:34:10 +00:00
16d584b2ff
feat: Add db_name/namespace to DmlWrite and DmlDelete ( #3531 )
...
* feat: Add db_name/namespace to DmlWrite and DmlDelete
This is required for the new ingester to be able to work with the write buffer. The protobuf that gets serialized over Kafka already includes the database name, it just wasn't getting carried through to the marshaled Dml operation.
* fix: database != namespace, propagation through write buffer
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 14:12:20 +00:00
9c19cd6cc4
fix: clamp start/end of TimestampRange to min/max valid timestamp values ( #3487 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-20 16:08:00 +00:00
f3f6f335a9
chore: upgrade to snafu 0.7 ( #3440 )
2022-01-11 19:22:36 +00:00
471c3181bb
feat: Make a top-level influxdb_iox feature for kafka functionality
...
Thread the feature through router and server to the write buffer crate.
Move an end-to-end test that uses Kafka behind the feature flag.
2021-12-09 08:49:34 -05:00
16d8ae5e04
fix: Match tokio features to what's actually in use in each crate
...
Some crates listed features they don't use; other crates ware relying on
feature flags enabled by something else. I tested these changes by
disabling the workspace hack crate and testing each crate.
2021-12-06 09:37:16 -05:00
02c297e850
fix: Always specify the parking_lot feature of tokio to get potential perf boost
2021-12-06 09:37:15 -05:00
30c4da7ca7
fix: Be consistent with regex version range specification
2021-12-06 09:37:15 -05:00
f17b670fe9
refactor: cleanup gRPC client error handling ( #3251 ) ( #3277 )
...
* refactor: cleanup gRPC client error handling (#3251 )
* chore: review feedback
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-02 20:56:58 +00:00
ad65687cba
refactor: consolidate sharding implementation
...
Move sharding implementation from router to `dml` (where the types can
be consumed w/o cloning the data) and only support the new sharding
configs.
The old sharding configs will be removed soon.
2021-11-26 13:00:36 +01:00
7f2e4f4342
refactor: remove write buffer direction
...
The direction was required when a database could read or write from/to a
write buffer. Now it is clear from the usage context of a write buffer
context which of the two applications is meant (databases read, routers
write) so the direction flag is no longer required.
2021-11-26 12:38:40 +01:00
5ecaa6d8a9
refactor: use serialized predicates instead of strings for gRPC deletes
...
IOx is the only consumer of this API so we might just use the serialized
form. Cloud2 uses the HTTP API which supports SQL-like predicates.
Fixes #3192 .
2021-11-24 09:30:32 +01:00
93cc79875a
refactor: simplify code
2021-11-23 15:39:53 +01:00
18cab1545c
test: improve test naming
...
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2021-11-23 15:39:53 +01:00
df19d67532
test: allow to specify `WriteBufferConfigFactory` when creating a test `RouterServer`
2021-11-23 15:39:53 +01:00
964411bdbb
feat: add delete support to router
2021-11-23 15:39:53 +01:00
3ed9dfc4db
fix: Make all usages of cache_loader_async use the same features
2021-11-19 14:33:55 -05:00
9fd4a560f5
feat: Results of running cargo hakari manage-deps
2021-11-19 09:21:57 -05:00
8155747735
feat: add write buffer delete encoding ( #2731 ) ( #3127 )
...
* feat: add write buffer delete encoding (#2731 )
* chore: fix doc
* chore: review feedback
* chore: review feedback
* chore: fmt
* chore: review feedback
2021-11-17 16:12:19 +00:00
e6fdd79a0f
feat: emit Kafka stats as metrics instead of logs
...
This maps a subset of Kafka stats as metrics. The set can -- of course
-- be changed in the future depending on our needs.
Fixes #3100 .
2021-11-16 17:18:41 +01:00
553e412226
refactor: DMLOperation write path ( #2731 ) ( #3121 )
...
* refactor: DMLOperation write path (#2731 )
* chore: fmt
* chore: review feedback
2021-11-16 12:42:19 +00:00
6f268f8260
refactor: extract DML types ( #2731 ) ( #3084 )
...
* refactor: extract DML types (#2731 )
* chore: fmt
2021-11-11 12:34:07 +00:00
09239c4943
feat: improve testing interface for the `router` crate
2021-11-10 17:51:58 +01:00
ea2604b7b6
docs: explain write sink test
2021-11-08 14:09:33 +01:00
4130d508c0
docs: document write buffer variants
2021-11-08 14:09:33 +01:00
1d34ba0183
docs: explain router name argument
2021-11-08 14:09:33 +01:00
12c684b546
test: extend `test_write` test
2021-11-08 14:09:33 +01:00
dcc376456a
docs: explain iteration order in `Router`
2021-11-08 14:09:33 +01:00
6a090b785a
fix: use new `WriteMeta` constructor
2021-11-08 14:09:33 +01:00
b226d629c1
feat: router write logic
2021-11-08 14:09:33 +01:00
c745a0a94c
feat: impl write sinks logic for router
2021-11-08 14:09:33 +01:00
fc4ea71921
feat: `RouterServer::router`
2021-11-08 14:09:33 +01:00
9984e4d386
fix: debug msg
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-11-08 11:11:52 +01:00
7f6280305d
fix: `ConnectionPool::new` requires tokio
2021-11-08 11:11:52 +01:00
11f4a1dee8
feat: add connection management for router
2021-11-08 11:11:52 +01:00
898567e221
feat: migrate server to DbWrite ( #2724 ) ( #3035 )
...
* feat: migrate server to DbWrite (#2724 )
* chore: print perf log output
* fix: don't suppress CI status code
* chore: review feedback
* fix: don't error on empty line protocol write payloads
* fix: test
* fix: test
2021-11-05 11:09:33 +00:00
92e6acaacd
feat: implement sharder for new router
2021-11-05 09:49:33 +01:00
78101eb52d
feat: implement `RemoteService` for new router mode
2021-11-04 15:35:05 +01:00
4840dc11cb
feat: add basic CRUD operations for router configs
2021-11-04 12:03:14 +01:00
a91e572b43
feat: implement basic router server type
...
This only implements the deployment service and nothing else yet.
2021-11-03 16:34:51 +01:00
Marco Neumann
Andrew Lamb
Paul Dix
Carol (Nichols || Goulding)
Raphael Taylor-Davies