ad3868ed7c
chore(deps): Bump tokio from 1.16.1 to 1.17.0 ( #3814 )
...
* chore(deps): Bump tokio from 1.16.1 to 1.17.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.16.1 to 1.17.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.16.1...tokio-1.17.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build: update workspace-hack
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:27:43 +00:00
89105ccfab
chore(deps): Bump tokio-util from 0.6.9 to 0.7.0 ( #3743 )
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.6.9 to 0.7.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/commits )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-15 11:33:41 +00:00
26fd5273f0
feat: static database configuration ( #2436 ) ( #3732 )
...
* feat: static database configuration (#2436 )
* chore: fmt
* feat: don't base64 encode UUIDs in ServerConfigFile
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 19:42:49 +00:00
c79050254f
refactor: traitify database configuration ( #2436 ) ( #3730 )
...
* refactor: traitify database configuration (#2436 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-13 09:26:44 +00:00
0cc34b2a00
feat: server and database state gauge metrics ( #3722 )
2022-02-11 10:49:28 +00:00
4e3f66ed07
feat: CLI and gRPC APIs for shutting down and restarting databases ( #3720 )
...
* feat: allow catalog wipe and rebuild whilst shutdown
* feat: CLI and gRPC APIs for shutting down and restarting databases
* feat: add ability to skip replay on restart
* fix: test_wipe_persisted_catalog_error_db_exists
* fix: wipe_preserved_catalog
2022-02-11 10:14:43 +00:00
b7a4ef46dd
feat: preempt init shutdown ( #3368 ) ( #3717 )
2022-02-10 20:39:51 +00:00
910f381355
refactor: require UUID to create Database ( #3715 )
...
* refactor: require UUID to create Database
* chore: review feedback
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 20:04:06 +00:00
b1190262b7
feat: restartable `Database` ( #3368 ) ( #3711 )
...
* feat: restartable `Database` (#3368 )
* chore: fmt
* fix: wipe_preserved_catalog
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 18:32:05 +00:00
6c3b208438
chore(deps): bump cache_loader_async from 0.1.2 to 0.2.0 ( #3665 )
...
* chore(deps): bump cache_loader_async from 0.1.2 to 0.2.0
Bumps [cache_loader_async](https://github.com/ZeroTwo-Bot/cache-loader-async-rs ) from 0.1.2 to 0.2.0.
- [Release notes](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/releases )
- [Changelog](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/commits/v0.2.0 )
---
updated-dependencies:
- dependency-name: cache_loader_async
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: breaking changes from `cache_loader_async`
* fix: Update workspace hack: tokio full is no longer needed
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
2022-02-10 17:01:11 +00:00
c31739de2c
refactor: simplify Database retry backoff logic ( #3368 ) ( #3689 )
2022-02-09 17:06:32 +00:00
ba87ae2918
chore(deps): bump crc32fast from 1.3.1 to 1.3.2 ( #3668 )
...
Bumps [crc32fast](https://github.com/srijs/rust-crc32fast ) from 1.3.1 to 1.3.2.
- [Release notes](https://github.com/srijs/rust-crc32fast/releases )
- [Commits](https://github.com/srijs/rust-crc32fast/compare/v1.3.1...v1.3.2 )
---
updated-dependencies:
- dependency-name: crc32fast
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-08 14:48:31 +00:00
e2db1df11f
refactor: improve writer buffer consumer interface ( #3631 )
...
* refactor: improve writer buffer consumer interface
The change looks huge but is actually rather simple. To
understand the interface change, let me first explain what we want:
- be able to fetch watermarks for any sequencer
- have streams:
- each streams tracks a sequencer and has an offset state (no read
multiplexing)
- we can seek a stream
- seeking and streaming cannot be done at the same time (that would be
weird and likely leads to many bugs both in write buffer and in the
user code)
- ideally we don't need to create streams of all sequencers but can
choose a subset
Before this change we had one mutable consumer struct where you can get
all streams and watermark functions (this mutable-borrows the consumer)
or you can seek a single stream (this also mutable-borrows the
consumer). This is a bit weird for multiple reasons:
- you cannot seek a single stream without dropping all of them
- the mutable-borrow construct makes it really difficult to pass the
streams into separate threads
- the consumer is boxed (because its mutable) which makes it more
difficult to handle in a large-scale application
What this change does is the following:
- you have an immutable consumer (similar to the producer)
- the consumer offers the following methods:
- get the set of sequencer IDs
- get watermark for any sequencer
- get a stream handler (see next point) for any sequencer
- the stream handler captures the stream state (offset) and provides you
a standard `Stream<_>` interface as well as a seek function.
Mutable-borrows ensure that you cannot use both at the same time.
The stream handler provides you the stream via `handler.stream()`. It
doesn't implement `Stream<_>` itself because the way boxing, dynamic
dispatch work, and pinning interact (i.e. I couldn't get it to work
without the indirection).
As a bonus point (which we don't use however) you can now create
multiple streams for the same sequencer and they all have their own
offset.
* fix: review comments
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-07 12:24:17 +00:00
50cff27b01
chore: remove rdkafka dependency ( #3625 )
...
All features are now covered by rskafka. This also removes the need to
specify a server ID for write buffer consumers. This was only used for
rdkafka since there we needed to specify a consumer group, even though
we did not use any transactions.
2022-02-03 13:33:56 +00:00
22778a3a80
chore: upgrade rskafka and parking_lot ( #3592 )
2022-02-01 11:50:42 +00:00
4101d16f71
chore: feature flag consistency ( #3574 )
...
* chore: feature flag consistency
* chore: add aarch64-apple-darwin to hakari
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-28 16:38:59 +00:00
2062267d0f
chore: Update hashbrown ( #3551 )
...
* chore: Update hashbrown
* fix: hakari
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 15:34:10 +00:00
21c1824a7a
refactor: remove table_names from Predicate ( #3545 )
...
* refactor: remove table_names from Predicate
* chore: fix benchmarks
* chore: review feedback
Co-authored-by: Edd Robinson <me@edd.io>
* chore: review feedback
* chore: replace Default::default with InfluxRpcPredicate::default()
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 14:44:49 +00:00
16d584b2ff
feat: Add db_name/namespace to DmlWrite and DmlDelete ( #3531 )
...
* feat: Add db_name/namespace to DmlWrite and DmlDelete
This is required for the new ingester to be able to work with the write buffer. The protobuf that gets serialized over Kafka already includes the database name, it just wasn't getting carried through to the marshaled Dml operation.
* fix: database != namespace, propagation through write buffer
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 14:12:20 +00:00
9c19cd6cc4
fix: clamp start/end of TimestampRange to min/max valid timestamp values ( #3487 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-20 16:08:00 +00:00
f3f6f335a9
chore: upgrade to snafu 0.7 ( #3440 )
2022-01-11 19:22:36 +00:00
6d20ce1f9e
feat: Allow wipe catalog in `ReplayError` and `WriteBufferCreationError` states ( #3425 )
...
* feat: feat: Allow wipe catalog in ReplayError
* fix: comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-07 17:07:44 +00:00
336ffd1966
refactor: Remove `Result` in QueryDatabase trait (none of the functions can fail) ( #3422 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-06 22:03:08 +00:00
8d732ad78c
refactor: Split database.rs into smaller modules ( #3424 )
...
* refactor: Split database.rs into smaller modules
* fix: clippy
2022-01-06 21:42:36 +00:00
f9174c483b
refactor: Extract server::db into its own crate ( #3417 )
...
* refactor: Extract JobRegistry from the server crate
Both the server crate and a db crate that I'm about to extract depend on
JobRegistry, so to avoid making circular dependencies, extract the
JobRegistry to its own crate.
* refactor: Move db out of server into its own crate
Fixes #2821 .
2021-12-23 22:01:17 +00:00
ccba68fe3e
chore: Update datafusion and sqlparser, and arrow ( #3385 )
...
* chore: Update datafusion and sqlparser, and arrow
* fix: cargo hakari
* test: fix metadata size
* fix: update some metadata sizes
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-16 12:13:40 +00:00
499265b01e
chore: add db_name to background worker log messages ( #3386 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-16 11:13:13 +00:00
290f8c7630
Merge branch 'main' into crepererum/issue3335
2021-12-15 16:50:57 +00:00
758b65dd29
feat: Add database initialization state and errors to CLI and remove list_databases_detailed gRPC ( #3377 )
...
* feat: Add database initialization state and errors to CLI:
* fix: do not use optional in protobuf
* fix: clippy
* fix: correct check I broke appeasing clippy
2021-12-15 12:18:41 +00:00
e9041e6da0
refactor: improve database init routine
...
1. Simpler locking/freezing
2. Don't commit the rollback from `ReplayError` to `RulesLoaded` because
it will confuse the heck out of our users/admins.
2021-12-15 12:43:33 +01:00
3841598296
fix: typo
...
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2021-12-15 12:41:30 +01:00
8520b9b9db
fix: improve docs and naming
2021-12-15 12:41:30 +01:00
0682fc208f
feat: also recover from `NoActiveDatabase`
2021-12-15 12:41:30 +01:00
0f3cfb024b
test: add database startup recovery test
2021-12-15 12:41:30 +01:00
e80ac94af5
fix: database state diagram
2021-12-15 12:41:30 +01:00
7a1f70f970
fix: move warning to better place
2021-12-15 12:41:30 +01:00
11df3091a1
fix: misleading comment
2021-12-15 12:41:30 +01:00
496043a767
fix: PR feedback
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-12-15 12:41:30 +01:00
10a51bac61
refactor: allow database init to recover from errors
...
Closes #3335 .
2021-12-15 12:41:30 +01:00
7783149af1
fix: typos
2021-12-13 10:06:17 -05:00
e76157f0e4
chore: cleanup
2021-12-10 12:08:04 -05:00
5253c71e03
test: replay test for compact OS chunks
2021-12-10 12:04:00 -05:00
2e3aa2e1ab
Merge branch 'main' into ntran/test_compact_os_delete_propagation
2021-12-10 15:11:11 +00:00
dc152abb55
Merge branch 'main' into alamb/update_state_machine
2021-12-10 09:51:40 +00:00
7d53fad427
Merge branch 'main' into ntran/test_compact_os_delete_propagation
2021-12-09 17:27:22 -05:00
0d6fefdd09
test: test concurrent deletes and OS compaction
2021-12-09 17:13:18 -05:00
c0ba69f09e
chore: marge main to branch and resolve conflict
2021-12-09 15:40:33 -05:00
191e743ce0
fix: Update server/src/database.rs
...
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-12-09 15:04:39 -05:00
b2f7306d5a
docs: Update database startup machine diagram
2021-12-09 14:22:29 -05:00
471c3181bb
feat: Make a top-level influxdb_iox feature for kafka functionality
...
Thread the feature through router and server to the write buffer crate.
Move an end-to-end test that uses Kafka behind the feature flag.
2021-12-09 08:49:34 -05:00
dependabot[bot]
Raphael Taylor-Davies
Marco Neumann
Andrew Lamb
Paul Dix
Carol (Nichols || Goulding)
kodiakhq[bot]
Nga Tran
Andrew Lamb
Carol (Nichols || Goulding)