4db27eec68
fix: defined behaviour when seeking to an unknown sequence number ( #3745 )
...
* chore: upgrade rskafka
* refactor: less cloning
* fix: defined behaviour when seeking to an unknown sequence number
The new, defined behavior is: "return an error once and then end the
stream".
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: Edd Robinson <me@edd.io>
2022-02-15 11:08:01 +00:00
1cd51b6071
chore(deps): Bump hyper from 0.14.16 to 0.14.17 ( #3744 )
...
Bumps [hyper](https://github.com/hyperium/hyper ) from 0.14.16 to 0.14.17.
- [Release notes](https://github.com/hyperium/hyper/releases )
- [Changelog](https://github.com/hyperium/hyper/blob/master/CHANGELOG.md )
- [Commits](https://github.com/hyperium/hyper/compare/v0.14.16...v0.14.17 )
---
updated-dependencies:
- dependency-name: hyper
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-15 10:23:46 +00:00
c6e374a025
feat: allow catalog access w/o a transaction ( #3735 )
...
* feat: allow catalog access w/o a transaction
Now the caller has the full control if they want to use a transaction or
not.
* fix: remove non-transaction-safe `create_many`
* fix: remove unnecessary transactions
2022-02-15 10:15:36 +00:00
2626e4891d
chore(deps): Bump tracing-subscriber from 0.3.7 to 0.3.8 ( #3741 )
...
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing ) from 0.3.7 to 0.3.8.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.7...tracing-subscriber-0.3.8 )
---
updated-dependencies:
- dependency-name: tracing-subscriber
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 20:51:21 +00:00
60a7f87645
chore(deps): Bump serde_json from 1.0.78 to 1.0.79 ( #3739 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.78 to 1.0.79.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.78...v1.0.79 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 20:42:54 +00:00
26fd5273f0
feat: static database configuration ( #2436 ) ( #3732 )
...
* feat: static database configuration (#2436 )
* chore: fmt
* feat: don't base64 encode UUIDs in ServerConfigFile
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 19:42:49 +00:00
85aa019f50
feat: Turn protobuf predicates into predicate::Predicate ( #3707 )
...
* feat: Turn protobuf predicates into predicate::Predicate
* fix: Take buf lint's suggestions
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 17:56:56 +00:00
d39cf0158b
chore: update gitops adapter k8s deps (via dependabot) ( #3736 )
...
* chore: update gitops adapter k8s deps (via dependabot)
* chore: workspace hack updates
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 17:47:15 +00:00
d1c71ba5d8
feat: predicate pushdown for Ingester's QueryableBatch ( #3728 )
...
* feat: predicate pushdown for Ingester's QueryableBatch
* chore: comment cleanup
* chore: Apply suggestions from code review
Co-authored-by: Edd Robinson <me@edd.io>
* refactor: address review comments
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 17:28:52 +00:00
cf5a5b77cb
fix: use encoded data size estimation instead of mutable batch ( #3734 )
...
For sparse data the PB-encoded data (our Kafka wire format) is way
smaller than the MutableBatch (up to a factor 20). So lets use this one
to estimate the size during batching.
2022-02-14 16:58:38 +00:00
c79050254f
refactor: traitify database configuration ( #2436 ) ( #3730 )
...
* refactor: traitify database configuration (#2436 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-13 09:26:44 +00:00
b77d43fc22
chore: Update datafusion ( #3727 )
2022-02-11 21:41:36 +00:00
866777ecd2
feat: static router configuration ( #2436 ) ( #3725 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-11 14:09:37 +00:00
5aada6beb8
fix: hard-code prod kafka config ( #3724 )
...
Prod has a larger max msg. size for Kafka (10MB instead of 1MB), but
currently we're unable to wire all the write buffer configs through. As
a quick fix lets hard code the config. This however breaks the write
buffer when running under default Kafka (1MB), so we should reverse this
(tracked under #3723 ).
2022-02-11 11:39:01 +00:00
0cc34b2a00
feat: server and database state gauge metrics ( #3722 )
2022-02-11 10:49:28 +00:00
4e3f66ed07
feat: CLI and gRPC APIs for shutting down and restarting databases ( #3720 )
...
* feat: allow catalog wipe and rebuild whilst shutdown
* feat: CLI and gRPC APIs for shutting down and restarting databases
* feat: add ability to skip replay on restart
* fix: test_wipe_persisted_catalog_error_db_exists
* fix: wipe_preserved_catalog
2022-02-11 10:14:43 +00:00
b7a4ef46dd
feat: preempt init shutdown ( #3368 ) ( #3717 )
2022-02-10 20:39:51 +00:00
910f381355
refactor: require UUID to create Database ( #3715 )
...
* refactor: require UUID to create Database
* chore: review feedback
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 20:04:06 +00:00
d3bd03e37a
feat: Support Projection Pushdown for a QueryableBatch ( #3712 )
...
* feat: projection pushdown for QueryableBatch
* chore: clean up and remove unwrap
* fix: Add Sync to a Snafu source to have the code compile
* chore: cleanup and add comments for tests
* refactor: Add tests for scanning non existing columns and fix related bugs
* chore: modify comment to trigger auto check in github work
2022-02-10 19:29:21 +00:00
b1190262b7
feat: restartable `Database` ( #3368 ) ( #3711 )
...
* feat: restartable `Database` (#3368 )
* chore: fmt
* fix: wipe_preserved_catalog
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 18:32:05 +00:00
6c3b208438
chore(deps): bump cache_loader_async from 0.1.2 to 0.2.0 ( #3665 )
...
* chore(deps): bump cache_loader_async from 0.1.2 to 0.2.0
Bumps [cache_loader_async](https://github.com/ZeroTwo-Bot/cache-loader-async-rs ) from 0.1.2 to 0.2.0.
- [Release notes](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/releases )
- [Changelog](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/commits/v0.2.0 )
---
updated-dependencies:
- dependency-name: cache_loader_async
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: breaking changes from `cache_loader_async`
* fix: Update workspace hack: tokio full is no longer needed
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
2022-02-10 17:01:11 +00:00
889804e46a
feat: avoid copying schemas while simplifying exprs ( #3620 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 15:07:00 +00:00
54890525da
feat: gitops adapter ci build ( #3692 )
...
* chore: remove references to perf_image in CI
* chore: adding gitops adapter image build in CI
* chore: gitops adapter bin now same as dir & package so docker build works
* fix: circle config package change after renaming gitops adapter package
2022-02-10 11:23:06 +00:00
ad60dc6949
chore(deps): bump httparse from 1.5.1 to 1.6.0 ( #3708 )
...
Bumps [httparse](https://github.com/seanmonstar/httparse ) from 1.5.1 to 1.6.0.
- [Release notes](https://github.com/seanmonstar/httparse/releases )
- [Commits](https://github.com/seanmonstar/httparse/compare/v1.5.1...v1.6.0 )
---
updated-dependencies:
- dependency-name: httparse
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 09:54:42 +00:00
f23574bc5f
chore(deps): bump futures from 0.3.19 to 0.3.21 ( #3706 )
...
Bumps [futures](https://github.com/rust-lang/futures-rs ) from 0.3.19 to 0.3.21.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.19...0.3.21 )
---
updated-dependencies:
- dependency-name: futures
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 09:19:19 +00:00
701633f1e6
chore: improve semantic check ( #3694 )
...
- bash context
- curl retry and fail with grace
- jq syntax improvement
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-09 22:44:06 +00:00
6a002f09e1
chore(deps): bump clap from 3.0.13 to 3.0.14 ( #3698 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.0.13 to 3.0.14.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.0.13...v3.0.14 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-09 21:21:24 +00:00
85d44552fa
chore(deps): bump cloud-storage from 0.10.3 to 0.11.0 ( #3697 )
...
Bumps [cloud-storage](https://github.com/ThouCheese/cloud-storage-rs ) from 0.10.3 to 0.11.0.
- [Release notes](https://github.com/ThouCheese/cloud-storage-rs/releases )
- [Changelog](https://github.com/ThouCheese/cloud-storage-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ThouCheese/cloud-storage-rs/commits )
---
updated-dependencies:
- dependency-name: cloud-storage
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-02-09 20:03:44 +00:00
d9f331ba2a
chore: update datafusion, stop repartitioning so aggressively ( #3633 )
...
* chore: update datafusion
* fix: Update to use new datafusion api
* chore: update expected plans
* fix: support zero output partitions
* fix: update test
* fix: Update for new DataFusion API
* fix: newly added system table
* fix: update cargo lock
2022-02-09 19:53:41 +00:00
803296e8cf
chore(deps): bump sqlparser from 0.13.0 to 0.14.0 ( #3696 )
...
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs ) from 0.13.0 to 0.14.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases )
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md )
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.13.0...v0.14.0 )
---
updated-dependencies:
- dependency-name: sqlparser
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-02-09 19:40:03 +00:00
73828323ac
feat: Ingester Flight gRPC API ( #3623 )
...
* feat: Add a way to run ingester with an in-memory catalog from the CLI
If you set the --catalog-dsn string to "mem", rather than using that as
a Postgres connection URL, create an in-memory catalog.
Planning on using this in tests, so not documenting.
* fix: Set default topic to the same value as SHARED_KAFKA_TOPIC
Namely, both should use an underscore. I don't think there's a way to
directly share these values between a constant and an annotation.
* feat: Add a flight API (handshake only) to ingester
* fix: Create partitions if using file-based write buffer
* fix: Change the server fixture to handle ingester server type
For now, the ingester doesn't implement the deployment API. Not sure if
it should or not.
* feat: Start implementing ingester do_get, namely decoding the query
Skip serialization of the predicate for the moment.
* refactor: Rename ingest protos to ingester to match crate name
* refactor: Rename QueryResults to QueryData
* feat: Move ingester flight client to new querier crate
* fix: Off by one error, different starting indexes in sequencers
* fix: Create new CLI argument to pick the catalog type
* fix: Create a CLI option to set the number of topics to auto-create in the write buffer
* fix: Check the arrow flight service's health to tell that the ingester gRPC is up
* fix: Set postgres as the default catalog type
* fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
0459e18aa0
feat: install cargo hakari in ci image ( #3691 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-09 17:17:18 +00:00
c31739de2c
refactor: simplify Database retry backoff logic ( #3368 ) ( #3689 )
2022-02-09 17:06:32 +00:00
092268fc87
chore: remove references to perf_image in CI ( #3673 )
...
* chore: remove references to perf_image in CI
* chore: typos in circle config comments
* chore: restored ability to build branch using parameter; documented it
* chore: fixed indentation faux-pas in circle config
* docs: clarified build-triggering instructions in circle config comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-09 16:51:23 +00:00
92f6413b3e
Merge pull request #3688 from influxdata/er/feat/storage_read_aggregate
...
feat: add read_window_aggregate storage sub-command
2022-02-09 14:05:22 +00:00
2334e779eb
feat: implement read_window_aggregate sub-command
2022-02-09 12:32:48 +00:00
0774e1d328
feat: add read_window_aggregate request builder
2022-02-09 12:32:48 +00:00
4bddab56e2
feat: create new sequencers in ingester on demand ( #3671 )
...
There is no need to introduce yet another admin action to do that. If
the sequencer does not exist yet, we can just create it and set the
`min_unpersisted_sequence_number` to 0 (which is done be `create_or_get`).
2022-02-09 12:26:30 +00:00
70881270c6
fix: increase default `producer_max_batch_size` ( #3686 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-09 12:09:58 +00:00
85004831a3
refactor: extract predicate sql tests out of rust ( #3683 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-09 11:24:34 +00:00
49edfebdf6
refactor: Complete move of test scenarios into library.rs ( #3682 )
...
Co-authored-by: Edd Robinson <me@edd.io>
2022-02-09 10:46:00 +00:00
a8c805e6ed
Merge pull request #3684 from influxdata/er/feat/storage_time_only
...
feat: add timed/quiet output for storage CLI
2022-02-08 21:41:42 +00:00
dfa6fd8579
feat: add quiet option to storage
2022-02-08 21:27:29 +00:00
11855a5eff
feat: add format flag
2022-02-08 21:15:07 +00:00
c175ccd1b4
feat: make stop/stop/predicate global ( #3681 )
2022-02-08 20:06:47 +00:00
c21119a9c6
Merge pull request #3676 from influxdata/crepererum/wb_tracing_fixes
...
fix: write buffer traces
2022-02-08 16:54:54 +00:00
35f60945e1
Merge branch 'main' into crepererum/wb_tracing_fixes
2022-02-08 16:44:30 +00:00
7d11485c77
chore(deps): bump tracing from 0.1.29 to 0.1.30 ( #3677 )
...
Bumps [tracing](https://github.com/tokio-rs/tracing ) from 0.1.29 to 0.1.30.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.29...tracing-0.1.30 )
---
updated-dependencies:
- dependency-name: tracing
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 16:44:16 +00:00
6bc8c3d471
chore: semantic checks allow Merge commits ( #3678 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 16:32:58 +00:00
8ae442497b
Merge pull request #3644 from influxdata/dom/sharded-cache
...
perf(router2): sharded namespace cache
2022-02-08 16:20:14 +00:00
Marco Neumann
dependabot[bot]
Raphael Taylor-Davies
Carol (Nichols || Goulding)
Luke Bond
Nga Tran
Andrew Lamb
Jacob Marble
kodiakhq[bot]
Edd Robinson