26fd5273f0
feat: static database configuration ( #2436 ) ( #3732 )
...
* feat: static database configuration (#2436 )
* chore: fmt
* feat: don't base64 encode UUIDs in ServerConfigFile
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 19:42:49 +00:00
85aa019f50
feat: Turn protobuf predicates into predicate::Predicate ( #3707 )
...
* feat: Turn protobuf predicates into predicate::Predicate
* fix: Take buf lint's suggestions
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 17:56:56 +00:00
d39cf0158b
chore: update gitops adapter k8s deps (via dependabot) ( #3736 )
...
* chore: update gitops adapter k8s deps (via dependabot)
* chore: workspace hack updates
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 17:47:15 +00:00
d1c71ba5d8
feat: predicate pushdown for Ingester's QueryableBatch ( #3728 )
...
* feat: predicate pushdown for Ingester's QueryableBatch
* chore: comment cleanup
* chore: Apply suggestions from code review
Co-authored-by: Edd Robinson <me@edd.io>
* refactor: address review comments
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 17:28:52 +00:00
cf5a5b77cb
fix: use encoded data size estimation instead of mutable batch ( #3734 )
...
For sparse data the PB-encoded data (our Kafka wire format) is way
smaller than the MutableBatch (up to a factor 20). So lets use this one
to estimate the size during batching.
2022-02-14 16:58:38 +00:00
c79050254f
refactor: traitify database configuration ( #2436 ) ( #3730 )
...
* refactor: traitify database configuration (#2436 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-13 09:26:44 +00:00
b77d43fc22
chore: Update datafusion ( #3727 )
2022-02-11 21:41:36 +00:00
866777ecd2
feat: static router configuration ( #2436 ) ( #3725 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-11 14:09:37 +00:00
5aada6beb8
fix: hard-code prod kafka config ( #3724 )
...
Prod has a larger max msg. size for Kafka (10MB instead of 1MB), but
currently we're unable to wire all the write buffer configs through. As
a quick fix lets hard code the config. This however breaks the write
buffer when running under default Kafka (1MB), so we should reverse this
(tracked under #3723 ).
2022-02-11 11:39:01 +00:00
0cc34b2a00
feat: server and database state gauge metrics ( #3722 )
2022-02-11 10:49:28 +00:00
4e3f66ed07
feat: CLI and gRPC APIs for shutting down and restarting databases ( #3720 )
...
* feat: allow catalog wipe and rebuild whilst shutdown
* feat: CLI and gRPC APIs for shutting down and restarting databases
* feat: add ability to skip replay on restart
* fix: test_wipe_persisted_catalog_error_db_exists
* fix: wipe_preserved_catalog
2022-02-11 10:14:43 +00:00
b7a4ef46dd
feat: preempt init shutdown ( #3368 ) ( #3717 )
2022-02-10 20:39:51 +00:00
910f381355
refactor: require UUID to create Database ( #3715 )
...
* refactor: require UUID to create Database
* chore: review feedback
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 20:04:06 +00:00
d3bd03e37a
feat: Support Projection Pushdown for a QueryableBatch ( #3712 )
...
* feat: projection pushdown for QueryableBatch
* chore: clean up and remove unwrap
* fix: Add Sync to a Snafu source to have the code compile
* chore: cleanup and add comments for tests
* refactor: Add tests for scanning non existing columns and fix related bugs
* chore: modify comment to trigger auto check in github work
2022-02-10 19:29:21 +00:00
b1190262b7
feat: restartable `Database` ( #3368 ) ( #3711 )
...
* feat: restartable `Database` (#3368 )
* chore: fmt
* fix: wipe_preserved_catalog
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 18:32:05 +00:00
6c3b208438
chore(deps): bump cache_loader_async from 0.1.2 to 0.2.0 ( #3665 )
...
* chore(deps): bump cache_loader_async from 0.1.2 to 0.2.0
Bumps [cache_loader_async](https://github.com/ZeroTwo-Bot/cache-loader-async-rs ) from 0.1.2 to 0.2.0.
- [Release notes](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/releases )
- [Changelog](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/commits/v0.2.0 )
---
updated-dependencies:
- dependency-name: cache_loader_async
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: breaking changes from `cache_loader_async`
* fix: Update workspace hack: tokio full is no longer needed
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
2022-02-10 17:01:11 +00:00
889804e46a
feat: avoid copying schemas while simplifying exprs ( #3620 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 15:07:00 +00:00
54890525da
feat: gitops adapter ci build ( #3692 )
...
* chore: remove references to perf_image in CI
* chore: adding gitops adapter image build in CI
* chore: gitops adapter bin now same as dir & package so docker build works
* fix: circle config package change after renaming gitops adapter package
2022-02-10 11:23:06 +00:00
ad60dc6949
chore(deps): bump httparse from 1.5.1 to 1.6.0 ( #3708 )
...
Bumps [httparse](https://github.com/seanmonstar/httparse ) from 1.5.1 to 1.6.0.
- [Release notes](https://github.com/seanmonstar/httparse/releases )
- [Commits](https://github.com/seanmonstar/httparse/compare/v1.5.1...v1.6.0 )
---
updated-dependencies:
- dependency-name: httparse
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 09:54:42 +00:00
f23574bc5f
chore(deps): bump futures from 0.3.19 to 0.3.21 ( #3706 )
...
Bumps [futures](https://github.com/rust-lang/futures-rs ) from 0.3.19 to 0.3.21.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.19...0.3.21 )
---
updated-dependencies:
- dependency-name: futures
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 09:19:19 +00:00
701633f1e6
chore: improve semantic check ( #3694 )
...
- bash context
- curl retry and fail with grace
- jq syntax improvement
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-09 22:44:06 +00:00
6a002f09e1
chore(deps): bump clap from 3.0.13 to 3.0.14 ( #3698 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.0.13 to 3.0.14.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.0.13...v3.0.14 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-09 21:21:24 +00:00
85d44552fa
chore(deps): bump cloud-storage from 0.10.3 to 0.11.0 ( #3697 )
...
Bumps [cloud-storage](https://github.com/ThouCheese/cloud-storage-rs ) from 0.10.3 to 0.11.0.
- [Release notes](https://github.com/ThouCheese/cloud-storage-rs/releases )
- [Changelog](https://github.com/ThouCheese/cloud-storage-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ThouCheese/cloud-storage-rs/commits )
---
updated-dependencies:
- dependency-name: cloud-storage
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-02-09 20:03:44 +00:00
d9f331ba2a
chore: update datafusion, stop repartitioning so aggressively ( #3633 )
...
* chore: update datafusion
* fix: Update to use new datafusion api
* chore: update expected plans
* fix: support zero output partitions
* fix: update test
* fix: Update for new DataFusion API
* fix: newly added system table
* fix: update cargo lock
2022-02-09 19:53:41 +00:00
803296e8cf
chore(deps): bump sqlparser from 0.13.0 to 0.14.0 ( #3696 )
...
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs ) from 0.13.0 to 0.14.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases )
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md )
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.13.0...v0.14.0 )
---
updated-dependencies:
- dependency-name: sqlparser
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-02-09 19:40:03 +00:00
73828323ac
feat: Ingester Flight gRPC API ( #3623 )
...
* feat: Add a way to run ingester with an in-memory catalog from the CLI
If you set the --catalog-dsn string to "mem", rather than using that as
a Postgres connection URL, create an in-memory catalog.
Planning on using this in tests, so not documenting.
* fix: Set default topic to the same value as SHARED_KAFKA_TOPIC
Namely, both should use an underscore. I don't think there's a way to
directly share these values between a constant and an annotation.
* feat: Add a flight API (handshake only) to ingester
* fix: Create partitions if using file-based write buffer
* fix: Change the server fixture to handle ingester server type
For now, the ingester doesn't implement the deployment API. Not sure if
it should or not.
* feat: Start implementing ingester do_get, namely decoding the query
Skip serialization of the predicate for the moment.
* refactor: Rename ingest protos to ingester to match crate name
* refactor: Rename QueryResults to QueryData
* feat: Move ingester flight client to new querier crate
* fix: Off by one error, different starting indexes in sequencers
* fix: Create new CLI argument to pick the catalog type
* fix: Create a CLI option to set the number of topics to auto-create in the write buffer
* fix: Check the arrow flight service's health to tell that the ingester gRPC is up
* fix: Set postgres as the default catalog type
* fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
0459e18aa0
feat: install cargo hakari in ci image ( #3691 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-09 17:17:18 +00:00
c31739de2c
refactor: simplify Database retry backoff logic ( #3368 ) ( #3689 )
2022-02-09 17:06:32 +00:00
092268fc87
chore: remove references to perf_image in CI ( #3673 )
...
* chore: remove references to perf_image in CI
* chore: typos in circle config comments
* chore: restored ability to build branch using parameter; documented it
* chore: fixed indentation faux-pas in circle config
* docs: clarified build-triggering instructions in circle config comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-09 16:51:23 +00:00
92f6413b3e
Merge pull request #3688 from influxdata/er/feat/storage_read_aggregate
...
feat: add read_window_aggregate storage sub-command
2022-02-09 14:05:22 +00:00
2334e779eb
feat: implement read_window_aggregate sub-command
2022-02-09 12:32:48 +00:00
0774e1d328
feat: add read_window_aggregate request builder
2022-02-09 12:32:48 +00:00
4bddab56e2
feat: create new sequencers in ingester on demand ( #3671 )
...
There is no need to introduce yet another admin action to do that. If
the sequencer does not exist yet, we can just create it and set the
`min_unpersisted_sequence_number` to 0 (which is done be `create_or_get`).
2022-02-09 12:26:30 +00:00
70881270c6
fix: increase default `producer_max_batch_size` ( #3686 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-09 12:09:58 +00:00
85004831a3
refactor: extract predicate sql tests out of rust ( #3683 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-09 11:24:34 +00:00
49edfebdf6
refactor: Complete move of test scenarios into library.rs ( #3682 )
...
Co-authored-by: Edd Robinson <me@edd.io>
2022-02-09 10:46:00 +00:00
a8c805e6ed
Merge pull request #3684 from influxdata/er/feat/storage_time_only
...
feat: add timed/quiet output for storage CLI
2022-02-08 21:41:42 +00:00
dfa6fd8579
feat: add quiet option to storage
2022-02-08 21:27:29 +00:00
11855a5eff
feat: add format flag
2022-02-08 21:15:07 +00:00
c175ccd1b4
feat: make stop/stop/predicate global ( #3681 )
2022-02-08 20:06:47 +00:00
c21119a9c6
Merge pull request #3676 from influxdata/crepererum/wb_tracing_fixes
...
fix: write buffer traces
2022-02-08 16:54:54 +00:00
35f60945e1
Merge branch 'main' into crepererum/wb_tracing_fixes
2022-02-08 16:44:30 +00:00
7d11485c77
chore(deps): bump tracing from 0.1.29 to 0.1.30 ( #3677 )
...
Bumps [tracing](https://github.com/tokio-rs/tracing ) from 0.1.29 to 0.1.30.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.29...tracing-0.1.30 )
---
updated-dependencies:
- dependency-name: tracing
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 16:44:16 +00:00
6bc8c3d471
chore: semantic checks allow Merge commits ( #3678 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 16:32:58 +00:00
8ae442497b
Merge pull request #3644 from influxdata/dom/sharded-cache
...
perf(router2): sharded namespace cache
2022-02-08 16:20:14 +00:00
ace76cef14
Merge branch 'main' into dom/sharded-cache
2022-02-08 16:09:48 +00:00
c18ad4ac97
feat: special case max timestamp range for table_names and field_columns ( #3642 )
2022-02-08 16:09:36 +00:00
a9fe3362bd
chore: add semantic commit check as GH action ( #3638 )
...
* chore: add semantic commit check as GH action
* chore: remove spaces
* chore: indentation
* chore: yamllint config compatible with GH actions 'on' key
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 15:44:09 +00:00
ca331503a5
feat: add WriteBufferErrorKind ( #3664 )
...
* feat: add WriteBufferErrorKind
* fix: test_offset_after_broken_message
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 15:34:05 +00:00
59b2141c0b
feat: Add lifecycle manager to ingester ( #3645 )
...
This adds the lifecycle manager to the ingester. It will trigger based on a threshold for max partition size or age or based on keeping total memory under a certain threshold.
It defines a new interface for a persister, which is stubbed out for IngesterData. I'm not sure yet how persistence errors should be handled. The assumption here is that the persister continues to retry persistence forever until it succeeds.
There is one scenario I can think of that may cause this lifecycle manager problems. If a single partition is very high throughput, it could cause things to back up as persistence is not parallelized within a single partition. Any given partition can currently only run one persistence operation at a time. We can address this later.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 15:23:40 +00:00
Raphael Taylor-Davies
Carol (Nichols || Goulding)
Luke Bond
Nga Tran
Marco Neumann
Andrew Lamb
dependabot[bot]
Jacob Marble
kodiakhq[bot]
Edd Robinson
Paul Dix