37c65fc24f
fix: Use a string literal with unreachable ( #3768 )
...
This should have been taken care of with the update to Rust 2021
Edition, but rustc had a bug [1]. The bug will be fixed in Rust 1.59, which
means this code will stop compiling.
[1] https://github.com/rust-lang/rust/pull/93179
2022-02-16 16:44:48 +00:00
a3e15ade62
Merge pull request #3746 from influxdata/dom/mb-partitioning
...
feat(router2): MutableBatch partitioning
2022-02-16 11:39:28 +00:00
d0965bb0b2
Merge branch 'main' into dom/mb-partitioning
2022-02-16 11:30:42 +00:00
4655d4b4f1
docs: Improve iox_catalog testing docs ( #3760 )
...
* docs: Improve iox_catalog testing docs
* fix: Update iox_catalog/README.md
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2022-02-16 10:23:53 +00:00
ea814e9aa4
feat: API and steps to prepare data to send back to the Querier per its request ( #3756 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-16 02:45:58 +00:00
f542045485
feat: wire up persistence in ingester ( #3685 )
...
This adds persistence into the ingester with a lifecycle manager. The persist operation must still be updated to keep track of the min_unpersisted_sequence_number for each sequencer.
2022-02-16 00:13:40 +00:00
5be546173a
Merge pull request #3751 from influxdata/er/logging
...
refactor: influxRPC operation logging
2022-02-15 20:18:11 +00:00
7c8ea53d03
Merge branch 'main' into er/logging
2022-02-15 20:09:36 +00:00
4a35547af4
chore: The k8s-iox promotion is no longer used and can be removed ( #3749 )
2022-02-15 17:20:27 +00:00
a66e29e5b3
chore: port sqlx-hotswap-pool over from conductor ( #3750 )
...
* chore: port sqlx-hotswap-pool over from conductor
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
* chore: workspace hack fixes
* fix: unique schema per test db connection
* fix: adjust search path in catalog pg tests to see if it fixes test schema issue
* fix: actually fixed sqlx hotswap pool test
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-15 16:18:36 +00:00
0b3f76462d
feat: build Query Plan that queries QueryableBatch with filters ( #3742 )
...
* feat: initial implementaion the Query Plan that query QueryableBatch with filters
* fix: read_filter of QueryableBatch should provide the shema of the columns/projection it needs
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* chore: address review comment
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-02-15 16:06:26 +00:00
7ac9e216c4
refactor: use same log message
2022-02-15 14:36:55 +00:00
8a5ea29190
refactor: add measurement to log
2022-02-15 14:31:26 +00:00
44ee0166a0
fix: start Kafka write buffer stream at "earliest" offset, not at "0" ( #3748 )
2022-02-15 13:36:59 +00:00
9e7a27b344
fix: default Kafka topic name is `iox-shared` ( #3747 )
...
Do NOT use underscores in the Kafka topic because this is not supported
by Kafka. This was initially fixed by #3555 but reverted by #3623 .
2022-02-15 12:34:46 +00:00
a30803e692
chore: Update datafusion, update `arrow`/`parquet`/`arrow-flight` to 9.0 ( #3733 )
...
* chore: Update datafusion
* chore: Update arrow
* fix: missing updates
* chore: Update cargo.lock
* fix: update for smaller parquet size
* fix: update test for smaller parquet files
* test: ensure parquet_file tests write multiple row groups
* fix: update callsite
* fix: Update for tests
* fix: harkari
* fix: use IoxObjectStore::existing
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-15 12:10:24 +00:00
e055800039
refactor: enable Partitioner in request pipeline
...
Adds the Partitioner DML handler into the handler stack, modifying the
input types of down-stream handlers to accept the partitioned data.
2022-02-15 11:34:33 +00:00
89105ccfab
chore(deps): Bump tokio-util from 0.6.9 to 0.7.0 ( #3743 )
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.6.9 to 0.7.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/commits )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-15 11:33:41 +00:00
c64e9f0d40
refactor: namespace auto-creator generic input
...
Changes the NamespaceAutocreation handler to be generic over any
WriteInput.
This allows the NamespaceAutocreation layer to be placed anywhere in the
handler stack, without needing a prior transformation or specific write
type.
2022-02-15 11:29:33 +00:00
92218ce8aa
feat: write partitioner
...
Implements a write partitioning DML handler that splits per-table
MutableBatch instances into per-partition, per-table MutableBatch and
concurrently calls the inner DML handler with each.
2022-02-15 11:29:32 +00:00
5c254339fa
test: MockDmlHandler generic over write input
...
Allow the MockDmlHandler to capture any input type given to the write()
method. This lets us reuse the mock across all handler implementations,
regardless of their expected write input type.
2022-02-15 11:27:16 +00:00
e99922d518
refactor: parametrise DML handler input type
...
Allow a DML handler to specify the write input type on which it
operates.
This allows us to construct a write handler pipeline that transforms the
request as it passes through the various handlers. We'll use this to
implement a handler that annotates a normal set of table writes with the
partition key, modifying downstream handlers to expect this annotated
input.
2022-02-15 11:23:45 +00:00
4db27eec68
fix: defined behaviour when seeking to an unknown sequence number ( #3745 )
...
* chore: upgrade rskafka
* refactor: less cloning
* fix: defined behaviour when seeking to an unknown sequence number
The new, defined behavior is: "return an error once and then end the
stream".
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: Edd Robinson <me@edd.io>
2022-02-15 11:08:01 +00:00
1cd51b6071
chore(deps): Bump hyper from 0.14.16 to 0.14.17 ( #3744 )
...
Bumps [hyper](https://github.com/hyperium/hyper ) from 0.14.16 to 0.14.17.
- [Release notes](https://github.com/hyperium/hyper/releases )
- [Changelog](https://github.com/hyperium/hyper/blob/master/CHANGELOG.md )
- [Commits](https://github.com/hyperium/hyper/compare/v0.14.16...v0.14.17 )
---
updated-dependencies:
- dependency-name: hyper
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-15 10:23:46 +00:00
c6e374a025
feat: allow catalog access w/o a transaction ( #3735 )
...
* feat: allow catalog access w/o a transaction
Now the caller has the full control if they want to use a transaction or
not.
* fix: remove non-transaction-safe `create_many`
* fix: remove unnecessary transactions
2022-02-15 10:15:36 +00:00
2626e4891d
chore(deps): Bump tracing-subscriber from 0.3.7 to 0.3.8 ( #3741 )
...
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing ) from 0.3.7 to 0.3.8.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.7...tracing-subscriber-0.3.8 )
---
updated-dependencies:
- dependency-name: tracing-subscriber
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 20:51:21 +00:00
60a7f87645
chore(deps): Bump serde_json from 1.0.78 to 1.0.79 ( #3739 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.78 to 1.0.79.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.78...v1.0.79 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 20:42:54 +00:00
26fd5273f0
feat: static database configuration ( #2436 ) ( #3732 )
...
* feat: static database configuration (#2436 )
* chore: fmt
* feat: don't base64 encode UUIDs in ServerConfigFile
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 19:42:49 +00:00
85aa019f50
feat: Turn protobuf predicates into predicate::Predicate ( #3707 )
...
* feat: Turn protobuf predicates into predicate::Predicate
* fix: Take buf lint's suggestions
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 17:56:56 +00:00
d39cf0158b
chore: update gitops adapter k8s deps (via dependabot) ( #3736 )
...
* chore: update gitops adapter k8s deps (via dependabot)
* chore: workspace hack updates
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 17:47:15 +00:00
d1c71ba5d8
feat: predicate pushdown for Ingester's QueryableBatch ( #3728 )
...
* feat: predicate pushdown for Ingester's QueryableBatch
* chore: comment cleanup
* chore: Apply suggestions from code review
Co-authored-by: Edd Robinson <me@edd.io>
* refactor: address review comments
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 17:28:52 +00:00
cf5a5b77cb
fix: use encoded data size estimation instead of mutable batch ( #3734 )
...
For sparse data the PB-encoded data (our Kafka wire format) is way
smaller than the MutableBatch (up to a factor 20). So lets use this one
to estimate the size during batching.
2022-02-14 16:58:38 +00:00
c79050254f
refactor: traitify database configuration ( #2436 ) ( #3730 )
...
* refactor: traitify database configuration (#2436 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-13 09:26:44 +00:00
b77d43fc22
chore: Update datafusion ( #3727 )
2022-02-11 21:41:36 +00:00
866777ecd2
feat: static router configuration ( #2436 ) ( #3725 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-11 14:09:37 +00:00
5aada6beb8
fix: hard-code prod kafka config ( #3724 )
...
Prod has a larger max msg. size for Kafka (10MB instead of 1MB), but
currently we're unable to wire all the write buffer configs through. As
a quick fix lets hard code the config. This however breaks the write
buffer when running under default Kafka (1MB), so we should reverse this
(tracked under #3723 ).
2022-02-11 11:39:01 +00:00
0cc34b2a00
feat: server and database state gauge metrics ( #3722 )
2022-02-11 10:49:28 +00:00
4e3f66ed07
feat: CLI and gRPC APIs for shutting down and restarting databases ( #3720 )
...
* feat: allow catalog wipe and rebuild whilst shutdown
* feat: CLI and gRPC APIs for shutting down and restarting databases
* feat: add ability to skip replay on restart
* fix: test_wipe_persisted_catalog_error_db_exists
* fix: wipe_preserved_catalog
2022-02-11 10:14:43 +00:00
b7a4ef46dd
feat: preempt init shutdown ( #3368 ) ( #3717 )
2022-02-10 20:39:51 +00:00
910f381355
refactor: require UUID to create Database ( #3715 )
...
* refactor: require UUID to create Database
* chore: review feedback
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 20:04:06 +00:00
d3bd03e37a
feat: Support Projection Pushdown for a QueryableBatch ( #3712 )
...
* feat: projection pushdown for QueryableBatch
* chore: clean up and remove unwrap
* fix: Add Sync to a Snafu source to have the code compile
* chore: cleanup and add comments for tests
* refactor: Add tests for scanning non existing columns and fix related bugs
* chore: modify comment to trigger auto check in github work
2022-02-10 19:29:21 +00:00
b1190262b7
feat: restartable `Database` ( #3368 ) ( #3711 )
...
* feat: restartable `Database` (#3368 )
* chore: fmt
* fix: wipe_preserved_catalog
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 18:32:05 +00:00
6c3b208438
chore(deps): bump cache_loader_async from 0.1.2 to 0.2.0 ( #3665 )
...
* chore(deps): bump cache_loader_async from 0.1.2 to 0.2.0
Bumps [cache_loader_async](https://github.com/ZeroTwo-Bot/cache-loader-async-rs ) from 0.1.2 to 0.2.0.
- [Release notes](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/releases )
- [Changelog](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/commits/v0.2.0 )
---
updated-dependencies:
- dependency-name: cache_loader_async
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: breaking changes from `cache_loader_async`
* fix: Update workspace hack: tokio full is no longer needed
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
2022-02-10 17:01:11 +00:00
889804e46a
feat: avoid copying schemas while simplifying exprs ( #3620 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 15:07:00 +00:00
54890525da
feat: gitops adapter ci build ( #3692 )
...
* chore: remove references to perf_image in CI
* chore: adding gitops adapter image build in CI
* chore: gitops adapter bin now same as dir & package so docker build works
* fix: circle config package change after renaming gitops adapter package
2022-02-10 11:23:06 +00:00
ad60dc6949
chore(deps): bump httparse from 1.5.1 to 1.6.0 ( #3708 )
...
Bumps [httparse](https://github.com/seanmonstar/httparse ) from 1.5.1 to 1.6.0.
- [Release notes](https://github.com/seanmonstar/httparse/releases )
- [Commits](https://github.com/seanmonstar/httparse/compare/v1.5.1...v1.6.0 )
---
updated-dependencies:
- dependency-name: httparse
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 09:54:42 +00:00
f23574bc5f
chore(deps): bump futures from 0.3.19 to 0.3.21 ( #3706 )
...
Bumps [futures](https://github.com/rust-lang/futures-rs ) from 0.3.19 to 0.3.21.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.19...0.3.21 )
---
updated-dependencies:
- dependency-name: futures
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 09:19:19 +00:00
701633f1e6
chore: improve semantic check ( #3694 )
...
- bash context
- curl retry and fail with grace
- jq syntax improvement
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-09 22:44:06 +00:00
6a002f09e1
chore(deps): bump clap from 3.0.13 to 3.0.14 ( #3698 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.0.13 to 3.0.14.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.0.13...v3.0.14 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-09 21:21:24 +00:00
85d44552fa
chore(deps): bump cloud-storage from 0.10.3 to 0.11.0 ( #3697 )
...
Bumps [cloud-storage](https://github.com/ThouCheese/cloud-storage-rs ) from 0.10.3 to 0.11.0.
- [Release notes](https://github.com/ThouCheese/cloud-storage-rs/releases )
- [Changelog](https://github.com/ThouCheese/cloud-storage-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ThouCheese/cloud-storage-rs/commits )
---
updated-dependencies:
- dependency-name: cloud-storage
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-02-09 20:03:44 +00:00
Carol (Nichols || Goulding)
kodiakhq[bot]
Marko Mikulicic
Nga Tran
Paul Dix
Edd Robinson
jaymebrd
Luke Bond
Marco Neumann
Andrew Lamb
Dom Dwyer
dependabot[bot]
Raphael Taylor-Davies
Jacob Marble