8574bb695c
chore(deps): Bump tikv-jemalloc-sys ( #3813 )
...
Bumps [tikv-jemalloc-sys](https://github.com/tikv/jemallocator ) from 0.4.2+5.2.1-patched.2 to 0.4.3+5.2.1-patched.2.
- [Release notes](https://github.com/tikv/jemallocator/releases )
- [Changelog](https://github.com/tikv/jemallocator/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tikv/jemallocator/commits )
---
updated-dependencies:
- dependency-name: tikv-jemalloc-sys
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:12:33 +00:00
9ff44ce90b
feat: sequencer metrics during replay ( #3806 )
...
* feat: sequencer metrics during replay
* feat: add replay metrics test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:01:27 +00:00
ae5712fab0
chore(deps): Bump kube-derive from 0.69.0 to 0.69.1 ( #3812 )
...
Bumps [kube-derive](https://github.com/kube-rs/kube-rs ) from 0.69.0 to 0.69.1.
- [Release notes](https://github.com/kube-rs/kube-rs/releases )
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.69.0...0.69.1 )
---
updated-dependencies:
- dependency-name: kube-derive
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 15:49:14 +00:00
63f35d0006
refactor: generic+reusable task registry ( #3811 )
...
* feat: introduce `AbstractTaskRegistry`
* feat: make `TaskRegistryWithHistory` generic
* feat: reusable task<>metric integration
* refactor: simplify `FAttributes`
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 15:35:29 +00:00
1dca869c3f
chore(deps): Bump sqlx from 0.5.10 to 0.5.11 ( #3799 )
...
Bumps [sqlx](https://github.com/launchbadge/sqlx ) from 0.5.10 to 0.5.11.
- [Release notes](https://github.com/launchbadge/sqlx/releases )
- [Changelog](https://github.com/launchbadge/sqlx/blob/master/CHANGELOG.md )
- [Commits](https://github.com/launchbadge/sqlx/compare/v0.5.10...v0.5.11 )
---
updated-dependencies:
- dependency-name: sqlx
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 15:14:36 +00:00
69480eaa5c
feat: remove remaining table-granularity metrics ( #3808 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 15:02:57 +00:00
65ab5213e5
chore(deps): Bump clap from 3.0.14 to 3.1.1 ( #3809 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.0.14 to 3.1.1.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.0.14...v3.1.1 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-22 14:51:53 +00:00
c2a01c2407
chore(deps): Bump comfy-table from 5.0.0 to 5.0.1 ( #3800 )
...
Bumps [comfy-table](https://github.com/nukesor/comfy-table ) from 5.0.0 to 5.0.1.
- [Release notes](https://github.com/nukesor/comfy-table/releases )
- [Changelog](https://github.com/Nukesor/comfy-table/blob/main/CHANGELOG.md )
- [Commits](https://github.com/nukesor/comfy-table/compare/v5.0.0...v5.0.1 )
---
updated-dependencies:
- dependency-name: comfy-table
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 14:36:52 +00:00
1960645055
feat: add wildcard support to persist partition CLI command ( #3790 )
...
* feat: add wildcard support to persist partition
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 14:21:06 +00:00
0229147909
feat: preserve catalog on error ( #1522 ) ( #3802 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 14:10:20 +00:00
454ccc7566
Merge pull request #3788 from influxdata/dom/router-handler-chain
...
refactor(router2): non-intrusive handler chain
2022-02-22 13:59:28 +00:00
326871f149
Merge branch 'main' into dom/router-handler-chain
2022-02-22 13:49:09 +00:00
3d8f0246c5
Merge pull request #3803 from influxdata/feat/new-compactor-command-and-crate
...
feat: adding compactor CLI command and crate
2022-02-22 11:27:41 +00:00
4ef704376f
Merge branch 'main' into feat/new-compactor-command-and-crate
2022-02-22 10:57:29 +00:00
0f012de70c
feat: adding compactor CLI command and crate
...
Closes : #3777
2022-02-21 12:24:09 +00:00
50aa39ff3f
Merge pull request #3789 from influxdata/dom/fix-catalog-migrations
...
fix: "catalog setup" migrations in non-public schema
2022-02-21 12:22:35 +00:00
4f0bd29a5b
build: run "catalog setup" in CI
...
Ensures the "catalog setup" command executes successfully by executing
it in CI.
2022-02-21 11:58:57 +00:00
2500dc3ac7
fix: add public to schema search path
...
The _sql_migrations table cannot be created by the `catalog setup`
bootstrap command because it is created before the migrations run, one
of which creates the iox_catalog namespace the catalog operates in.
This change allows the _sql_migrations table to be created in the public
schema to start the migration process, with everything else living in
the iox_catalog schema.
2022-02-21 11:58:55 +00:00
a16d4b80c4
chore(deps): Bump tracing from 0.1.30 to 0.1.31 ( #3798 )
...
Bumps [tracing](https://github.com/tokio-rs/tracing ) from 0.1.30 to 0.1.31.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.30...tracing-0.1.31 )
---
updated-dependencies:
- dependency-name: tracing
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 10:40:09 +00:00
df9b51fff3
docs: Add prerequisites. ( #3731 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 10:31:25 +00:00
474a4099e1
chore(deps): Bump kube from 0.69.0 to 0.69.1 ( #3796 )
...
Bumps [kube](https://github.com/kube-rs/kube-rs ) from 0.69.0 to 0.69.1.
- [Release notes](https://github.com/kube-rs/kube-rs/releases )
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.69.0...0.69.1 )
---
updated-dependencies:
- dependency-name: kube
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 10:22:58 +00:00
39c42678d7
feat: trigger persistence if over soft limit and no evictable chunks ( #3791 )
...
* feat: trigger persistence if over soft limit and no evictable chunks
* chore: fmt
* fix: avoid test_full_lifecycle exceeding soft limit
* fix: don't expect chunk to be unloaded
* feat: only trigger if no outstanding persist
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 10:14:27 +00:00
271daa5eac
chore(deps): Bump sqlx-core from 0.5.10 to 0.5.11 ( #3792 )
...
Bumps [sqlx-core](https://github.com/launchbadge/sqlx ) from 0.5.10 to 0.5.11.
- [Release notes](https://github.com/launchbadge/sqlx/releases )
- [Changelog](https://github.com/launchbadge/sqlx/blob/master/CHANGELOG.md )
- [Commits](https://github.com/launchbadge/sqlx/compare/v0.5.10...v0.5.11 )
---
updated-dependencies:
- dependency-name: sqlx-core
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 10:05:31 +00:00
9ad10a5a5a
chore(deps): Bump libc from 0.2.117 to 0.2.119 ( #3795 )
...
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.117 to 0.2.119.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.117...0.2.119 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 09:56:17 +00:00
09e69a57b2
chore(deps): Bump cc from 1.0.72 to 1.0.73 ( #3793 )
...
Bumps [cc](https://github.com/alexcrichton/cc-rs ) from 1.0.72 to 1.0.73.
- [Release notes](https://github.com/alexcrichton/cc-rs/releases )
- [Commits](https://github.com/alexcrichton/cc-rs/compare/1.0.72...1.0.73 )
---
updated-dependencies:
- dependency-name: cc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 09:47:47 +00:00
d6317a2ffd
chore(deps): Bump kube-runtime from 0.69.0 to 0.69.1 ( #3794 )
...
Bumps [kube-runtime](https://github.com/kube-rs/kube-rs ) from 0.69.0 to 0.69.1.
- [Release notes](https://github.com/kube-rs/kube-rs/releases )
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.69.0...0.69.1 )
---
updated-dependencies:
- dependency-name: kube-runtime
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-21 09:37:52 +00:00
1b9212540b
feat: Send IngesterQueryResponse data back as response of doGet Flight request ( #3772 )
...
* fix: Adjust fields of IngesterQueryResponse
* feat: Adjust IngestHandler query method to call prepare_data_to_querier
* feat: Send ingest query result data back through Flight doGet
* feat: Send delete predicates and max sequencer number in metadata
* fix: greater_than_sequence_number should be of type SequenceNumber
* fix: Remove DeletePredicates from IngesterQueryResponse
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-18 17:42:49 +00:00
7969ec682c
feat: catalog now supports dsn-file:// ( #3787 )
...
* feat: catalog now supports dsn-file://
* chore: rename fn in catalog PG hotswap code
2022-02-18 16:42:55 +00:00
497615d715
test: router handler stack integration test
...
Adds an integration test covering the router's HTTP handler stack.
Given a well-formed HTTP write, the test asserts:
* Write passes through the stack without error
* Response code sent to client
* Write buffer message is enqueued
* Catalog namespace record is created
* Metric handler is invoked and the hit is recorded
2022-02-18 14:39:03 +00:00
bb132b61ad
refactor: chain DML handlers
...
The router is composed of several DML handlers called in sequence in
order to construct the full request handling pipeline. Prior to this
commit, each handler nested the next handler it calls internally,
producing a nested call chain that resulted metrics (added in #3764 )
recording cumulative latency like this:
┌ ─
│ ┌───────────────┐
│ NS Creation │
│ └───────────────┘
│ ┌───────────────┐
│ │ │ Partitioner │
│ └───────────────┘
│ │ │
│ │
Cumulative │ │ │ ┌───────────────┐
Timings 1.5s 1s │ etc... │
│ │ │ └───────────────┘
│ │
│ │ │
│ ┌───────────────┐
│ │ │ Partitioner │
│ └───────────────┘
│ ┌───────────────┐
│ NS Creation │
│ └───────────────┘
└ ─
This meant it was hard to determine the latency of a single handler
without knowing (and subtracting the latency of) all the child handlers
it calls.
This commit replaces the intrusive nested handler call chain with an
external Chain combinator type to compose together individual handlers,
resulting in correct per-handler timings and simpler code/tests:
┌───────────────┐
│ NS Creation │
└───────────────┘
│
.5s ┌───────────────┐
└───────▶│ Partitioner │
└───────────────┘
│
1s ┌───────────────┐
└───▶│ etc... │
└───────────────┘
2022-02-18 14:19:53 +00:00
a686c2ab02
Merge pull request #3779 from influxdata/dom/catalog-spurious-writes
...
fix: catalog spurious writes
2022-02-18 12:42:27 +00:00
e105b3a84c
Merge branch 'main' into dom/catalog-spurious-writes
2022-02-18 12:33:05 +00:00
83cba3d2fb
feat: template static router config ( #3781 )
...
* feat: template static router config
* chore: lint and improved failure output
* chore: clarify docs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-18 10:53:10 +00:00
f54ef92b77
fix: supervise and shutdown ingester background tasks ( #3769 )
...
* fix: supervise and shutdown ingester background tasks
Closes #3761 .
Closes #3762 .
* docs: improve wording
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
* test: join/shutdown handling for ingester
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2022-02-18 09:35:29 +00:00
23b3942306
fix: compact persisting panics on single row ( #3784 )
2022-02-17 18:33:04 +00:00
568a15510e
fix: assert on conflicting tombstone creation
...
If two callers call create_or_get() for a tombstone, providing the same
(table_id, sequencer_id, sequence_number) triplet but a different
predicate / timestamps the catalog MUST NOT silently continue.
As this is unexpected, this behaviour causes a panic.
2022-02-17 16:29:55 +00:00
0cc5c979c6
fix: assert on conflicting partition creation
...
If two callers call create_or_get() for a partition, providing the same
partition key & table ID, but different sequence numbers the catalog
MUST NOT continue silently.
As this is unexpected, this behaviour causes a panic.
2022-02-17 16:29:55 +00:00
0796bd079b
test(iox_catalog): assert no spurious writes
...
This commit introduces test cases that ensure that for each of the
tombstone & partition repos:
* create_or_get() is idempotent
* create_or_get() does not silently drop conflicting writes
The former covers the expected use case: callers issuing potentially
multiple calls to create_or_get() with the same args, causing the
catalog impl to transparently turn the subsequent calls into NOPs.
The latter illustrates an issue where multiple calls to create_or_get()
with differing arguments is silently accepted by the catalog, causing
both callers to believe they have committed the (differing) details they
provided. This test is expected to pass but fails in this commit.
2022-02-17 16:29:55 +00:00
80a487b653
Merge pull request #3776 from influxdata/dom/isolated-catalog-tests
...
test: isolated catalog tests
2022-02-17 15:47:48 +00:00
74768db827
Merge branch 'main' into dom/isolated-catalog-tests
2022-02-17 15:39:29 +00:00
9588b43a90
fix: Make errors in rewriting return `Error` rather than a `panic` ( #3767 )
...
* test: add test for predicate errors
* fix: Return errors properly rather than panic
* fix: handle errors in influxrpc planner
* fix: appease clippy
* fix: tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-17 15:39:14 +00:00
9d25e5bac3
Merge pull request #3764 from influxdata/dom/router2-metrics
...
feat: router2 metrics
2022-02-17 15:25:16 +00:00
c89fa3701e
Merge branch 'main' into dom/router2-metrics
2022-02-17 15:16:39 +00:00
52fd2af851
refactor: DML handler metric name labels
...
Emit metrics labelled with "handler=<name>" and a common metric name,
instead of constructing metrics prefixed with the DML handler name.
2022-02-17 15:11:20 +00:00
4d54f8b42c
refactor: remove migration create schema
2022-02-17 14:41:32 +00:00
44e9eaf92b
test(iox_catalog): isolated catalog tests
...
This commit changes the iox_catalog test harness so that each test is
run in an independent, randomly generated schema to avoid concurrent
tests interfering with each other.
Each test creates a randomly-named schema, grants permissions to the new
schema, runs the full migration stack against the new schema, and then
executes the code under test.
2022-02-17 14:19:01 +00:00
3b378418f7
refactor: do not specify schema in migrations
...
Allow the caller to set the Postgres schema a migration should be
applied to, rather than restricting the migration to a specific,
hard-coded schema.
BREAKING CHANGE: manually adds a new migration that precedes the
existing migration to ensure the iox_catalog schema exists before
applying the migration. You'll probably have to drop any existing
databases and migrate from scratch:
sqlx database drop; sqlx database create;
2022-02-17 14:15:58 +00:00
b1215805a8
fix: upgrade rskafka for bugfixes and race-free start offsets ( #3775 )
...
This fixes the concerns that were brought up during the review of #3748 .
2022-02-17 11:22:25 +00:00
da2a1badc7
Merge pull request #3771 from influxdata/er/tracing/gen_trace
...
feat: generate a trace ID when using gRPC client
2022-02-16 19:32:39 +00:00
44d3d76ab9
Merge branch 'main' into er/tracing/gen_trace
2022-02-16 19:23:15 +00:00
dependabot[bot]
Raphael Taylor-Davies
Marco Neumann
kodiakhq[bot]
Luke Bond
Dom Dwyer
jakevin
Carol (Nichols || Goulding)
Paul Dix
Andrew Lamb
Edd Robinson