Commit Graph

6942 Commits (f7df54623e8c2424621c907c7ffa759a429b0d6a)

Author SHA1 Message Date
dependabot[bot] f7df54623e
chore(deps): Bump getrandom from 0.2.4 to 0.2.5 (#3818)
Bumps [getrandom](https://github.com/rust-random/getrandom) from 0.2.4 to 0.2.5.
- [Release notes](https://github.com/rust-random/getrandom/releases)
- [Changelog](https://github.com/rust-random/getrandom/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-random/getrandom/compare/v0.2.4...v0.2.5)

---
updated-dependencies:
- dependency-name: getrandom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 08:59:05 +00:00
Paul Dix 276d9b123a
feat: Add min_sequence_number tracking for sequencers in ingester (#3785)
Fixes #3702. This pulls the min sequence tracking into the LifecycleManager. Because the number requires looking at all other partitions in memory, this was the most efficient place to put it. The manager updates the sequencer state after it calls persist. The number is meant to be a lower bound on the sequence number. Issue #3783 will add functionality for the ingester to ignore replayed data that has already been persisted.
2022-02-22 21:53:33 +00:00
Nga Tran a91e2eadc7
feat: apply tombstones to the batches of the ingest life-cycle (#3770)
* feat: changes needed to apply tombstones correctly on the life-cycle ingest bacthes

* refactor: adjust the  design after discussing with Paul

* feat: apply the coming tombstone on all data but persiting one

* chore: fmt

* fix: build on buffer tombstone

* test: delete & write tests for a parition and some cleanup

* feat: No need add processed tombstones for newly created parquet file in the ingester becasue all deletes before that parquet file is created were applied

* chore: cleanup

Co-authored-by: Paul Dix <paul@pauldix.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 18:54:21 +00:00
Luke Bond e19609ab7b
feat: routing service protection (#3807)
* chore: db migration for namespace table & column limits

* feat: impl table & column limits in catalog

* chore: improved comment in catalog

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 17:26:37 +00:00
dependabot[bot] 25c525f187
chore(deps): Bump tracing-subscriber from 0.3.8 to 0.3.9 (#3815)
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.3.8 to 0.3.9.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.8...tracing-subscriber-0.3.9)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-22 17:01:24 +00:00
dependabot[bot] ad3868ed7c
chore(deps): Bump tokio from 1.16.1 to 1.17.0 (#3814)
* chore(deps): Bump tokio from 1.16.1 to 1.17.0

Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.16.1 to 1.17.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.16.1...tokio-1.17.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build: update workspace-hack

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:27:43 +00:00
dependabot[bot] 8574bb695c
chore(deps): Bump tikv-jemalloc-sys (#3813)
Bumps [tikv-jemalloc-sys](https://github.com/tikv/jemallocator) from 0.4.2+5.2.1-patched.2 to 0.4.3+5.2.1-patched.2.
- [Release notes](https://github.com/tikv/jemallocator/releases)
- [Changelog](https://github.com/tikv/jemallocator/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tikv/jemallocator/commits)

---
updated-dependencies:
- dependency-name: tikv-jemalloc-sys
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:12:33 +00:00
Raphael Taylor-Davies 9ff44ce90b
feat: sequencer metrics during replay (#3806)
* feat: sequencer metrics during replay

* feat: add replay metrics test

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:01:27 +00:00
dependabot[bot] ae5712fab0
chore(deps): Bump kube-derive from 0.69.0 to 0.69.1 (#3812)
Bumps [kube-derive](https://github.com/kube-rs/kube-rs) from 0.69.0 to 0.69.1.
- [Release notes](https://github.com/kube-rs/kube-rs/releases)
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.69.0...0.69.1)

---
updated-dependencies:
- dependency-name: kube-derive
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 15:49:14 +00:00
Marco Neumann 63f35d0006
refactor: generic+reusable task registry (#3811)
* feat: introduce `AbstractTaskRegistry`

* feat: make `TaskRegistryWithHistory` generic

* feat: reusable task<>metric integration

* refactor: simplify `FAttributes`

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 15:35:29 +00:00
dependabot[bot] 1dca869c3f
chore(deps): Bump sqlx from 0.5.10 to 0.5.11 (#3799)
Bumps [sqlx](https://github.com/launchbadge/sqlx) from 0.5.10 to 0.5.11.
- [Release notes](https://github.com/launchbadge/sqlx/releases)
- [Changelog](https://github.com/launchbadge/sqlx/blob/master/CHANGELOG.md)
- [Commits](https://github.com/launchbadge/sqlx/compare/v0.5.10...v0.5.11)

---
updated-dependencies:
- dependency-name: sqlx
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 15:14:36 +00:00
Raphael Taylor-Davies 69480eaa5c
feat: remove remaining table-granularity metrics (#3808)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 15:02:57 +00:00
dependabot[bot] 65ab5213e5
chore(deps): Bump clap from 3.0.14 to 3.1.1 (#3809)
Bumps [clap](https://github.com/clap-rs/clap) from 3.0.14 to 3.1.1.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v3.0.14...v3.1.1)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-22 14:51:53 +00:00
dependabot[bot] c2a01c2407
chore(deps): Bump comfy-table from 5.0.0 to 5.0.1 (#3800)
Bumps [comfy-table](https://github.com/nukesor/comfy-table) from 5.0.0 to 5.0.1.
- [Release notes](https://github.com/nukesor/comfy-table/releases)
- [Changelog](https://github.com/Nukesor/comfy-table/blob/main/CHANGELOG.md)
- [Commits](https://github.com/nukesor/comfy-table/compare/v5.0.0...v5.0.1)

---
updated-dependencies:
- dependency-name: comfy-table
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 14:36:52 +00:00
Raphael Taylor-Davies 1960645055
feat: add wildcard support to persist partition CLI command (#3790)
* feat: add wildcard support to persist partition

* chore: fmt

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 14:21:06 +00:00
Raphael Taylor-Davies 0229147909
feat: preserve catalog on error (#1522) (#3802)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 14:10:20 +00:00
kodiakhq[bot] 454ccc7566
Merge pull request #3788 from influxdata/dom/router-handler-chain
refactor(router2): non-intrusive handler chain
2022-02-22 13:59:28 +00:00
kodiakhq[bot] 326871f149
Merge branch 'main' into dom/router-handler-chain 2022-02-22 13:49:09 +00:00
kodiakhq[bot] 3d8f0246c5
Merge pull request #3803 from influxdata/feat/new-compactor-command-and-crate
feat: adding compactor CLI command and crate
2022-02-22 11:27:41 +00:00
kodiakhq[bot] 4ef704376f
Merge branch 'main' into feat/new-compactor-command-and-crate 2022-02-22 10:57:29 +00:00
Luke Bond 0f012de70c
feat: adding compactor CLI command and crate
Closes: #3777
2022-02-21 12:24:09 +00:00
kodiakhq[bot] 50aa39ff3f
Merge pull request #3789 from influxdata/dom/fix-catalog-migrations
fix: "catalog setup" migrations in non-public schema
2022-02-21 12:22:35 +00:00
Dom Dwyer 4f0bd29a5b build: run "catalog setup" in CI
Ensures the "catalog setup" command executes successfully by executing
it in CI.
2022-02-21 11:58:57 +00:00
Dom Dwyer 2500dc3ac7 fix: add public to schema search path
The _sql_migrations table cannot be created by the `catalog setup`
bootstrap command because it is created before the migrations run, one
of which creates the iox_catalog namespace the catalog operates in.

This change allows the _sql_migrations table to be created in the public
schema to start the migration process, with everything else living in
the iox_catalog schema.
2022-02-21 11:58:55 +00:00
dependabot[bot] a16d4b80c4
chore(deps): Bump tracing from 0.1.30 to 0.1.31 (#3798)
Bumps [tracing](https://github.com/tokio-rs/tracing) from 0.1.30 to 0.1.31.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.30...tracing-0.1.31)

---
updated-dependencies:
- dependency-name: tracing
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 10:40:09 +00:00
jakevin df9b51fff3
docs: Add prerequisites. (#3731)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 10:31:25 +00:00
dependabot[bot] 474a4099e1
chore(deps): Bump kube from 0.69.0 to 0.69.1 (#3796)
Bumps [kube](https://github.com/kube-rs/kube-rs) from 0.69.0 to 0.69.1.
- [Release notes](https://github.com/kube-rs/kube-rs/releases)
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.69.0...0.69.1)

---
updated-dependencies:
- dependency-name: kube
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 10:22:58 +00:00
Raphael Taylor-Davies 39c42678d7
feat: trigger persistence if over soft limit and no evictable chunks (#3791)
* feat: trigger persistence if over soft limit and no evictable chunks

* chore: fmt

* fix: avoid test_full_lifecycle exceeding soft limit

* fix: don't expect chunk to be unloaded

* feat: only trigger if no outstanding persist

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 10:14:27 +00:00
dependabot[bot] 271daa5eac
chore(deps): Bump sqlx-core from 0.5.10 to 0.5.11 (#3792)
Bumps [sqlx-core](https://github.com/launchbadge/sqlx) from 0.5.10 to 0.5.11.
- [Release notes](https://github.com/launchbadge/sqlx/releases)
- [Changelog](https://github.com/launchbadge/sqlx/blob/master/CHANGELOG.md)
- [Commits](https://github.com/launchbadge/sqlx/compare/v0.5.10...v0.5.11)

---
updated-dependencies:
- dependency-name: sqlx-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 10:05:31 +00:00
dependabot[bot] 9ad10a5a5a
chore(deps): Bump libc from 0.2.117 to 0.2.119 (#3795)
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.117 to 0.2.119.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.117...0.2.119)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 09:56:17 +00:00
dependabot[bot] 09e69a57b2
chore(deps): Bump cc from 1.0.72 to 1.0.73 (#3793)
Bumps [cc](https://github.com/alexcrichton/cc-rs) from 1.0.72 to 1.0.73.
- [Release notes](https://github.com/alexcrichton/cc-rs/releases)
- [Commits](https://github.com/alexcrichton/cc-rs/compare/1.0.72...1.0.73)

---
updated-dependencies:
- dependency-name: cc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 09:47:47 +00:00
dependabot[bot] d6317a2ffd
chore(deps): Bump kube-runtime from 0.69.0 to 0.69.1 (#3794)
Bumps [kube-runtime](https://github.com/kube-rs/kube-rs) from 0.69.0 to 0.69.1.
- [Release notes](https://github.com/kube-rs/kube-rs/releases)
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.69.0...0.69.1)

---
updated-dependencies:
- dependency-name: kube-runtime
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-21 09:37:52 +00:00
Carol (Nichols || Goulding) 1b9212540b
feat: Send IngesterQueryResponse data back as response of doGet Flight request (#3772)
* fix: Adjust fields of IngesterQueryResponse

* feat: Adjust IngestHandler query method to call prepare_data_to_querier

* feat: Send ingest query result data back through Flight doGet

* feat: Send delete predicates and max sequencer number in metadata

* fix: greater_than_sequence_number should be of type SequenceNumber

* fix: Remove DeletePredicates from IngesterQueryResponse

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-18 17:42:49 +00:00
Luke Bond 7969ec682c
feat: catalog now supports dsn-file:// (#3787)
* feat: catalog now supports dsn-file://

* chore: rename fn in catalog PG hotswap code
2022-02-18 16:42:55 +00:00
Dom Dwyer 497615d715 test: router handler stack integration test
Adds an integration test covering the router's HTTP handler stack.

Given a well-formed HTTP write, the test asserts:

    * Write passes through the stack without error
    * Response code sent to client
    * Write buffer message is enqueued
    * Catalog namespace record is created
    * Metric handler is invoked and the hit is recorded
2022-02-18 14:39:03 +00:00
Dom Dwyer bb132b61ad refactor: chain DML handlers
The router is composed of several DML handlers called in sequence in
order to construct the full request handling pipeline. Prior to this
commit, each handler nested the next handler it calls internally,
producing a nested call chain that resulted metrics (added in #3764)
recording cumulative latency like this:

              ┌ ─

              │     ┌───────────────┐
                    │  NS Creation  │
              │     └───────────────┘
                            │  ┌───────────────┐
              │             │  │  Partitioner  │
                            │  └───────────────┘
              │             │          │
                            │          │
 Cumulative   │             │          │  ┌───────────────┐
   Timings                1.5s        1s  │    etc...     │
              │             │          │  └───────────────┘
                            │          │
              │             │          │
                            │  ┌───────────────┐
              │             │  │  Partitioner  │
                            │  └───────────────┘
              │     ┌───────────────┐
                    │  NS Creation  │
              │     └───────────────┘

              └ ─

This meant it was hard to determine the latency of a single handler
without knowing (and subtracting the latency of) all the child handlers
it calls.

This commit replaces the intrusive nested handler call chain with an
external Chain combinator type to compose together individual handlers,
resulting in correct per-handler timings and simpler code/tests:

          ┌───────────────┐
          │  NS Creation  │
          └───────────────┘
                  │
                 .5s       ┌───────────────┐
                  └───────▶│  Partitioner  │
                           └───────────────┘
                                   │
                                  1s    ┌───────────────┐
                                   └───▶│    etc...     │
                                        └───────────────┘
2022-02-18 14:19:53 +00:00
kodiakhq[bot] a686c2ab02
Merge pull request #3779 from influxdata/dom/catalog-spurious-writes
fix: catalog spurious writes
2022-02-18 12:42:27 +00:00
kodiakhq[bot] e105b3a84c
Merge branch 'main' into dom/catalog-spurious-writes 2022-02-18 12:33:05 +00:00
Raphael Taylor-Davies 83cba3d2fb
feat: template static router config (#3781)
* feat: template static router config

* chore: lint and improved failure output

* chore: clarify docs

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-18 10:53:10 +00:00
Marco Neumann f54ef92b77
fix: supervise and shutdown ingester background tasks (#3769)
* fix: supervise and shutdown ingester background tasks

Closes #3761.
Closes #3762.

* docs: improve wording

Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>

* test: join/shutdown handling for ingester

Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2022-02-18 09:35:29 +00:00
Paul Dix 23b3942306
fix: compact persisting panics on single row (#3784) 2022-02-17 18:33:04 +00:00
Dom Dwyer 568a15510e fix: assert on conflicting tombstone creation
If two callers call create_or_get() for a tombstone, providing the same
(table_id, sequencer_id, sequence_number) triplet but a different
predicate / timestamps the catalog MUST NOT silently continue.

As this is unexpected, this behaviour causes a panic.
2022-02-17 16:29:55 +00:00
Dom Dwyer 0cc5c979c6 fix: assert on conflicting partition creation
If two callers call create_or_get() for a partition, providing the same
partition key & table ID, but different sequence numbers the catalog
MUST NOT continue silently.

As this is unexpected, this behaviour causes a panic.
2022-02-17 16:29:55 +00:00
Dom Dwyer 0796bd079b test(iox_catalog): assert no spurious writes
This commit introduces test cases that ensure that for each of the
tombstone & partition repos:

    * create_or_get() is idempotent
    * create_or_get() does not silently drop conflicting writes

The former covers the expected use case: callers issuing potentially
multiple calls to create_or_get() with the same args, causing the
catalog impl to transparently turn the subsequent calls into NOPs.

The latter illustrates an issue where multiple calls to create_or_get()
with differing arguments is silently accepted by the catalog, causing
both callers to believe they have committed the (differing) details they
provided. This test is expected to pass but fails in this commit.
2022-02-17 16:29:55 +00:00
kodiakhq[bot] 80a487b653
Merge pull request #3776 from influxdata/dom/isolated-catalog-tests
test: isolated catalog tests
2022-02-17 15:47:48 +00:00
kodiakhq[bot] 74768db827
Merge branch 'main' into dom/isolated-catalog-tests 2022-02-17 15:39:29 +00:00
Andrew Lamb 9588b43a90
fix: Make errors in rewriting return `Error` rather than a `panic` (#3767)
* test: add test for predicate errors

* fix: Return errors properly rather than panic

* fix: handle errors in influxrpc planner

* fix: appease clippy

* fix: tests

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-17 15:39:14 +00:00
kodiakhq[bot] 9d25e5bac3
Merge pull request #3764 from influxdata/dom/router2-metrics
feat: router2 metrics
2022-02-17 15:25:16 +00:00
kodiakhq[bot] c89fa3701e
Merge branch 'main' into dom/router2-metrics 2022-02-17 15:16:39 +00:00
Dom Dwyer 52fd2af851 refactor: DML handler metric name labels
Emit metrics labelled with "handler=<name>" and a common metric name,
instead of constructing metrics prefixed with the DML handler name.
2022-02-17 15:11:20 +00:00