c965221df1
feat: have ingester ignore already persisted data ( #3849 )
2022-02-25 00:08:33 +00:00
2f868f2222
Merge pull request #3860 from influxdata/dom/parallel-column-resolution
...
refactor: parallel column resolution
2022-02-24 21:20:49 +00:00
b07f15bec7
refactor: parallel column resolution
...
A quick change to perform the ColumnRepo::create_or_get() calls in
parallel (up to a maximum of 3 in-flight at any one time) in order to
mitigate the latency of the call and reduce the overall schema
validation call duration.
The in-flight limit is enforced to avoid starving the DB connection pool
of connections.
2022-02-24 21:04:25 +00:00
723a0c659f
fix: Remove greater_than_sequence_number from IngesterQueryRequest ( #3856 )
2022-02-24 19:23:44 +00:00
a3a628c783
refactor: use ChunkMetadata in catalog interface ( #3845 ) ( #3858 )
...
* refactor: use ChunkMetadata in catalog interface (#3845 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-24 18:05:07 +00:00
26ac93334f
Merge pull request #3854 from influxdata/dom/ns-cache-content-metrics
...
feat: namespace cache table/column count metrics
2022-02-24 17:29:02 +00:00
a1b38b7102
Merge branch 'main' into dom/ns-cache-content-metrics
2022-02-24 16:31:09 +00:00
34e06e8689
fix: compactor server stays up; removed unused delegates ( #3855 )
...
* fix: compactor server stays up; removed unused delegates
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-24 16:30:44 +00:00
8571c132cc
feat: add method to get table persistence information from catalog ( #3848 )
2022-02-24 16:18:14 +00:00
3d77cf5845
test: validate metrics for adding namespace
2022-02-24 16:07:02 +00:00
49d1be30e7
feat: wire up `ParquetFilePath` for NG ( #3853 )
...
It's a bit of a duck-type hack, but if we wanna just `ParquetFileChunk`
in the new architecture, we somehow need it to accept new-gen paths.
Also path handling should be somewhat centralized since
ingester/compactor/querier all need to construct them. So having a
`ParquetFilePath` that supports both path styles seems to be a
not-to-bad solution. This should obviously be cleaned up in some
not-to-distant future.
2022-02-24 16:05:38 +00:00
4731913c44
feat: skeleton of querier CLI ( #3843 )
...
* feat: skeleton of querier CLI
* chore: wrap metrics in opt&arc in querier to satisfy new api
* chore: derive debug in querier handler
* chore: add join handles and their shutdown to nascent querier server
* chore: querier server http unimpl -> 404
* fix: join/shutdown fix in querier; removed unused delegates
2022-02-24 15:42:56 +00:00
252ced7adf
feat: Add row count to the parquet_file record in the catalog ( #3847 )
...
Fixes #3842 .
2022-02-24 15:20:50 +00:00
0ddc35ce73
feat: instrument namespace cache contents
...
Adds two new metrics:
* namespace_cache_table_count: total number of tables in cache
* namespace_cache_column_count: total number of columns in cache
The metric decorator keeps a running total of each of the table and
column counts as namespaces are inserted into the cache, and adjusts the
value accordingly when an existing namespace is overwrote.
2022-02-24 15:11:14 +00:00
4024e95ce9
refactor: borrow metric registry
...
There's no need for the namespace metrics to take (shared) ownership of
the metric registry, so lend it at the call site instead of cloning the
arc.
2022-02-24 15:04:49 +00:00
77f649210d
feat: inc/dec gauge metrics
...
Adds methods to increment, or decrement gauge metrics.
2022-02-24 15:04:49 +00:00
d62a052394
feat: extend catalog so we can recover `ParquetChunk`s from it ( #3852 )
...
* refactor: less parquet data copying
* feat: `PartitionRepo::get_by_id`
* feat: `TableRepo::get_by_id`
* feat: `ParquetFile::file_size_bytes`
* feat: `ParquetFile::parquet_metadata`
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-24 13:16:15 +00:00
c8b964ebef
Merge pull request #3846 from influxdata/dom/schema-before-partition
...
refactor: early schema validation
2022-02-24 10:49:01 +00:00
d7eda88581
refactor: early schema validation
...
Changes the configuration of the router request pipeline to move schema
validation before partitioning.
This reduces the concurrency of callsm into the schema validator when a
single write is split into one or more partitions, reducing contention
and cash thrashing. It also ensures we don't bother partitioning the
writes if the request will fail.
2022-02-23 18:59:14 +00:00
9079e6ddb0
feat: backoff retries in ingester ( #3841 )
...
* feat: add `backoff` crate
* feat: backoff retries in ingester
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 17:58:16 +00:00
bc1cc8dc5c
chore: update database rebuild instructions ( #3838 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 17:46:22 +00:00
6d15bed6be
Merge pull request #3830 from influxdata/dom/router-traces
...
feat: emit trace spans for router stages
2022-02-23 17:34:28 +00:00
e1f54f67af
Merge branch 'main' into dom/router-traces
2022-02-23 17:23:17 +00:00
9707d85e5e
test: InstrumentationDecorator DML handler impls
2022-02-23 17:23:02 +00:00
c4dda758b2
Merge pull request #3837 from influxdata/dom/catalog-instrumentation
...
feat: catalog instrumentation
2022-02-23 16:27:15 +00:00
3e69a5e1b4
Merge branch 'main' into dom/catalog-instrumentation
2022-02-23 16:14:40 +00:00
71f62eee68
fix: Remove min_time and max_time from IngesterQueryRequest ( #3839 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 15:46:31 +00:00
657ac249e9
feat: track ingester jobs ( #3836 )
2022-02-23 15:33:47 +00:00
aaf8951927
feat: instrument postgres catalog impl
...
Wraps the postgres implementation of the catalog with a MetricDecorator.
This is slightly intrusive, with the metrics registry being pushed into
the PostgresCatalog type in order to decorate the impls returned when
calling the Catalog::repositories() and Catalog::start_transaction()
methods (rather than being a pure decorator) in order to use static
dispatch and let the compiler optimise away as much overhead as
possible.
2022-02-23 14:34:26 +00:00
a0b43323d6
feat: catalog instrumentation decorator
...
Adds a MetricDecorator type that wraps a Catalog impl, delegating calls
to the underlying impl and recording per-method call latency broken down
by success/error state.
2022-02-23 14:34:26 +00:00
10a1670341
test: catalog `maybe_skip_integration!` and `should_panic` ( #3834 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 13:25:30 +00:00
ade6475570
feat: add sort ordinal to chunk_columns system table ( #2035 ) ( #3831 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 12:51:56 +00:00
746b18bbed
feat: allow overriding CLI request timeout ( #3824 )
...
* feat: allow overriding CLI request timeout
* chore: rename to --rpc-timeout to avoid name collision
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 11:35:45 +00:00
b63f920d4c
chore(deps): Bump parquet from 9.0.2 to 9.1.0 ( #3828 )
...
* chore(deps): Bump parquet from 9.0.2 to 9.1.0
Bumps [parquet](https://github.com/apache/arrow-rs ) from 9.0.2 to 9.1.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/9.0.2...9.1.0 )
---
updated-dependencies:
- dependency-name: parquet
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: update chunk size test
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <r.taylordavies@googlemail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 11:25:15 +00:00
18c2ca1bb5
chore(deps): Bump tower from 0.4.11 to 0.4.12 ( #3817 )
...
* chore(deps): Bump tower from 0.4.11 to 0.4.12
Bumps [tower](https://github.com/tower-rs/tower ) from 0.4.11 to 0.4.12.
- [Release notes](https://github.com/tower-rs/tower/releases )
- [Commits](https://github.com/tower-rs/tower/compare/tower-0.4.11...tower-0.4.12 )
---
updated-dependencies:
- dependency-name: tower
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: update workspace-hack
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <r.taylordavies@googlemail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 11:14:15 +00:00
5a79b3a68b
chore(deps): Bump arrow-flight from 9.0.2 to 9.1.0 ( #3829 )
...
Bumps [arrow-flight](https://github.com/apache/arrow-rs ) from 9.0.2 to 9.1.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/9.0.2...9.1.0 )
---
updated-dependencies:
- dependency-name: arrow-flight
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 11:03:22 +00:00
b20dce80a2
feat: emit trace spans for router stages
...
Configures the instrumentation decorator to emit a trace span covering
the duration of the decorated handler's execution, recording the
success/error result and and error message, if any.
2022-02-23 10:39:13 +00:00
79144e227b
chore(deps): Bump integer-encoding from 3.0.2 to 3.0.3 ( #3827 )
...
Bumps [integer-encoding](https://github.com/dermesser/integer-encoding-rs ) from 3.0.2 to 3.0.3.
- [Release notes](https://github.com/dermesser/integer-encoding-rs/releases )
- [Commits](https://github.com/dermesser/integer-encoding-rs/commits )
---
updated-dependencies:
- dependency-name: integer-encoding
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 09:37:26 +00:00
3b7d31c88a
chore(deps): Bump arrow from 9.0.2 to 9.1.0 ( #3826 )
...
Bumps [arrow](https://github.com/apache/arrow-rs ) from 9.0.2 to 9.1.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/9.0.2...9.1.0 )
---
updated-dependencies:
- dependency-name: arrow
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-23 09:25:46 +00:00
459974d1b8
refactor: move `DedicatedExecutor` from `query` to `executor` ( #3819 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 09:10:43 +00:00
f7df54623e
chore(deps): Bump getrandom from 0.2.4 to 0.2.5 ( #3818 )
...
Bumps [getrandom](https://github.com/rust-random/getrandom ) from 0.2.4 to 0.2.5.
- [Release notes](https://github.com/rust-random/getrandom/releases )
- [Changelog](https://github.com/rust-random/getrandom/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-random/getrandom/compare/v0.2.4...v0.2.5 )
---
updated-dependencies:
- dependency-name: getrandom
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 08:59:05 +00:00
276d9b123a
feat: Add min_sequence_number tracking for sequencers in ingester ( #3785 )
...
Fixes #3702 . This pulls the min sequence tracking into the LifecycleManager. Because the number requires looking at all other partitions in memory, this was the most efficient place to put it. The manager updates the sequencer state after it calls persist. The number is meant to be a lower bound on the sequence number. Issue #3783 will add functionality for the ingester to ignore replayed data that has already been persisted.
2022-02-22 21:53:33 +00:00
a91e2eadc7
feat: apply tombstones to the batches of the ingest life-cycle ( #3770 )
...
* feat: changes needed to apply tombstones correctly on the life-cycle ingest bacthes
* refactor: adjust the design after discussing with Paul
* feat: apply the coming tombstone on all data but persiting one
* chore: fmt
* fix: build on buffer tombstone
* test: delete & write tests for a parition and some cleanup
* feat: No need add processed tombstones for newly created parquet file in the ingester becasue all deletes before that parquet file is created were applied
* chore: cleanup
Co-authored-by: Paul Dix <paul@pauldix.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 18:54:21 +00:00
e19609ab7b
feat: routing service protection ( #3807 )
...
* chore: db migration for namespace table & column limits
* feat: impl table & column limits in catalog
* chore: improved comment in catalog
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 17:26:37 +00:00
25c525f187
chore(deps): Bump tracing-subscriber from 0.3.8 to 0.3.9 ( #3815 )
...
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing ) from 0.3.8 to 0.3.9.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.8...tracing-subscriber-0.3.9 )
---
updated-dependencies:
- dependency-name: tracing-subscriber
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-22 17:01:24 +00:00
ad3868ed7c
chore(deps): Bump tokio from 1.16.1 to 1.17.0 ( #3814 )
...
* chore(deps): Bump tokio from 1.16.1 to 1.17.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.16.1 to 1.17.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.16.1...tokio-1.17.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build: update workspace-hack
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:27:43 +00:00
8574bb695c
chore(deps): Bump tikv-jemalloc-sys ( #3813 )
...
Bumps [tikv-jemalloc-sys](https://github.com/tikv/jemallocator ) from 0.4.2+5.2.1-patched.2 to 0.4.3+5.2.1-patched.2.
- [Release notes](https://github.com/tikv/jemallocator/releases )
- [Changelog](https://github.com/tikv/jemallocator/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tikv/jemallocator/commits )
---
updated-dependencies:
- dependency-name: tikv-jemalloc-sys
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:12:33 +00:00
9ff44ce90b
feat: sequencer metrics during replay ( #3806 )
...
* feat: sequencer metrics during replay
* feat: add replay metrics test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:01:27 +00:00
ae5712fab0
chore(deps): Bump kube-derive from 0.69.0 to 0.69.1 ( #3812 )
...
Bumps [kube-derive](https://github.com/kube-rs/kube-rs ) from 0.69.0 to 0.69.1.
- [Release notes](https://github.com/kube-rs/kube-rs/releases )
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.69.0...0.69.1 )
---
updated-dependencies:
- dependency-name: kube-derive
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 15:49:14 +00:00
63f35d0006
refactor: generic+reusable task registry ( #3811 )
...
* feat: introduce `AbstractTaskRegistry`
* feat: make `TaskRegistryWithHistory` generic
* feat: reusable task<>metric integration
* refactor: simplify `FAttributes`
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 15:35:29 +00:00
Paul Dix
kodiakhq[bot]
Dom Dwyer
Carol (Nichols || Goulding)
Raphael Taylor-Davies
Luke Bond
Marco Neumann
dependabot[bot]
Nga Tran