9b3f946c10
feat: all in 1 IOx NG mode ( #3965 )
...
* feat: Add all_in_one mode
* fix: doc
* docs: fix truncated docs
* refactor: correctly identify PG connections
* refactor: resolve failed merge
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-15 16:28:37 +00:00
c0e0bcbb1a
Merge pull request #4041 from influxdata/dom/cargo-deny
...
build: manually clone advisory-db repo
2022-03-15 16:04:41 +00:00
7a9618b218
Merge branch 'main' into dom/cargo-deny
2022-03-15 15:53:38 +00:00
e87313f88b
build: manually clone advisory-db repo
...
Use the git CLI to pull the advisory-db repo, instead of letting
cargo-deny do it as it seems to ignore the "pull with CLI" cargo
setting:
https://github.com/EmbarkStudios/cargo-deny/issues/388
Potentially related to GitHub killing old key support today:
https://github.blog/2021-09-01-improving-git-protocol-security-github/
2022-03-15 15:42:24 +00:00
74ae8bc024
Merge pull request #4019 from influxdata/dom/object-store-metrics
...
feat: object store instrumentation
2022-03-15 15:19:07 +00:00
f4cd5c9ba8
Merge branch 'main' into dom/object-store-metrics
2022-03-15 14:27:19 +00:00
7a7dfade94
feat(catalog): instrument list() ops
...
List (but not list_with_delimiter?!) returns a stream of ListResult
which previously wasn't instrumented - this commit uses the
StreamMetricRecorder to record the wall clock duration of the entire
list operation.
2022-03-15 14:23:11 +00:00
ee01225fec
refactor: polymorphic stream recorder impl
...
Changes the StreamMetricRecorder to be generic over the Ok types in the
result stream, invoking a T-specific delegate when Ok(T) is observed.
This enables the stream instrumentation to be reused across different
stream types while keeping the hairy state checks DRY.
This will allow the StreamMetricRecorder to decorate the streams
returned by both the get() and list() operations, but this commit causes
no functional change.
2022-03-15 14:13:17 +00:00
cc7f71126c
chore(deps): Bump syn from 1.0.86 to 1.0.88 ( #4037 )
...
Bumps [syn](https://github.com/dtolnay/syn ) from 1.0.86 to 1.0.88.
- [Release notes](https://github.com/dtolnay/syn/releases )
- [Commits](https://github.com/dtolnay/syn/compare/1.0.86...1.0.88 )
---
updated-dependencies:
- dependency-name: syn
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-15 12:22:18 +00:00
97d595e4fb
feat: `Cache::set` ( #4036 )
...
* feat: `Cache::set`
This will be helpful to fill caches if we got the information from
somewhere else.
For #3985 .
* docs: improve
Co-authored-by: Edd Robinson <me@edd.io>
* docs: explain lock gap
* feat: add debug log to `Cache`
Co-authored-by: Edd Robinson <me@edd.io>
2022-03-15 12:12:26 +00:00
e2cf5c63e5
chore(deps): Bump nom from 7.1.0 to 7.1.1 ( #4035 )
...
Bumps [nom](https://github.com/Geal/nom ) from 7.1.0 to 7.1.1.
- [Release notes](https://github.com/Geal/nom/releases )
- [Changelog](https://github.com/Geal/nom/blob/main/CHANGELOG.md )
- [Commits](https://github.com/Geal/nom/compare/7.1.0...7.1.1 )
---
updated-dependencies:
- dependency-name: nom
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-15 11:29:01 +00:00
eeb918ca2b
chore(deps): Bump libc from 0.2.119 to 0.2.120 ( #4034 )
...
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.119 to 0.2.120.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.119...0.2.120 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-15 10:56:52 +00:00
10831a0530
chore(deps): Bump reqwest from 0.11.9 to 0.11.10 ( #4032 )
...
Bumps [reqwest](https://github.com/seanmonstar/reqwest ) from 0.11.9 to 0.11.10.
- [Release notes](https://github.com/seanmonstar/reqwest/releases )
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md )
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.9...v0.11.10 )
---
updated-dependencies:
- dependency-name: reqwest
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-15 10:43:28 +00:00
4b5cf6a70e
feat: cache processed tombstones ( #4030 )
...
* test: allow to mock time in `iox_test`
* feat: cache processed tombstones
For #3974 .
* refactor: introduce `TTL_NOT_PROCESSED`
2022-03-15 10:28:08 +00:00
8f1938a482
chore: Update datafusion ( #4022 )
...
* chore: Update datafusion
* chore: update for change in Expr
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-14 17:24:00 +00:00
87e53f30d1
refactor: add TTL cache backend ( #4027 )
...
* feat: `CacheBackend::as_any`
* refactor: add TTL cache backend
This is based on the new `AddressableHeap`, which simplifies the
implementation quite a lot.
For #3985 .
* refactor: `TtlBackend::{update->evict_expired}`
* docs: exlain ttl cache eviction
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-14 15:35:27 +00:00
6fd3c3f1ea
fix: Kick c2updater ( #4029 )
2022-03-14 15:24:54 +00:00
4c674b931a
fix: Remove partition_id from metric attributes ( #4028 )
2022-03-14 14:12:34 +00:00
27efb66237
test: add proptest for `AddressableHeap` ( #4025 )
...
* test: add proptest for `AddressableHeap`
For #3985 .
* refactor: simplify code
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: Edd Robinson <me@edd.io>
2022-03-14 12:38:30 +00:00
1aaf193299
refactor: take last_yielded_at
2022-03-14 11:54:02 +00:00
3802401b4f
docs: explicit stream duration semantics
2022-03-14 11:54:02 +00:00
1aee8402bf
refactor: suffix duration vars with _ms
2022-03-14 11:54:02 +00:00
f01382f578
docs: fix typos
...
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2022-03-14 11:36:49 +00:00
3f0f090c4e
chore(deps): Bump pretty_assertions from 1.1.0 to 1.2.0 ( #4024 )
...
Bumps [pretty_assertions](https://github.com/colin-kiegel/rust-pretty-assertions ) from 1.1.0 to 1.2.0.
- [Release notes](https://github.com/colin-kiegel/rust-pretty-assertions/releases )
- [Changelog](https://github.com/colin-kiegel/rust-pretty-assertions/blob/main/CHANGELOG.md )
- [Commits](https://github.com/colin-kiegel/rust-pretty-assertions/commits )
---
updated-dependencies:
- dependency-name: pretty_assertions
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-14 10:33:27 +00:00
632c4953b4
feat: add addressable heap for query cache ( #4016 )
...
* feat: add addressable heap for query cache
This will be used as a helper data structure for TTL and LRU. It's
probably not the most performant implementation but it's good enough for
now.
This is for #3985 .
* fix: test + explain tie breaking in `AddressableHeap`
2022-03-14 09:35:38 +00:00
b77e8a40e5
Merge pull request #4021 from influxdata/cn/compactor-level-0-1-from-catalog
...
feat: Add catalog/compactor functions for getting level 0 and level 1 files; marking files as level 1
2022-03-14 00:42:35 +00:00
268138ceef
fix: Make SQL queries more consistent
...
- Use "SELECT *" when possible
- Left align
- Wrap at 100 chars
- Include semicolon
2022-03-13 20:28:12 -04:00
8888e4c3a2
fix: Remove MAX_COMPACT_SIZE from the compaction queries
2022-03-13 20:09:30 -04:00
1dacf567d9
feat: Add a function to the catalog to fetch level 1 parquet files
...
Fixes #3946 .
2022-03-11 15:40:34 -05:00
f184b7023c
feat: Update specified parquet file records to compaction level 1
...
Fixes #3950 .
2022-03-11 15:34:40 -05:00
fabd262442
feat: Add a function to the catalog to fetch level 0 parquet files
...
Connects to #3946 .
2022-03-11 15:34:05 -05:00
5a29d070ea
feat: Implement the compact function for NG Compactor ( #4001 )
...
* feat: initial implementation of compact a given list of overlapped parquet files
* feat: Add QueryableParquetChunk and some refactoring
* feat: build queryable parquet chunks for parquet files with tombstones
* feat: second half the implementation for Compactor's compact. Tests will be next
* fix: comments for trait funnctions fof QueryChunkMeta
* test: add tests for compactor's compact function
* fix: typos
* refactor: address Jake's review comments
* refactor: address Andrew's comments and add one more test for files in different order in the vector
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-11 20:25:19 +00:00
cc4875cca0
refactor: decouple ingester setup and creation logic from the config structs ( #4020 )
...
* refactor: decouple ingester setup and creation logic from the config structs
* fix: clippy
* refactor: remove comments
2022-03-11 19:25:50 +00:00
b24ae7d23b
refactor: extract out compactor creation from config ( #4018 )
...
* refactor: extract out compactor creation from config
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-11 14:46:34 +00:00
29df46975f
refactor: extract out querier creation from config ( #4017 )
...
* refactor: extract out querier creation from config
* fix: clippy
2022-03-11 14:38:09 +00:00
dc8a708fb7
refactor: use DummyObjectStore for error tests
2022-03-11 14:36:34 +00:00
bfc473b8cd
feat: object store instrumentation
...
Implements a decorator of the ObjectStoreAPI trait, recording:
* Bytes uploaded / downloaded through the instrumented API
* Call latencies, broken down by operation & success / error state
All the current implementations that return a Stream from the get
operation actually return a "fake" stream containing all the data in one
go rather than streaming chunks from the upstream. I've instrumented the
Stream to future-proof any actual streaming impls in the future.
2022-03-11 13:38:48 +00:00
d46de98183
feat: extract "backend" from querier cache ( #4015 )
...
* feat: extract "backend" from querier cache
The backend will implement pruning policies like LRU and TTL as well as
where/how the data is stored. Having a proper interface for that
simplifies the implementation since we don't need to have one massive
`Cache` object with a super complex mechanism.
This is for #3985 .
* refactor: `Backend` -> `CacheBackend`
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-11 11:49:57 +00:00
bd2824aa59
chore(deps): Bump pprof from 0.6.2 to 0.7.0 ( #4004 )
...
* chore(deps): Bump pprof from 0.6.2 to 0.7.0
Bumps [pprof](https://github.com/tikv/pprof-rs ) from 0.6.2 to 0.7.0.
- [Release notes](https://github.com/tikv/pprof-rs/releases )
- [Changelog](https://github.com/tikv/pprof-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tikv/pprof-rs/commits )
---
updated-dependencies:
- dependency-name: pprof
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: `pprof` features for version 0.7
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-11 11:08:59 +00:00
7f7c672ac5
Merge pull request #3992 from influxdata/dom/ingester-lock-fix
...
refactor(ingester): eliminate unnecessary lock
2022-03-11 10:40:40 +00:00
30fc29c296
Merge branch 'main' into dom/ingester-lock-fix
2022-03-11 10:32:26 +00:00
4a7364d63f
refactor: rename lifecycle_manager args
2022-03-11 10:31:58 +00:00
95f461b907
Merge pull request #4012 from influxdata/cn/parquet-file-params
...
refactor: Extract a ParquetFileParams catalog type for use in ParquetFileRepo create
2022-03-10 19:17:21 +00:00
ecd06c6ec3
fix: ParquetFileRepo create should be responsible for setting INITIAL_COMPACTION_LEVEL
...
When created in the catalog, parquet files should always have compaction
level 0. Updating the compaction level should always happen in the
compactor.
Only the catalog should need to know about the initial compaction level
value.
2022-03-10 13:51:18 -05:00
ff31407dce
refactor: Extract a ParquetFileParams type for create
...
This has the advantages of:
- Not needing to create fake parquet file IDs or fake deleted_at
values that aren't used by create before insertion
- Not needing too many arguments for create
- Naming the arguments so it's easier to see what value is what
argument, especially in tests
- Easier to reuse arguments or parts of arguments by using copies of
params, which makes it easier to see differences, especially in tests
2022-03-10 13:51:18 -05:00
f03ebd79ab
refactor: move querier's test utils to a new crate to get reused by tests in other crates ( #4013 )
...
* refactor: move querier's test utils to a new crate to be able resued by tests in other crates
* chore: remove unused import
2022-03-10 18:17:58 +00:00
27999ff72f
feat: add compaction_level and created_at to parquet_file ( #3972 )
2022-03-10 15:56:57 +00:00
67209fce7c
chore(deps): Bump clap from 3.1.5 to 3.1.6 ( #4010 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.1.5 to 3.1.6.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.1.5...v3.1.6 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-10 14:38:53 +00:00
e1344f7ca8
chore(deps): Bump sqlparser from 0.14.0 to 0.15.0 ( #4007 )
...
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs ) from 0.14.0 to 0.15.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases )
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md )
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.14.0...v0.15.0 )
---
updated-dependencies:
- dependency-name: sqlparser
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-10 14:28:48 +00:00
68e09264f2
chore(deps): Bump pbjson from 0.2.2 to 0.2.3 ( #4006 )
...
Bumps [pbjson](https://github.com/influxdata/pbjson ) from 0.2.2 to 0.2.3.
- [Release notes](https://github.com/influxdata/pbjson/releases )
- [Commits](https://github.com/influxdata/pbjson/compare/0.2.2...0.2.3 )
---
updated-dependencies:
- dependency-name: pbjson
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-10 14:20:10 +00:00
Andrew Lamb
kodiakhq[bot]
Dom Dwyer
dependabot[bot]
Marco Neumann
Marko Mikulicic
Carol (Nichols || Goulding)
Nga Tran
Paul Dix