fc539cfe33
feat: compaction target level detection
2023-02-02 15:09:29 -05:00
5a896764cc
refactor: Framework for compact version2: hotcold ( #6810 )
...
* refactor: Framework for compact version2: hotcold
* chore: Apply suggestions from code review
Co-authored-by: Marco Neumann <marco@crepererum.net>
* refactor: address review comments
---------
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-02 17:07:58 +00:00
cb582b859d
revert: throttle partitions that do not receive commits ( #6778 ) ( #6820 )
...
This will stall the compactor and only touch each partition ones because
the "unique" combo thinks that partitions never finish. This will need
more thought.
2023-02-02 16:29:45 +00:00
c9b6e269d7
test: Port remaining query_tests/src/sql.rs tests to end-to-end ( #6807 )
...
* fix: Reading file error reported the wrong path
When the `.expected` SQL file couldn't be found, this error reported
the input file path instead.
* test: Port SQL query_tests to end-to-end tests
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-02 15:35:15 +00:00
7055abcddf
Merge pull request #6817 from influxdata/dom/wal-metrics
...
feat(metrics): ingester2 WAL replay
2023-02-02 14:17:12 +00:00
67903a4bf2
feat(metrics): ingester2 WAL replay
...
Adds two metrics:
* Number of files replayed (counted at the start of, not completion)
* Number of applied ops
This will help identify when WAL replay is happening (an indication of
an ungraceful shutdown & potential temporary read unavailability).
2023-02-02 14:52:09 +01:00
0ddf11adab
chore: add definition for the output of compacting a partition ( #6806 )
...
* chore: add definition for the output of compacting a partition
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* chore: address review comments
---------
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-02 13:35:32 +00:00
4da6b8adf4
Merge pull request #6815 from influxdata/dom/utc-db
...
feat(catalog): use explicit UTC time zone
2023-02-02 11:43:54 +00:00
c343ea60b7
feat(catalog): use explicit UTC time zone
...
We use UTC, but that doesn't mean everyone does. Queries that utilise
NOW() will return incorrect results when the server is using a non-UTC
tz, but application-provided UTC timestamps / epochs.
2023-02-02 11:51:25 +01:00
11a7abbbc2
Merge pull request #6814 from influxdata/dom/gpg-keys
...
ci: use new influx gpg signing key
2023-02-02 10:00:16 +00:00
a56301858a
Merge branch 'main' into dom/gpg-keys
2023-02-02 09:47:04 +00:00
a6d543556d
chore(deps): Bump heck from 0.4.0 to 0.4.1 ( #6812 )
...
Bumps [heck](https://github.com/withoutboats/heck ) from 0.4.0 to 0.4.1.
- [Release notes](https://github.com/withoutboats/heck/releases )
- [Changelog](https://github.com/withoutboats/heck/blob/master/CHANGELOG.md )
- [Commits](https://github.com/withoutboats/heck/commits )
---
updated-dependencies:
- dependency-name: heck
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-02 09:46:15 +00:00
37cf9009b3
Merge branch 'main' into dom/gpg-keys
2023-02-02 09:38:40 +00:00
c90ddd12d1
feat: throttle partitions that do not receive commits ( #6778 )
...
This was a bit tricky to design so it is testable and modular, but I
think this turned out quite nicely. It will even work w/ #6750 .
Fixes #6727 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-02 09:38:03 +00:00
8ca12d488f
ci: use new influx gpg signing key
...
We recently rotated our GPG keys:
https://www.influxdata.com/blog/linux-package-signing-key-rotation/
2023-02-02 10:34:42 +01:00
170492eaa5
Merge pull request #6813 from influxdata/dependabot/cargo/io-lifetimes-1.0.5
...
chore(deps): Bump io-lifetimes from 1.0.4 to 1.0.5
2023-02-02 09:29:50 +00:00
c5b81f4ac8
chore: Run cargo hakari tasks
2023-02-02 01:06:39 +00:00
00b697d76a
chore(deps): Bump io-lifetimes from 1.0.4 to 1.0.5
...
Bumps [io-lifetimes](https://github.com/sunfishcode/io-lifetimes ) from 1.0.4 to 1.0.5.
- [Release notes](https://github.com/sunfishcode/io-lifetimes/releases )
- [Commits](https://github.com/sunfishcode/io-lifetimes/compare/1.0.4...v1.0.5 )
---
updated-dependencies:
- dependency-name: io-lifetimes
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-02-02 01:05:31 +00:00
8e931514fe
feat: Regular expression operator support ( #6792 )
...
* feat: Regular expression operator support
* chore: Added additional comments
2023-02-01 23:17:40 +00:00
0ef4f9fa9b
feat: start adding piece by piece for hotcold compaction ( #6804 )
...
* feat: start adding piece by piece for hotcold compaction
* chore: remove upgrade from commit results
* chore: address review comments
* chore: comments for stop conditions
2023-02-01 19:52:48 +00:00
2574082776
Merge pull request #6801 from influxdata/cn/tag-values
...
test: Port tag_values query_tests to end-to-end tests
2023-02-01 17:59:15 +00:00
23aea749cf
Merge branch 'main' into cn/tag-values
2023-02-01 17:51:20 +00:00
167cde1838
fix(iox): Add transition shard to catalog setup ( #6799 )
2023-02-01 17:04:03 +00:00
7590fa5aaa
fix: Change test data and predicates to get better coverage
...
These two tests weren't actually ensuring that the combination of these
predicates worked, because the tests would still pass if some of the
predicate parts were removed.
2023-02-01 11:09:10 -05:00
c77e1214ff
Merge pull request #6788 from influxdata/dom/fatal-panics
...
feat: fatal panics for ingester2 and router2
2023-02-01 15:55:20 +00:00
cffc4d7233
Merge branch 'main' into dom/fatal-panics
2023-02-01 15:43:52 +00:00
d16eaac64b
refactor: ensure that we only have a partition ID ONCE within the pipeline ( #6774 )
...
* refactor: `MockPartitionDoneSink::{errors => results}`
* feat: ensure we don't concurrently process the same partition
Helps w/ #6750 .
* test: extend `test_unique`
* docs: explain `unique_partitions`
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-01 15:42:40 +00:00
7dace5a50c
Merge branch 'main' into dom/fatal-panics
2023-02-01 15:34:10 +00:00
0a20bd404e
feat: allow selection of different compactor algo versions ( #6800 )
...
Required to lift&shift to hot-cold compaction w/ keeping the codebase
maintainable.
2023-02-01 15:33:41 +00:00
a0b0bb0a93
refactor: fatal panics for ingester2, not ingester
...
I put the calls in the wrong "ingester".
2023-02-01 16:26:24 +01:00
45c08d58b1
feat(router): fatal panics
...
Panics are exceptional, fatal events.
2023-02-01 16:26:24 +01:00
246e3a4684
feat(ingester): fatal panics
...
Panics are exceptional, fatal events.
2023-02-01 16:26:23 +01:00
f2ecfa2adb
feat(iox): fatal panics
...
Allows code to opt-in to fatal panic semantics instead of just logging
them using our normal handler.
2023-02-01 16:26:23 +01:00
d49dd48900
fix: Clean up remaining query_tests influx RPC helper code
2023-02-01 10:26:14 -05:00
072e228d35
test: Port tag_values query_tests to end-to-end tests
2023-02-01 10:26:03 -05:00
629867c6ac
refactor: Start a new file for tag values tests; move the one existing test
2023-02-01 10:25:48 -05:00
967450f0de
Merge pull request #6798 from influxdata/dom/doc-retention-0
...
docs: retention period value mappings
2023-02-01 14:11:57 +00:00
77ff4dc4b8
docs: retention period values in Namespace client
...
Document the 0/null and negative value behaviours for retention periods
in the namespace RPC client.
2023-02-01 14:41:10 +01:00
52ac1b97a9
docs: namespace retention protobuf mappings
...
Document that the caller can specify 0 or NULL for an infinite retention
period, and that IOx will respond with NULL.
Document that negative retention periods are rejected.
2023-02-01 14:37:21 +01:00
a8cb3aeba8
Merge pull request #6795 from influxdata/dom/0-retention
...
fix(router): 0 value retention periods
2023-02-01 11:44:07 +00:00
ab32781cf1
Merge branch 'main' into dom/0-retention
2023-02-01 11:36:34 +00:00
b8b59bf73b
feat: allow partition filters to error + flag partitions ( #6796 )
...
* refactor: error handling clean-up and testing
* feat: allow partition filters to error + flag partitions
Closes #6748 .
2023-02-01 11:34:53 +00:00
9ec92eedbf
Merge branch 'main' into dom/0-retention
2023-02-01 11:10:44 +00:00
c8da5a8e07
refactor: extract early ID-based partition filters into new components ( #6794 )
...
This should greatly improve efficiency of the two filters ("by ID" -
used for mocking / tests / dev, "shard"). This also changes the metrics
slightly since the partitions filtered there no longer count into the
overall "backlog" (which makes sense for the two filters).
I've left the "never skipped" filter where it is, because it needs to
perform IO (i.e. shouldn't be done for all partitions upfront) and we
shall only have a few skipped partitions, so it's not such a big if
needlessly fetch the parquet files for them.
Closes #6783 .
2023-02-01 11:08:47 +00:00
7f363b55df
test(router): e2e namespace retention coverage
...
Assert the correct handling of 0 and negative retention periods when
interacting with the namespace create & update gRPC handlers.
2023-02-01 11:49:53 +01:00
ec5b35e3cc
fix: reject negative retention periods
...
Do not allow a negative retention period to be specified.
2023-02-01 11:29:56 +01:00
bf6ae3dc4d
fix: map retention period 0 values to NULL
...
Treat a retention period of 0 as "infinite", and not "none".
2023-02-01 11:29:13 +01:00
bfbd939ef3
Merge pull request #6793 from influxdata/dependabot/cargo/uuid-1.3.0
...
chore(deps): Bump uuid from 1.2.2 to 1.3.0
2023-02-01 09:47:11 +00:00
1a9c27cd9a
chore(deps): Bump uuid from 1.2.2 to 1.3.0
...
Bumps [uuid](https://github.com/uuid-rs/uuid ) from 1.2.2 to 1.3.0.
- [Release notes](https://github.com/uuid-rs/uuid/releases )
- [Commits](https://github.com/uuid-rs/uuid/compare/1.2.2...1.3.0 )
---
updated-dependencies:
- dependency-name: uuid
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-02-01 09:32:39 +00:00
27eb2f9096
Merge pull request #6791 from influxdata/dependabot/cargo/bytes-1.4.0
...
chore(deps): Bump bytes from 1.3.0 to 1.4.0
2023-02-01 09:28:57 +00:00
NGA-TRAN
Marco Neumann
Carol (Nichols || Goulding)
Dom
dependabot[bot]
CircleCI[bot]
Stuart Carnie
kodiakhq[bot]
Marko Mikulicic
Carol (Nichols || Goulding)