30939cfe96
refactor(wal): Remove op-level `sequence_number`, use per table map
...
This commit removes the op-level sequence number from the proto
definition, now reading and writing solely to the per table (and thus
per partition) sequence number map. Tables/partitions within the same
write op are still assigned the same number for now, so there should be
no semantic different
2023-07-05 14:20:43 +01:00
3827257f94
chore(deps): Bump thiserror from 1.0.40 to 1.0.41 ( #8149 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.40 to 1.0.41.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.40...1.0.41 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-07-05 09:25:14 +00:00
7c771ce6f3
chore(deps): Bump serde_json from 1.0.99 to 1.0.100 ( #8147 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.99 to 1.0.100.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.99...v1.0.100 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-05 09:12:11 +00:00
b827b76583
chore(deps): Bump console-subscriber from 0.1.9 to 0.1.10 ( #8142 )
...
Bumps [console-subscriber](https://github.com/tokio-rs/console ) from 0.1.9 to 0.1.10.
- [Release notes](https://github.com/tokio-rs/console/releases )
- [Commits](https://github.com/tokio-rs/console/compare/console-subscriber-v0.1.9...console-subscriber-v0.1.10 )
---
updated-dependencies:
- dependency-name: console-subscriber
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-04 12:13:49 +00:00
71795f265f
chore(deps): Bump serde from 1.0.164 to 1.0.166 ( #8140 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.164 to 1.0.166.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.164...v1.0.166 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-04 10:56:56 +00:00
70b44f78ee
test: correctly decode ingester reponses in end2end tests
2023-07-03 17:25:01 +02:00
b1a4e3955e
test: `ingester_partition_pruning` must perform type coercion
2023-07-03 17:25:00 +02:00
cd28bf0337
test: Query an ingester with a predicate that should prune partitions
2023-07-03 17:24:58 +02:00
e8065f871a
Merge branch 'main' into dom/no-rustflags-mess
2023-07-03 15:57:46 +01:00
e5a9e1534a
test: assert 1 file persisted
...
There should be a single file persisted during graceful shutdown.
2023-07-03 15:51:02 +02:00
5d0c172e61
test(e2e): query shutdown-persisted files
...
Ensure buffered ingester data is persisted and remains queryable after a
graceful ingester shutdown.
2023-07-03 15:51:02 +02:00
3a8d234b1c
build: optional tokio_unstable support
...
This makes using the tokio_unstable feature optional - the build
configuration defines what is enabled, rather than the other way around.
By default, and for prod builds and users who have not set a RUSTFLAGS,
the tokio_unstable is enabled and this is leveraged to provide the
metrics exposed by the flag. For builds where tokio_unstable is not
enabled, these metrics are not included, without causing a compilation
error.
CI will against the flag-enabled/prod code by virtue of being the
default.
2023-07-03 15:28:59 +02:00
0cf7d06b9d
chore(deps): Bump num_cpus from 1.15.0 to 1.16.0 ( #8115 )
...
Bumps [num_cpus](https://github.com/seanmonstar/num_cpus ) from 1.15.0 to 1.16.0.
- [Release notes](https://github.com/seanmonstar/num_cpus/releases )
- [Changelog](https://github.com/seanmonstar/num_cpus/blob/master/CHANGELOG.md )
- [Commits](https://github.com/seanmonstar/num_cpus/compare/v1.15.0...v1.16.0 )
---
updated-dependencies:
- dependency-name: num_cpus
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-06-30 08:32:43 +00:00
ca31c1eade
feat: hook up tokio metrics ( #8050 )
...
* feat: metrics for main tokio runtime
* feat: instrument executor tokio runtime
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-06-29 11:11:44 +00:00
4638b89d93
refactor: migrate retention to proper predicates ( #8092 )
...
Do not (ab)use per-chunk delete predicates for the retention policy.
Instead use a per-table predicate.
This makes the code way cleaner, since the scoping is correct (i.e.
delete predicates are a table-wide attribute, not a chunk-based one) and
it is consistent time predicates that the user providers (e.g. via
`WHERE time > x`).
It also allows us to remove delete predicates (in their current,
non-scalable form) from the query path. A potential future version would
likely not use per chunk predicates (and "is processed" markers) but use
the timestamp / chunk order to determine to which data the predicate
should be applied.
Note that the lowering of the retention policy changed slightly from
```text
(time > (now() - retention)) AND (time < MAX)
```
to
```text
time > (now() - retention)
```
Since the `MAX` cut is just an artifact of the lowering and was unnecessary.
Closes #7409 .
Closes #7410 .
2023-06-29 08:36:37 +00:00
511a0bae78
feat(influxql): add derivative and non_negative_derivative ( #8103 )
...
Add the DERIVATIVE and NON_NEGATIVE_DERIVATIVE functions to influxql.
These are used to calculate derivatives over arbitrary time units.
The implementation is modeled after the DIFFERENCE and
NON_NEGATIVE_DIFFERENCE functions, with a difference that the unit
parameters is a configuration of the user-defined aggregator function
and therefore there cannot be a single shared definition of the
function.
The NON_NEGATIVE_DIFFERENCE function implementation has been
refactored to be an arbitrary NON_NEGATIVE wrapper for any Accumulator
function.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-06-29 05:53:18 +00:00
3a8a8a153e
feat(idpe 17789): provide scheduler interface ( #8057 )
...
* feat: provide convenience methods to create Scheduler, and keep the scheduler implementations crate private. External crates can only create a Scheduler based upon configs.
* feat: provide Scheduler as a component to compactor. Specifically, the scheduler configs are present within the compactor run config, and the scheduler in created within the compactor hardcoded components.
* feat: within the compactor ScheduledPartitionsSource, utilize the dyn Scheduler and Scheduler.get_jobs()
* feat: CompactionJob should be per partition, and have a uniqueness characteristic independent of the partition
* feat: keep compactor_scheduler separate from clap_blocks. Only interface is within ioxd_compactor where the CLI configs are transformed into ShardConfig and PartitionsSourceConfig.
* chore: make IdOnlyPartitionFilter into only pub(crate)
* chore: update scheduler display to include any report information (a.k.a. shard_config, if present)
2023-06-28 15:04:00 -07:00
33be68477f
fix: Use modern form of read_window_aggregate ( #8098 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-06-28 17:49:17 +00:00
b15c6062a9
chore(deps): Bump tokio from 1.28.2 to 1.29.0 ( #8100 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.28.2 to 1.29.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.28.2...tokio-1.29.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-28 13:18:08 +00:00
54aaafd496
chore(deps): Bump rustyline from 11.0.0 to 12.0.0 ( #8078 )
...
Bumps [rustyline](https://github.com/kkawakam/rustyline ) from 11.0.0 to 12.0.0.
- [Release notes](https://github.com/kkawakam/rustyline/releases )
- [Changelog](https://github.com/kkawakam/rustyline/blob/master/History.md )
- [Commits](https://github.com/kkawakam/rustyline/compare/v11.0.0...v12.0.0 )
---
updated-dependencies:
- dependency-name: rustyline
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-26 10:05:29 +00:00
24ca0fef30
chore(deps): Bump serde_json from 1.0.97 to 1.0.99 ( #8076 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.97 to 1.0.99.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.97...v1.0.99 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-26 09:25:38 +00:00
c14d8551e5
refactor: set up jemalloc metrics in a more central place ( #8062 )
...
We already have a method that adds some default metrics / instruments to
the metric registry. Use that for jemalloc as well. This makes it easier
to follow how metrics are setup up for our prod binary.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-06-23 10:53:27 +00:00
74a48a8f63
chore(deps): Bump itertools from 0.10.5 to 0.11.0 ( #8060 )
...
* chore(deps): Bump itertools from 0.10.5 to 0.11.0
Bumps [itertools](https://github.com/rust-itertools/itertools ) from 0.10.5 to 0.11.0.
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-itertools/itertools/compare/v0.10.5...v0.11.0 )
---
updated-dependencies:
- dependency-name: itertools
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-06-23 08:11:56 +00:00
178483c1a0
feat: basic non-aggregates w/ InfluxQL selector functions ( #8016 )
...
* test: ensure that selectors check arg count
* feat: basic non-aggregates w/ InfluxQL selector functions
See #7533 .
* refactor: clean up code
* feat: get more advanced cases to work
* docs: remove stale comments
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-06-23 08:05:50 +00:00
cb2968f0ef
Merge branch 'main' into dom/compactor-query-rate-limit
2023-06-22 13:08:24 +01:00
7b4a1a0660
chore: PR feedback
...
Add tests for fewer rows than N for `moving_average`
See: https://github.com/influxdata/influxdb_iox/pull/8023#discussion_r1237298376
2023-06-22 12:15:47 +10:00
13726c2a76
Merge branch 'main' into sgc/issue/7600_moving_average
2023-06-22 10:10:22 +10:00
d1cbbd27b1
feat(compactor): config partition query rate limit
...
Allow the partition fetch queries to be (optionally) rate limited via
runtime config.
2023-06-21 15:50:12 +02:00
83a5037e61
feat: query support for custom partitioning ( #8025 )
...
* feat: querier-specific stat creation routine
* feat: prune querier chunks using partition col ranges
* feat: add table client
* test: custom partitioning
* fix: correctly set up stats for chunks with col subsets
* fix: flaky test
* refactor: remove obsolete dead_code markers
* feat: add partition template to `create_namespace`
* test: extend custom partitioning end2end tests
* fix: explain shuffling, make it actual deterministic
2023-06-21 09:03:19 +00:00
2cbaf9cffa
chore: more tests, renamed avg_n → moving_average
2023-06-21 15:05:08 +10:00
edaac28498
Merge branch 'main' into sgc/issue/7600_moving_average
2023-06-21 11:39:06 +10:00
34b5fadde0
refactor: move scheduler related configs to compactor_scheduler ( #8013 )
2023-06-20 09:55:35 -07:00
123ba32fb9
chore(deps): Bump serde_json from 1.0.96 to 1.0.97 ( #8021 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.96 to 1.0.97.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.96...v1.0.97 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-20 08:53:15 +00:00
a2521bbf35
feat: moving_average, difference and non_negative_difference
...
There is a `todo` regarding `update_batch` to be discussed with @alamb
2023-06-20 16:37:28 +10:00
8670b28445
Merge branch 'main' into sgc/issue/7600_moving_average
2023-06-18 09:41:19 +10:00
7a1f54ac64
refactor: remove compactor type ( #8011 )
...
* refactor: remove cold compactions
* refactor: remove compaction_type
2023-06-16 09:40:13 -07:00
5889c96501
chore: Update `datafusion` and other dependencies ( #7981 )
...
* chore: Update DatFaFusion pin
* chore: Update other dependencies
* chore: Update hakari
* fix: Update for API changes
* fix: Update explain plan
* fix: Update influxql plans
* fix: rustdoc links
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-06-16 09:48:55 +00:00
2407be8062
feat: trialed retractable UDAF
...
Unfortunately, this is not suitable when the source data has nulls,
as InfluxQL OG ignores these values.
2023-06-16 13:10:47 +10:00
5d0bb68c5b
chore: add compactor option to disable scratchpad ( #7995 )
2023-06-15 14:35:55 +00:00
3d8e06f6e9
fix: influxdb_iox sql command warns on flight healthcheck failure
2023-06-15 00:17:29 +02:00
335d9f7357
chore: minimize proptest features ( #7993 )
2023-06-14 12:28:18 +00:00
51f45fd710
refactor(cli): Simplify and improve sequence range prop test
...
Co-authored-by: Dom <dom@itsallbroken.com>
2023-06-12 15:14:08 +01:00
71e47b59ab
refactor(wal): Make more use of combinators for WAL segment reading logic
2023-06-12 12:27:20 +01:00
e809aadfe5
chore(cli): Fix error typo in `wal inspect` test
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2023-06-12 10:12:18 +01:00
73c0c28bd0
feat(cli): Add `influxdb_iox debug wal inspect` command
...
This commit adds an `inspect` command to read through the sequenced
operations in a WAL file and debug pretty print their contents to
stdout, optionally filtering by a sequence number range.
2023-06-09 18:16:57 +01:00
d26ad8e079
feat: Allow passing service protection limits in create db gRPC call ( #7941 )
...
* feat: Allow passing service protection limits in create db gRPC call
* fix: Move the impl into the catalog namespace trait
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-06-08 14:28:32 +00:00
5acca21f12
chore(deps): Bump serde from 1.0.163 to 1.0.164 ( #7951 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.163 to 1.0.164.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.163...v1.0.164 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-06-08 09:20:19 +00:00
17c0d837b3
chore: Update DataFusion, arrow, object_store pins ( #7942 )
...
* chore: Update DataFusion, arrow, object_store pins
* chore: Update for hakari
* chore: Update for new APIs
* fix: update test
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-06-07 17:08:31 +00:00
ccd11b52b0
chore(deps): Bump nu-ansi-term from 0.47.0 to 0.48.0 ( #7936 )
...
Bumps [nu-ansi-term](https://github.com/nushell/nu-ansi-term ) from 0.47.0 to 0.48.0.
- [Release notes](https://github.com/nushell/nu-ansi-term/releases )
- [Changelog](https://github.com/nushell/nu-ansi-term/blob/main/CHANGELOG.md )
- [Commits](https://github.com/nushell/nu-ansi-term/compare/v0.47.0...v0.48.0 )
---
updated-dependencies:
- dependency-name: nu-ansi-term
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-06-07 09:02:33 +00:00
7b6efae62c
chore(deps): Bump tempfile from 3.5.0 to 3.6.0
...
Bumps [tempfile](https://github.com/Stebalien/tempfile ) from 3.5.0 to 3.6.0.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md )
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.5.0...v3.6.0 )
---
updated-dependencies:
- dependency-name: tempfile
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-06-07 08:21:40 +00:00
Fraser Savage
dependabot[bot]
Marco Neumann
Carol (Nichols || Goulding)
Dom
Martin Hilton
wiedld
Andrew Lamb
Stuart Carnie
Joe-Blount
Marko Mikulicic