a566cda344
fix: Look up partitions in the cache first, then the bloom filter
2023-08-18 09:21:56 -04:00
755d461cca
docs: Clarify requirements of the `list_old_style` function
...
Namely, that it must not have a `LIMIT`.
Co-authored-by: Dom <dom@itsallbroken.com>
2023-08-18 09:21:55 -04:00
dd5e15d615
fix: Upsert partition when attempting to fetch the sort key
...
Rather than expecting the partition to exist in the catalog.
Fetching the sort key happens during ingester persist.
2023-08-18 09:21:11 -04:00
fc67aa0cb6
feat: Wrap the partition provider in the bloom filter
2023-08-18 09:19:25 -04:00
99d12ef600
feat: Add a catalog method to get all old-style partitions
...
To be fed to the bloom filter.
2023-08-18 09:19:25 -04:00
84468ad4ec
Merge pull request #8514 from influxdata/dom/gossip-topics
...
feat(gossip): topic subscriptions
2023-08-18 13:04:43 +01:00
09d88288e2
Merge branch 'main' into dom/gossip-topics
2023-08-18 12:57:17 +01:00
4eb120951c
Merge pull request #8516 from influxdata/cn/import-export-cleanup
...
fix: Clean up some tiny things in import-export
2023-08-18 11:11:50 +00:00
12eac3378f
Merge branch 'main' into cn/import-export-cleanup
2023-08-18 11:06:34 +00:00
d2c71bfe67
chore(deps): Bump thiserror from 1.0.46 to 1.0.47 ( #8519 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.46 to 1.0.47.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.46...1.0.47 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-18 09:02:48 +00:00
f266df7dd9
chore(deps): Bump clap from 4.3.21 to 4.3.22 ( #8520 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 4.3.21 to 4.3.22.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v4.3.21...v4.3.22 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-18 08:34:36 +00:00
ac3ded55e6
chore(deps): Bump ordered-float from 3.7.0 to 3.8.0 ( #8518 )
...
Bumps [ordered-float](https://github.com/reem/rust-ordered-float ) from 3.7.0 to 3.8.0.
- [Release notes](https://github.com/reem/rust-ordered-float/releases )
- [Commits](https://github.com/reem/rust-ordered-float/compare/v3.7.0...v3.8.0 )
---
updated-dependencies:
- dependency-name: ordered-float
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-18 08:07:18 +00:00
8f854f46f2
fix: Add missing negative in log message
2023-08-17 15:12:14 -04:00
c11253d8ad
fix: Break up long line
2023-08-17 15:12:14 -04:00
fc18840bba
fix: Remove some commented-out code
2023-08-17 15:12:13 -04:00
af8967f9e1
chore: Update DataFusion to get fix for string functions on tags ( #8479 )
...
* chore: Update DataFusion pin
* test: add test
* fix: Update test with correct query
2023-08-17 17:00:04 +00:00
0f3d393b39
fix: do not panic in Influx selectors ( #8513 )
...
Selectors that are value-based (like "min" and "max") and that also
track "other" fields (so things apart from the timestamp and the value),
panicked when the saw the same min/max value in different batches but
the timestamp did was NOT smaller than the previously selected value. An
example would be:
Batch | time | value | other
----- | ---- | ----- | ------
1 | 0 | 0 | 0
2 | 0 | 0 | 1
This adds the fix and a test.
Closes https://github.com/influxdata/idpe/issues/17968 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-17 15:45:05 +00:00
ca29e9b0d8
feat(gossip): topic support
...
Adds "topic" support, allowing a node to subscribe to one or more types
of application payloads independently.
A gossip node is optionally initialised with a set of topics (defaulting
to "all topics") and this set of topic interests is propagated
throughout the cluster via the usual PEX mechanism, alongside the
existing connection & identity information.
When broadcasting an application payload, the sender only transmits it
to nodes that had registered an interest in this payload type. This
prevents wasted network bandwidth and CPU for all nodes, and allows
multiple, distinct payload types to be propagated independently to keep
subsystems that rely on gossip decoupled from each other (no giant,
brittle payload enum type).
2023-08-17 14:53:40 +02:00
736f9987eb
feat(gossip): topic enum
...
Defines a Topic enum in the common generated_types package (alongside
the proto definitions of the actual gossip payloads).
This Topic enum will be used to identify message types across the gossip
transport.
2023-08-17 14:53:39 +02:00
1f0ac4fbcf
feat(gossip): topics & topic sets
...
Adds an (unused) Topic and TopicSet type to the gossip crate.
A Topic is a number from 0 to 63 inclusive that uniquely identifies a
"type" of application payload. The TopicSet efficiently encodes the set
of topics a node is interested in as a 64-bit wide bitmap, with each
Topic encoded as a single bit, shifted right N times for the topic
value.
This allows for very cheap "is this node interested" set evaluation and
space-efficient propagation of interest sets during PEX.
2023-08-17 14:53:39 +02:00
28c38120fc
fix: InfluxRPC error logging ( #8512 )
...
1. Use `Display` instead of `Debug` for error formatting. The latter one
is pretty verbose (it's basically Rust struct syntax) but also misses
a LOT of documentation / wording that is important. Errors should
always be printed using `Display` (and we do that basically
everywhere in our codebase).
2. Use structured logging.
Ref https://github.com/influxdata/idpe/issues/17967 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-17 12:45:01 +00:00
0a0ef66a05
chore: Make it clear there is only a single DataFusion memory pool ( #8501 )
...
* chore: Make it clear there is only a single DataFusion memory pool
* fix: assert there is a single bridge
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-17 12:35:31 +00:00
535ff5f0c8
refactor: extract InfluxRPC-specific code to `iox_query_influxrpc` (part 1) ( #8508 )
...
* refactor: replace test usage of `Predicate`
* refactor: remove dead code
* refactor: decouple recorg planning from InfluxRPC planning
* refactor: move InfluxRPC-specific scan plan construction
* refactor: move InfluxRPC-specific "missing columns" handling to `iox_query_influxrpc`
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-17 11:59:04 +00:00
7e2f85a24e
feat: accept relative memory size for memory pools ( #8503 )
...
Closes https://github.com/influxdata/idpe/issues/18006 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-17 11:53:27 +00:00
3612b1c482
refactor: use DF `Expr` instead of `Predicate` for chunk pruning ( #8500 )
...
`Predicate` is InfluxRPC specific and contains way more than just
filter expression.
Ref #8097 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-17 08:18:45 +00:00
7094189004
chore(deps): Bump tokio from 1.31.0 to 1.32.0 ( #8507 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.31.0 to 1.32.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.31.0...tokio-1.32.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-17 08:06:29 +00:00
6e210f6da2
chore(ingester): Better logging for disk full protection
2023-08-16 14:49:54 +01:00
1cc0926a7f
feat: track why bytes are written in compactor simulator ( #8493 )
...
* feat: add tracking of why bytes are written in simulator
* chore: enable breakdown of why bytes are written in a few larger tests
* chore: enable writes breakdown in another test
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-16 13:39:37 +00:00
0d009e3467
Merge pull request #8454 from influxdata/savage/remove-router-health-check-error-window-configuration
...
refactor(router): Revert configurable health-check `ERROR_WINDOW`
2023-08-16 11:38:50 +00:00
b6a4213b4f
Merge branch 'main' into savage/remove-router-health-check-error-window-configuration
2023-08-16 11:33:59 +00:00
ee5c8e7c08
Merge pull request #8497 from influxdata/dependabot/cargo/serde_json-1.0.105
...
chore(deps): Bump serde_json from 1.0.104 to 1.0.105
2023-08-16 12:33:37 +01:00
34f63a67dc
Merge branch 'main' into savage/remove-router-health-check-error-window-configuration
2023-08-16 11:33:32 +00:00
ddcc59b9d1
feat(ingester): Run WAL-based disk full capacity guarding task
...
This commit hooks up the `disk_full_protection` module's
`guard_disk_capacity` task with the disk space metrics reporter attached
to the `IngesterGuard`.
2023-08-16 12:30:24 +01:00
bdc60b89a9
feat(ingester): Disk full protection task
...
This commit adds a `disk_full_protection` module for the ingester that
contains a task for guarding disk capacity, by setting the ingest error
state when at 90% disk used up, rotating the WAL, draining & persisting
the buffer tree and then tidying up the fully persisted closed segment
files.
2023-08-16 12:30:23 +01:00
33900936da
chore(deps): Bump serde_json from 1.0.104 to 1.0.105
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.104 to 1.0.105.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.104...v1.0.105 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-08-16 11:00:44 +00:00
fff313b80c
chore(deps): Bump thiserror from 1.0.44 to 1.0.46 ( #8496 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.44 to 1.0.46.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.44...1.0.46 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-16 10:54:47 +00:00
c7c9b629c3
chore(deps): Bump flate2 from 1.0.26 to 1.0.27 ( #8495 )
...
Bumps [flate2](https://github.com/rust-lang/flate2-rs ) from 1.0.26 to 1.0.27.
- [Release notes](https://github.com/rust-lang/flate2-rs/releases )
- [Commits](https://github.com/rust-lang/flate2-rs/compare/1.0.26...1.0.27 )
---
updated-dependencies:
- dependency-name: flate2
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-16 08:53:03 +00:00
39a08fab69
feat: expose DataFusion mem pool metrics ( #8492 )
...
Closes #8466 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-15 15:19:11 +00:00
0bfa0a7b38
Merge pull request #8490 from influxdata/jrb_82_split_2ns_files
...
feat: allow compactor to split files covering 2ns
2023-08-15 08:38:00 -05:00
6d4729db1d
chore: Insta test updates
2023-08-14 15:41:37 -05:00
3003e44e78
feat: allow compactor to split 2ns files
2023-08-14 15:40:58 -05:00
964b2f6b97
fix: compactor simulator math error creates 0 byte files ( #8478 )
...
* fix: math error in simulator results in 0 byte files during simulations
* chore: insta churn from simulator file size fix
2023-08-14 20:00:19 +00:00
fd1b4051da
Merge pull request #8475 from influxdata/cn/querier-changes-for-bloom-filter
...
feat: Just the querier changes needed for using the bloom filter to avoid catalog requests in the ingester
2023-08-14 14:01:45 +00:00
f43ab88148
Merge branch 'main' into cn/querier-changes-for-bloom-filter
2023-08-14 13:56:35 +00:00
a13ff617a4
refactor: decouple runtimes in `CachedObjectStore` ( #8474 )
...
This should prevent the CPU-bound DataFusion runtime from stalling our
main IO runtime.
This is similar to:
- `iox_query::exec::cross_rt_stream`
- https://github.com/apache/arrow-rs/pull/4015
- https://github.com/apache/arrow-rs/pull/4040
Note: I currently have no concrete evidence that this is an issue, but
worker stalling in tokio is really hard to debug and I would like to be
better safe than sorry.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-14 11:49:32 +00:00
ad4068bbea
refactor: decouple `QueryNamespace` from synchronous schema interface (`QueryNamespaceMeta`) ( #8472 )
...
* refactor: remove unused impl
* refactor: inline `ExecutionContextProvider` into `QueryNamespace`
* refactor: use global `DEFAULT_SCHEMA`
* refactor: decouple `QueryNamespace` from `QueryNamespaceMeta`
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-14 08:09:08 +00:00
86ef5e94c6
chore(deps): Bump log from 0.4.19 to 0.4.20 ( #8484 )
...
Bumps [log](https://github.com/rust-lang/log ) from 0.4.19 to 0.4.20.
- [Release notes](https://github.com/rust-lang/log/releases )
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/log/compare/0.4.19...0.4.20 )
---
updated-dependencies:
- dependency-name: log
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-14 08:00:46 +00:00
2a3b77e6c7
chore(deps): Bump sysinfo from 0.29.7 to 0.29.8 ( #8485 )
...
Bumps [sysinfo](https://github.com/GuillaumeGomez/sysinfo ) from 0.29.7 to 0.29.8.
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md )
- [Commits](https://github.com/GuillaumeGomez/sysinfo/commits )
---
updated-dependencies:
- dependency-name: sysinfo
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-14 07:37:36 +00:00
838c274618
chore(deps): Bump bitflags from 2.3.3 to 2.4.0 ( #8483 )
...
Bumps [bitflags](https://github.com/bitflags/bitflags ) from 2.3.3 to 2.4.0.
- [Release notes](https://github.com/bitflags/bitflags/releases )
- [Changelog](https://github.com/bitflags/bitflags/blob/main/CHANGELOG.md )
- [Commits](https://github.com/bitflags/bitflags/compare/2.3.3...2.4.0 )
---
updated-dependencies:
- dependency-name: bitflags
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-14 06:48:06 +00:00
34b8585931
chore(deps): Bump tokio from 1.30.0 to 1.31.0 ( #8482 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.30.0 to 1.31.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.30.0...tokio-1.31.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-14 06:32:34 +00:00
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
Dom
kodiakhq[bot]
dependabot[bot]
Andrew Lamb
Marco Neumann
Fraser Savage
Joe-Blount
Joe-Blount