Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
49,680 Commits
378 Branches
478 Tags
379 MiB
Commit Graph

2797 Commits (3.0)

Author SHA1 Message Date
dependabot[bot] 0973f60d5c
chore(deps): Bump clap from 4.3.19 to 4.3.21 (#8456) 
Bumps [clap](https://github.com/clap-rs/clap) from 4.3.19 to 4.3.21.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.3.19...v4.3.21)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-08-09 08:27:09 +00:00
Andrew Lamb 46bfa0badc
chore: Update DataFusion (#8447) 
* chore: Update DataFusion pin

* chore: Update for API changes
 2023-08-08 13:41:14 +00:00
dependabot[bot] 71b32fbdba
chore(deps): Bump cc from 1.0.81 to 1.0.82 (#8449) 
Bumps [cc](https://github.com/rust-lang/cc-rs) from 1.0.81 to 1.0.82.
- [Release notes](https://github.com/rust-lang/cc-rs/releases)
- [Commits](https://github.com/rust-lang/cc-rs/compare/1.0.81...1.0.82)

---
updated-dependencies:
- dependency-name: cc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-08-08 08:54:29 +00:00
Marco Neumann 8966cc88df
fix: `O(n^2)` delete handling in LRU (#8416) 
* test: add regression test

* feat: `AddressableHeap::iter`

* fix: `O(n^2)` delete handling in LRU

* refactor: address review comments

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-08-07 13:46:03 +00:00
dependabot[bot] db1429b37d
chore(deps): Bump serde from 1.0.181 to 1.0.183 (#8431) 
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.181 to 1.0.183.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.181...v1.0.183)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-08-07 10:24:11 +00:00
dependabot[bot] 548a68a71a
chore(deps): Bump rustix from 0.38.6 to 0.38.7 (#8430) 
Bumps [rustix](https://github.com/bytecodealliance/rustix) from 0.38.6 to 0.38.7.
- [Release notes](https://github.com/bytecodealliance/rustix/releases)
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.38.6...v0.38.7)

---
updated-dependencies:
- dependency-name: rustix
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-08-07 10:13:59 +00:00
dependabot[bot] 0e478af470
chore(deps): Bump tempfile from 3.7.0 to 3.7.1 (#8429) 
Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.7.0 to 3.7.1.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.7.0...v3.7.1)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-08-07 09:21:11 +00:00
dependabot[bot] b62e9bb0c8
chore(deps): Bump pin-project from 1.1.2 to 1.1.3 (#8427) 
Bumps [pin-project](https://github.com/taiki-e/pin-project) from 1.1.2 to 1.1.3.
- [Release notes](https://github.com/taiki-e/pin-project/releases)
- [Changelog](https://github.com/taiki-e/pin-project/blob/main/CHANGELOG.md)
- [Commits](https://github.com/taiki-e/pin-project/compare/v1.1.2...v1.1.3)

---
updated-dependencies:
- dependency-name: pin-project
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-08-07 09:09:23 +00:00
dependabot[bot] e69b2c40c6
chore(deps): Bump regex from 1.9.1 to 1.9.3 (#8426) 
Bumps [regex](https://github.com/rust-lang/regex) from 1.9.1 to 1.9.3.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.9.1...1.9.3)

---
updated-dependencies:
- dependency-name: regex
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-08-07 09:01:07 +00:00
dependabot[bot] ae6d416ad4
chore(deps): Bump filetime from 0.2.21 to 0.2.22 (#8425) 
Bumps [filetime](https://github.com/alexcrichton/filetime) from 0.2.21 to 0.2.22.
- [Commits](https://github.com/alexcrichton/filetime/compare/0.2.21...0.2.22)

---
updated-dependencies:
- dependency-name: filetime
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-08-07 08:18:54 +00:00
dependabot[bot] 758c42305d
chore(deps): Bump regex-automata from 0.3.4 to 0.3.6 (#8424) 
Bumps [regex-automata](https://github.com/rust-lang/regex) from 0.3.4 to 0.3.6.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/regex-automata-0.3.4...regex-automata-0.3.6)

---
updated-dependencies:
- dependency-name: regex-automata
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-08-07 08:05:01 +00:00
dependabot[bot] 5530d90074
chore(deps): Bump serde from 1.0.180 to 1.0.181 (#8413) 
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.180 to 1.0.181.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.180...v1.0.181)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-08-04 08:11:16 +00:00
CircleCI[bot] 5e00650413 chore: Run cargo hakari tasks 2023-08-03 14:44:06 +00:00
Dom Dwyer 03f7025211
perf: minimise partition catalog queries 
This commit implements a PartitionProvider decorator that
probabilistically determines if a partition is going to be a an
"old-style" row-addressed partition created prior to #7963, or a
"new-style" hash-addressed partition created after using a fast,
space-efficient, compressed bloom filter.

If a partition is identified as a new-style, hash-addressed partition,
the PartitionData is immediately initialised using the deterministic
hash ID without performing a catalog query at all.

If a partition is identified as an old-style, row-addressed partition, a
catalog query is performed to resolve the row ID as it would without
this filter.

A new-style, hash-addressed partition may sometimes be incorrectly
identified as a row-addressed partition, causing a spurious catalog
query, which is then correctly identified as a hash-addressed partition.
This is tuned to happen ~1-0.1% of the time, eliminating 99% to 99.9% of
unnecessary catalog queries.
 2023-08-03 16:40:38 +02:00
dependabot[bot] b386f68c11
chore(deps): Bump rustls from 0.21.5 to 0.21.6 (#8401) 
Bumps [rustls](https://github.com/rustls/rustls) from 0.21.5 to 0.21.6.
- [Release notes](https://github.com/rustls/rustls/releases)
- [Commits](https://github.com/rustls/rustls/compare/v/0.21.5...v/0.21.6)

---
updated-dependencies:
- dependency-name: rustls
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-08-03 09:40:49 +00:00
dependabot[bot] c159e99144
chore(deps): Bump regex-automata from 0.3.3 to 0.3.4 (#8403) 
Bumps [regex-automata](https://github.com/rust-lang/regex) from 0.3.3 to 0.3.4.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/regex-automata-0.3.3...regex-automata-0.3.4)

---
updated-dependencies:
- dependency-name: regex-automata
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-08-03 09:35:19 +00:00
dependabot[bot] 4469b16d1f
chore(deps): Bump rustix from 0.38.4 to 0.38.6 (#8402) 
Bumps [rustix](https://github.com/bytecodealliance/rustix) from 0.38.4 to 0.38.6.
- [Release notes](https://github.com/bytecodealliance/rustix/releases)
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.38.4...v0.38.6)

---
updated-dependencies:
- dependency-name: rustix
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-08-03 09:30:06 +00:00
dependabot[bot] f674832a32
chore(deps): Bump cc from 1.0.79 to 1.0.81 
Bumps [cc](https://github.com/rust-lang/cc-rs) from 1.0.79 to 1.0.81.
- [Release notes](https://github.com/rust-lang/cc-rs/releases)
- [Commits](https://github.com/rust-lang/cc-rs/compare/1.0.79...1.0.81)

---
updated-dependencies:
- dependency-name: cc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-08-03 01:48:36 +00:00
Andrew Lamb 6e13ff8cb8
chore: Update DataFusion pin (#8390) 
* chore: Update DataFusion pin

* chore: Update for API

* fix: update plans

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-08-02 14:58:16 +00:00
Marco Neumann 9e4e205ffd
refactor: migration checksum type (#8388) 
* refactor: use `Box<[...]>` instead of `Vec<...>`

We are not planning to modify the vector, so storing a capacity and a
length is somewhat pointless.

* feat: add printout test for PG migrations

* refactor: use dedicated checksum type

* feat: checksum string roundtrips

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-08-02 11:57:18 +00:00
dependabot[bot] 5004f3e460
chore(deps): Bump cc from 1.0.79 to 1.0.80 (#8386) 
Bumps [cc](https://github.com/rust-lang/cc-rs) from 1.0.79 to 1.0.80.
- [Release notes](https://github.com/rust-lang/cc-rs/releases)
- [Commits](https://github.com/rust-lang/cc-rs/compare/1.0.79...1.0.80)

---
updated-dependencies:
- dependency-name: cc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-08-02 08:29:07 +00:00
dependabot[bot] 72feefc3cc
chore(deps): Bump serde from 1.0.179 to 1.0.180 (#8376) 
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.179 to 1.0.180.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.179...v1.0.180)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-08-01 08:24:30 +00:00
Andrew Lamb de79619e71
chore: Update datafusion (#8355) 
* chore: Update datafusion pin

* fix: Update for change in API

* chore: Update plan

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-31 15:41:00 +00:00
Dom e98188b181
Merge branch 'main' into dom/enable-gossip 2023-07-31 15:27:44 +01:00
Marco Neumann aa7a38be55
fix: re-design LRU cache to be deadlock-free (#8345) 
* fix: re-design LRU cache to be deadlock-free

Fixes #8334.

* test: explain test

* test: add regression test

* docs: extend "overdelete" section
 2023-07-31 13:04:34 +00:00
Fraser Savage 8e0cee8e73
refactor(tracker): Return disk usage watcher from `DiskUsageMetrics` 
This allows the creator to pass around a handle to the latest observed
disk usage statistics, allowing other threads to act upon changes.
 2023-07-31 12:14:13 +01:00
Dom Dwyer 8da08fa574
feat(router): optionally enable gossip subsystem 
Allows the router to optionally enable and start the gossip subsystem
(disabled by default).

No code uses the gossip system, so no application-level messages are
exchanged, but this allows the gossip subsystem to run and exchange
control frames / perform discovery / etc.
 2023-07-31 11:01:30 +02:00
Dom Dwyer d8badbe9ca
feat(ingester): optionally enable gossip subsystem 
Allows the ingester to optionally enable and start the gossip subsystem
(disabled by default).

No code uses the gossip system, so no application-level messages are
exchanged, but this allows the gossip subsystem to run and exchange
control frames / perform discovery / etc.
 2023-07-31 11:01:30 +02:00
dependabot[bot] 4fa6ead27d
chore(deps): Bump serde from 1.0.177 to 1.0.179 
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.177 to 1.0.179.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.177...v1.0.179)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-07-31 08:38:48 +00:00
dependabot[bot] ec8033a65b
chore(deps): Bump regex-automata from 0.3.3 to 0.3.4 (#8370) 
Bumps [regex-automata](https://github.com/rust-lang/regex) from 0.3.3 to 0.3.4.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/regex-automata-0.3.3...regex-automata-0.3.4)

---
updated-dependencies:
- dependency-name: regex-automata
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-31 08:34:24 +00:00
dependabot[bot] 940f441259
chore(deps): Bump tikv-jemalloc-ctl from 0.5.0 to 0.5.4 (#8359) 
Bumps [tikv-jemalloc-ctl](https://github.com/tikv/jemallocator) from 0.5.0 to 0.5.4.
- [Release notes](https://github.com/tikv/jemallocator/releases)
- [Changelog](https://github.com/tikv/jemallocator/blob/main/CHANGELOG.md)
- [Commits](https://github.com/tikv/jemallocator/compare/0.5.0...0.5.4)

---
updated-dependencies:
- dependency-name: tikv-jemalloc-ctl
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
 2023-07-28 12:45:43 +00:00
dependabot[bot] 29d66444e5
chore(deps): Bump serde from 1.0.176 to 1.0.177 
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.176 to 1.0.177.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.176...v1.0.177)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-07-28 10:40:15 +00:00
dependabot[bot] 91baa9bab7
chore(deps): Bump tikv-jemalloc-sys (#8361) 
Bumps [tikv-jemalloc-sys](https://github.com/tikv/jemallocator) from 0.5.3+5.3.0-patched to 0.5.4+5.3.0-patched.
- [Release notes](https://github.com/tikv/jemallocator/releases)
- [Changelog](https://github.com/tikv/jemallocator/blob/main/CHANGELOG.md)
- [Commits](https://github.com/tikv/jemallocator/compare/tikv-jemalloc-sys-0.5.3...0.5.4)

---
updated-dependencies:
- dependency-name: tikv-jemalloc-sys
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
 2023-07-28 10:38:16 +00:00
dependabot[bot] 854c4c25e9
chore(deps): Bump sysinfo from 0.29.6 to 0.29.7 (#8341) 
Bumps [sysinfo](https://github.com/GuillaumeGomez/sysinfo) from 0.29.6 to 0.29.7.
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GuillaumeGomez/sysinfo/commits)

---
updated-dependencies:
- dependency-name: sysinfo
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-27 09:22:52 +00:00
dependabot[bot] ac810aab8a
chore(deps): Bump serde from 1.0.175 to 1.0.176 (#8343) 
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.175 to 1.0.176.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.175...v1.0.176)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-27 09:17:39 +00:00
dependabot[bot] 700830d7d3
chore(deps): Bump serde_json from 1.0.103 to 1.0.104 (#8342) 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.103 to 1.0.104.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.103...v1.0.104)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-27 09:12:35 +00:00
dependabot[bot] cdaa3bc720
chore(deps): Bump pprof from 0.12.0 to 0.12.1 
Bumps [pprof](https://github.com/tikv/pprof-rs) from 0.12.0 to 0.12.1.
- [Changelog](https://github.com/tikv/pprof-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tikv/pprof-rs/commits)

---
updated-dependencies:
- dependency-name: pprof
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-07-27 01:54:25 +00:00
wiedld fe3897dd62
Merge branch 'main' into idpe-17789/compaction-job-in-compactor 2023-07-26 10:45:09 -07:00
Marco Neumann b883c7c554
chore: manual cargo update (#8328) 
* chore: manual cargo update

Dependabot seemed to have fallen behind a bit.

```console
❯ cargo update
    Updating crates.io index
    Updating git repository `https://github.com/apache/arrow-datafusion.git`
    Updating git repository `https://github.com/mkmik/heappy`
    Updating allocator-api2 v0.2.15 -> v0.2.16
    Updating anyhow v1.0.71 -> v1.0.72
    Updating async-compression v0.4.0 -> v0.4.1
    Updating axum v0.6.18 -> v0.6.19
    Updating blake3 v1.4.0 -> v1.4.1
    Updating bstr v1.5.0 -> v1.6.0
    Updating constant_time_eq v0.2.6 -> v0.3.0
    Updating cpufeatures v0.2.8 -> v0.2.9
    Updating dashmap v5.4.0 -> v5.5.0
    Updating equivalent v1.0.0 -> v1.0.1
    Updating http-range-header v0.3.0 -> v0.3.1
    Updating hyper-rustls v0.24.0 -> v0.24.1
    Updating itoa v1.0.7 -> v1.0.9
    Updating num v0.4.0 -> v0.4.1
    Updating pest v2.7.0 -> v2.7.1
    Updating pest_derive v2.7.0 -> v2.7.1
    Updating pest_generator v2.7.0 -> v2.7.1
    Updating pest_meta v2.7.0 -> v2.7.1
    Updating proc-macro2 v1.0.63 -> v1.0.66
    Updating quote v1.0.29 -> v1.0.32
    Updating rustversion v1.0.12 -> v1.0.14
    Updating ryu v1.0.13 -> v1.0.15
    Updating semver v1.0.17 -> v1.0.18
    Updating seq-macro v0.3.3 -> v0.3.5
    Updating stringprep v0.1.2 -> v0.1.3
    Updating strum_macros v0.25.0 -> v0.25.1
    Updating symbolic-common v12.2.0 -> v12.3.0
    Updating symbolic-demangle v12.2.0 -> v12.3.0
    Updating syn v2.0.26 -> v2.0.27
    Updating toml_edit v0.19.12 -> v0.19.14
    Updating ucd-trie v0.1.5 -> v0.1.6
    Updating unicode-ident v1.0.9 -> v1.0.11
    Updating winnow v0.4.7 -> v0.5.1
```

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2023-07-25 09:56:55 +00:00
wiedld 82d55fd6b6 refactor(idpe-17789): move PartitionsSource to be separate traits in scheduler vs compactor 2023-07-24 14:43:52 -07:00
wiedld 02088995b2
feat(idpe 17789): compactor to scheduler communication. `update_job_status()` and `end_job()` (#8216) 
* feat(idpe-17789): scheduler job_status() (#8121)

This block of work moves into the scheduler some of the specific downstream actions affiliated with compaction outcomes. Which responsibilities stay in the compactor, versus moved to the scheduler, roughly followed the heuristic of whether the action (a) had an impact on global catalog state (a.k.a. commits and partition skipping), (b) whether it's logging affiliated with compactor health (e.g. ParitionDoneSink logging outcomes) versus system health (e.g. logging commits), and (c) reporting to the scheduler on any errors encountered during compaction. This boundary is subject to change as we move forward.

Also, a noted caveat (TODO) on this commit. We have a CompactionJob which is used to track work handed off to each compactor. Currently it still uses the partition_id for tracking, but the followup PR will start moving the compactor to have more CompactionJob uuid awareness.

* fix(idpe-17789): need to remove partition from uniqueness tracking, so it becomes available again

* refactor(idpe-17789): split up the single-use end_job() from the multi-use update_job_status()

* feat(idpe-17789): Commit is now a scheduler trait, only used externally in the compactor_test_utils

* feat(idpe-17789): Propagate errors pertaining to commit, in both the scheduler and the compactor.

* feat(idpe-17789): PartitionDoneSink should have different crate-private traits for scheduler versus comactor.

* feat(idpe-17789): PartitionDoneSink should propagate errors

* test(idpe-17789): integration tests suite

* test(idpe-17789): test documenting what skip request does (as outcome)

* refactor(idpe-17789): make the validate of the upgrade commit, versus replacement commit, more explicit.

* feat(idpe-17789): switch to using parking_lot Mutex within the scheduler
 2023-07-24 12:01:28 -07:00
wiedld bf1e28ba96
refactor: implementations of PartitionsSource in compactor_scheduler should use the parking lot mutex (#8309) 2023-07-24 11:29:20 -07:00
dependabot[bot] fca624a039
chore(deps): Bump sqlparser from 0.36.0 to 0.36.1 (#8312) 
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs) from 0.36.0 to 0.36.1.
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.36.0...v0.36.1)

---
updated-dependencies:
- dependency-name: sqlparser
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-24 10:28:23 +00:00
dependabot[bot] b7ab20be0d
chore(deps): Bump either from 1.8.1 to 1.9.0 (#8314) 
Bumps [either](https://github.com/bluss/either) from 1.8.1 to 1.9.0.
- [Commits](https://github.com/bluss/either/compare/1.8.1...1.9.0)

---
updated-dependencies:
- dependency-name: either
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-24 10:23:38 +00:00
dependabot[bot] faa8d44492
chore(deps): Bump thiserror from 1.0.43 to 1.0.44 (#8315) 
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.43 to 1.0.44.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.43...1.0.44)

---
updated-dependencies:
- dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-24 10:18:44 +00:00
dependabot[bot] c6df58b30c
chore(deps): Bump nu-ansi-term from 0.48.0 to 0.49.0 (#8316) 
Bumps [nu-ansi-term](https://github.com/nushell/nu-ansi-term) from 0.48.0 to 0.49.0.
- [Release notes](https://github.com/nushell/nu-ansi-term/releases)
- [Changelog](https://github.com/nushell/nu-ansi-term/blob/main/CHANGELOG.md)
- [Commits](https://github.com/nushell/nu-ansi-term/compare/v0.48.0...v0.49.0)

---
updated-dependencies:
- dependency-name: nu-ansi-term
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-24 10:12:56 +00:00
dependabot[bot] cd31492e5b
chore(deps): Bump async-trait from 0.1.71 to 0.1.72 (#8317) 
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.71 to 0.1.72.
- [Release notes](https://github.com/dtolnay/async-trait/releases)
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.71...0.1.72)

---
updated-dependencies:
- dependency-name: async-trait
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-24 10:07:18 +00:00
dependabot[bot] 1d1cc86912
chore(deps): Bump serde from 1.0.173 to 1.0.175 (#8318) 
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.173 to 1.0.175.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.173...v1.0.175)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
 2023-07-24 10:02:07 +00:00
dependabot[bot] 79bb1347e6
chore(deps): Bump sysinfo from 0.29.5 to 0.29.6 (#8311) 
Bumps [sysinfo](https://github.com/GuillaumeGomez/sysinfo) from 0.29.5 to 0.29.6.
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GuillaumeGomez/sysinfo/commits)

---
updated-dependencies:
- dependency-name: sysinfo
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-24 08:40:05 +00:00
dependabot[bot] b2179e2f8b
chore(deps): Bump clap from 4.3.17 to 4.3.19 (#8310) 
Bumps [clap](https://github.com/clap-rs/clap) from 4.3.17 to 4.3.19.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.3.17...v4.3.19)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-24 08:11:17 +00:00
dependabot[bot] 30ac7d86eb
chore(deps): Bump tower-http from 0.4.2 to 0.4.3 (#8298) 
Bumps [tower-http](https://github.com/tower-rs/tower-http) from 0.4.2 to 0.4.3.
- [Release notes](https://github.com/tower-rs/tower-http/releases)
- [Commits](https://github.com/tower-rs/tower-http/compare/tower-http-0.4.2...tower-http-0.4.3)

---
updated-dependencies:
- dependency-name: tower-http
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-21 21:08:33 +00:00
Andrew Lamb 3eb48ef210
chore: Update datafusion again (#8247) 
* chore: Update datafusion to get new grouping

* chore: Update for new API

* chore: update tests

* fix: new API

* fix: state type

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-21 11:20:36 +00:00
dependabot[bot] 0d0f07b34e
chore(deps): Bump tempfile from 3.6.0 to 3.7.0 (#8297) 
* chore(deps): Bump tempfile from 3.6.0 to 3.7.0

Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.6.0 to 3.7.0.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.6.0...v3.7.0)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-21 09:06:34 +00:00
dependabot[bot] 867937157c
chore(deps): Bump num-traits from 0.2.15 to 0.2.16 (#8296) 
Bumps [num-traits](https://github.com/rust-num/num-traits) from 0.2.15 to 0.2.16.
- [Changelog](https://github.com/rust-num/num-traits/blob/master/RELEASES.md)
- [Commits](https://github.com/rust-num/num-traits/compare/num-traits-0.2.15...num-traits-0.2.16)

---
updated-dependencies:
- dependency-name: num-traits
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-21 09:00:43 +00:00
Joe-Blount 1bed99567c
chore: add DF metrics to compaction spans (#8270) 
* chore: add DF metrics to compaction spans

* chore: update string for test verification

* chore: update comment

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-20 15:00:22 +00:00
dependabot[bot] a49a0aaa21
chore(deps): Bump zstd from 0.12.3+zstd.1.5.2 to 0.12.4 (#8280) 
Bumps [zstd](https://github.com/gyscos/zstd-rs) from 0.12.3+zstd.1.5.2 to 0.12.4.
- [Release notes](https://github.com/gyscos/zstd-rs/releases)
- [Commits](https://github.com/gyscos/zstd-rs/commits)

---
updated-dependencies:
- dependency-name: zstd
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-20 14:14:08 +00:00
Marco Neumann 0173c50ba1
fix: use correct error code when querier is shutting down (#8282) 
When a long running query is in process and the querier is shutting
down, it might happen that the executor (= thread pool and tokio
executor responsible for the CPU-bound DataFusion execution) is shut
down while the query is running. From a "systems interaction" PoV I
think this is totally fine and I would like to avoid some weird
ref-counting. Or in other words: if the system is shutting down, shut it
down.

However the error was treated as "internal" which is not useful. The
client should rather be informed that its server was gone and that it is
OK (and desired) to retry. So as per
<https://grpc.github.io/grpc/core/md_doc_statuscodes.html> I think this
should signal "unavailable".

This change wires the error code in such a way that the gRPC service
layer can properly inspect it and then changes the error mapping.

Ref https://github.com/influxdata/idpe/issues/17917 .

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-20 12:08:22 +00:00
dependabot[bot] 9263b42c35
chore(deps): Bump clap from 4.3.16 to 4.3.17 (#8277) 
Bumps [clap](https://github.com/clap-rs/clap) from 4.3.16 to 4.3.17.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.3.16...v4.3.17)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-20 12:03:06 +00:00
dependabot[bot] 0994f54af4
chore(deps): Bump sqlparser from 0.35.0 to 0.36.0 (#8279) 
* chore(deps): Bump sqlparser from 0.35.0 to 0.36.0

Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs) from 0.35.0 to 0.36.0.
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.35.0...v0.36.0)

---
updated-dependencies:
- dependency-name: sqlparser
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-20 09:30:14 +00:00
dependabot[bot] 0f2cbcaef5
chore(deps): Bump zstd-safe from 6.0.5+zstd.1.5.4 to 6.0.6 (#8275) 
Bumps [zstd-safe](https://github.com/gyscos/zstd-rs) from 6.0.5+zstd.1.5.4 to 6.0.6.
- [Release notes](https://github.com/gyscos/zstd-rs/releases)
- [Commits](https://github.com/gyscos/zstd-rs/commits)

---
updated-dependencies:
- dependency-name: zstd-safe
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-20 09:05:54 +00:00
dependabot[bot] 4eff0fb359
chore(deps): Bump tower-http from 0.4.1 to 0.4.2 (#8278) 
Bumps [tower-http](https://github.com/tower-rs/tower-http) from 0.4.1 to 0.4.2.
- [Release notes](https://github.com/tower-rs/tower-http/releases)
- [Commits](https://github.com/tower-rs/tower-http/compare/tower-http-0.4.1...tower-http-0.4.2)

---
updated-dependencies:
- dependency-name: tower-http
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-20 08:36:49 +00:00
dependabot[bot] 59c532ee56
chore(deps): Bump serde from 1.0.171 to 1.0.173 (#8276) 
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.171 to 1.0.173.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.171...v1.0.173)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-20 08:05:38 +00:00
Marco Neumann 004b401a05
chore: upgrade to sqlx 0.7.1 (#8266) 
There are a bunch of dependencies in `Cargo.lock` that are related to
mysql. These are NOT compiled at all, and are also not part of `cargo
tree`. The reason for the inclusion is a bug in cargo:

https://github.com/rust-lang/cargo/issues/10801

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-19 12:18:57 +00:00
dependabot[bot] d3b3805e5f
chore(deps): Bump clap from 4.3.15 to 4.3.16 (#8263) 
Bumps [clap](https://github.com/clap-rs/clap) from 4.3.15 to 4.3.16.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.3.15...v4.3.16)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-19 09:26:04 +00:00
dependabot[bot] 67d1d41b11
chore(deps): Bump scopeguard from 1.1.0 to 1.2.0 (#8255) 
Bumps [scopeguard](https://github.com/bluss/scopeguard) from 1.1.0 to 1.2.0.
- [Commits](https://github.com/bluss/scopeguard/compare/v1.1.0...v1.2.0)

---
updated-dependencies:
- dependency-name: scopeguard
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-18 12:49:34 +00:00
dependabot[bot] 4c4c9f731c
chore(deps): Bump uuid from 1.4.0 to 1.4.1 (#8256) 
Bumps [uuid](https://github.com/uuid-rs/uuid) from 1.4.0 to 1.4.1.
- [Release notes](https://github.com/uuid-rs/uuid/releases)
- [Commits](https://github.com/uuid-rs/uuid/compare/1.4.0...1.4.1)

---
updated-dependencies:
- dependency-name: uuid
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-18 12:30:42 +00:00
dependabot[bot] 14f99f219b
chore(deps): Bump clap from 4.3.12 to 4.3.15 (#8254) 
Bumps [clap](https://github.com/clap-rs/clap) from 4.3.12 to 4.3.15.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.3.12...v4.3.15)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-18 09:18:58 +00:00
dependabot[bot] e33a078128
chore(deps): Bump paste from 1.0.13 to 1.0.14 (#8244) 
Bumps [paste](https://github.com/dtolnay/paste) from 1.0.13 to 1.0.14.
- [Release notes](https://github.com/dtolnay/paste/releases)
- [Commits](https://github.com/dtolnay/paste/compare/1.0.13...1.0.14)

---
updated-dependencies:
- dependency-name: paste
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-17 16:10:02 +00:00
Andrew Lamb 2b164c0037
chore: Update DataFusion pin (#8246) 2023-07-17 15:53:54 +00:00
Joe-Blount 85a9e13262
Merge branch 'main' into jrb_63_compactor_spans 2023-07-17 09:52:27 -05:00
dependabot[bot] 70d104b12d
chore(deps): Bump sysinfo from 0.29.4 to 0.29.5 (#8243) 
Bumps [sysinfo](https://github.com/GuillaumeGomez/sysinfo) from 0.29.4 to 0.29.5.
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GuillaumeGomez/sysinfo/commits)

---
updated-dependencies:
- dependency-name: sysinfo
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-17 14:08:55 +00:00
dependabot[bot] 4c0e5db3a5
chore(deps): Bump insta from 1.30.0 to 1.31.0 (#8242) 
Bumps [insta](https://github.com/mitsuhiko/insta) from 1.30.0 to 1.31.0.
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md)
- [Commits](https://github.com/mitsuhiko/insta/compare/1.30.0...1.31.0)

---
updated-dependencies:
- dependency-name: insta
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-17 14:01:21 +00:00
dependabot[bot] 49d8f829a8
chore(deps): Bump serde from 1.0.168 to 1.0.171 (#8241) 
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.168 to 1.0.171.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.168...v1.0.171)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-17 12:05:54 +00:00
dependabot[bot] b840ca3f1b
chore(deps): Bump serde_json from 1.0.102 to 1.0.103 (#8240) 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.102 to 1.0.103.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.102...v1.0.103)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-17 11:36:47 +00:00
dependabot[bot] 2fdf4b3ee9
chore(deps): Bump assert_cmd from 2.0.11 to 2.0.12 (#8239) 
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd) from 2.0.11 to 2.0.12.
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.11...v2.0.12)

---
updated-dependencies:
- dependency-name: assert_cmd
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-17 10:57:18 +00:00
Fraser Savage a2ca5ca17c
Merge branch 'main' into savage/hook-up-wal-reference-counter-actor 2023-07-17 10:49:45 +01:00
dependabot[bot] 283d77ca58
chore(deps): Bump clap from 4.3.11 to 4.3.12 (#8238) 
Bumps [clap](https://github.com/clap-rs/clap) from 4.3.11 to 4.3.12.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.3.11...v4.3.12)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-17 09:03:39 +00:00
kodiakhq[bot] 699fb70616
Merge branch 'main' into savage/propagate-tracing-spans-from-router-to-ingester 2023-07-14 12:28:56 +00:00
Joe-Blount 823a878675 Merge remote-tracking branch 'origin/main' into jrb_63_compactor_spans 2023-07-13 10:37:21 -05:00
dependabot[bot] 885767aa0a
chore(deps): Bump regex-automata from 0.3.2 to 0.3.3 (#8225) 
Bumps [regex-automata](https://github.com/rust-lang/regex) from 0.3.2 to 0.3.3.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/regex-automata-0.3.2...regex-automata-0.3.3)

---
updated-dependencies:
- dependency-name: regex-automata
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-13 14:50:35 +00:00
Joe-Blount 803122e3b4 Merge remote-tracking branch 'origin/main' into jrb_63_compactor_spans 
# Conflicts:
#	compactor/src/driver.rs
 2023-07-13 08:54:22 -05:00
Fraser Savage 4572057a11
Merge branch 'main' into savage/hook-up-wal-reference-counter-actor 2023-07-13 14:37:42 +01:00
wiedld a9e40c1a90
fix(8191): directory path resolve (#8205) 2023-07-12 13:50:03 -07:00
Andrew Lamb c3788ee2b8
chore: Update datafusion pin (#8223) 2023-07-12 18:18:41 +00:00
Fraser Savage 729851be58
test(ingester): Integration test for RPC write trace context inheritrance 2023-07-12 15:48:41 +01:00
Fraser Savage 5a37c92c2c
feat(router): Send tracing SpanContext header to ingester during RPC write 2023-07-12 11:30:50 +01:00
Dom ceecd11064
Merge branch 'main' into dom/gossip-basic 2023-07-12 10:50:57 +01:00
dependabot[bot] b8f1c8f68a
chore(deps): Bump rustix from 0.38.3 to 0.38.4 (#8217) 
Bumps [rustix](https://github.com/bytecodealliance/rustix) from 0.38.3 to 0.38.4.
- [Release notes](https://github.com/bytecodealliance/rustix/releases)
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.38.3...v0.38.4)

---
updated-dependencies:
- dependency-name: rustix
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-12 09:28:30 +00:00
dependabot[bot] d74f047b06
chore(deps): Bump serde_json from 1.0.100 to 1.0.102 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.100 to 1.0.102.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.100...v1.0.102)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-07-12 02:00:42 +00:00
wiedld d43300635e
Revert "feat(idpe-17789): scheduler job_status() (#8202)" (#8213) 
This reverts commit 3dabccd84b.
 2023-07-11 10:33:56 -07:00
wiedld 3dabccd84b
feat(idpe-17789): scheduler job_status() (#8202) 
* feat(idpe-17789): scheduler job_status() (#8121)

This block of work moves into the scheduler some of the specific downstream actions affiliated with compaction outcomes. Which responsibilities stay in the compactor, versus moved to the scheduler, roughly followed the heuristic of whether the action (a) had an impact on global catalog state (a.k.a. commits and partition skipping), (b) whether it's logging affiliated with compactor health (e.g. ParitionDoneSink logging outcomes) versus system health (e.g. logging commits), and (c) reporting to the scheduler on any errors encountered during compaction. This boundary is subject to change as we move forward.

Also, a noted caveat (TODO) on this commit. We have a CompactionJob which is used to track work handed off to each compactor. Currently it still uses the partition_id for tracking, but the followup PR will start moving the compactor to have more CompactionJob uuid awareness.
 2023-07-11 08:41:12 -07:00
Andrew Lamb b24f9c81ba
chore: Update DataFusion pin, updates for API changed (#8199) 2023-07-11 13:36:38 +00:00
dependabot[bot] 2d5decf108
chore(deps): Bump regex-syntax from 0.7.3 to 0.7.4 (#8206) 
Bumps [regex-syntax](https://github.com/rust-lang/regex) from 0.7.3 to 0.7.4.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/regex-syntax-0.7.3...regex-syntax-0.7.4)

---
updated-dependencies:
- dependency-name: regex-syntax
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-11 09:27:00 +00:00
dependabot[bot] f3b6c8bf15
chore(deps): Bump rustls from 0.21.3 to 0.21.5 (#8207) 
Bumps [rustls](https://github.com/rustls/rustls) from 0.21.3 to 0.21.5.
- [Release notes](https://github.com/rustls/rustls/releases)
- [Commits](https://github.com/rustls/rustls/compare/v/0.21.3...v/0.21.5)

---
updated-dependencies:
- dependency-name: rustls
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
 2023-07-11 09:21:38 +00:00
Joe-Blount 16939c849d chore: add tracing to compactor 2023-07-10 16:36:24 -05:00
Dom Dwyer 58c4874880
chore: workspace_hack support 
Add workspace_hack and whitelist the import.
 2023-07-10 12:11:15 +02:00
Dom Dwyer 48466bfa89
feat(metrics): bytes/frames sent/received & peers 
Emit metrics tracking the number of bytes sent / received, and number of
frames sent / received by the local node.

Track the number of discovered peers to record peer discovery rate and
current number of known peers per node.
 2023-07-10 12:11:15 +02:00
Dom Dwyer 93789d7abb
feat: refuse oversized gossip payloads 
Calculate the available byte size for a user payload sent via gossip,
and pro-actively check this limit earlier, when the caller is attempting
to send the frame, rather than later in the reactor where there's no
feedback to the caller.

DRY frame serialisation to simplify enforcement, and validate/refuse
oversized frames in the reactor so that frames are unlikely to be
truncated by receivers.
 2023-07-10 12:11:14 +02:00
Dom Dwyer bc9ebc9c66
feat: gossip primitive 
Adds a simple "gossip" implementation (more accurately described as a
pub/sub primitive currently) that supports broadcasting
application-level messages to the set of active peers.

This implementation uses UDP as a transport for best-effort delivery,
and enables zero-copy use of the payload using the Bytes crate.

Only peers explicitly provided as "seeds" when initialising will be
known to a gossip node - there's currently no peer exchange mechanism.
This implementation tolerates seeds changing their DNS entries when
restarting to point at new socket addresses (such as within Kubernetes
when pods move around).
 2023-07-10 12:11:14 +02:00
Dom Dwyer 48aa4a5e33
feat(gossip): frame proto definitions 
Adds a proto definition and configures prost to build the rust types
from it.

The gossip framing is intended to be flexible and decoupled - the gossip
library will batch together one or more opaque application messages
and/or control frames, and uniquely identify each peer with a
per-instance UUID to detect crashes/restarts and track peers.
 2023-07-10 12:11:13 +02:00
Dom Dwyer 69ab70ce99
feat: init gossip package 
Adds a new empty "gossip" package to the workspace.
 2023-07-10 12:11:12 +02:00
Andrew Lamb 3ce11d8d66
chore: Update DataFusion (#8190) 
* chore: Update DataFusion

* chore: Run cargo hakari tasks

* fix: Update for API changes

* fix: use display format

* chore: Update explain plan output

* fix: update plans

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-10 09:54:50 +00:00
dependabot[bot] 1f82f6b059
chore(deps): Bump snafu from 0.7.4 to 0.7.5 (#8193) 
Bumps [snafu](https://github.com/shepmaster/snafu) from 0.7.4 to 0.7.5.
- [Changelog](https://github.com/shepmaster/snafu/blob/main/CHANGELOG.md)
- [Commits](https://github.com/shepmaster/snafu/compare/0.7.4...0.7.5)

---
updated-dependencies:
- dependency-name: snafu
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-10 09:35:10 +00:00
dependabot[bot] f0789b74ce
chore(deps): Bump regex from 1.9.0 to 1.9.1 (#8195) 
Bumps [regex](https://github.com/rust-lang/regex) from 1.9.0 to 1.9.1.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.9.0...1.9.1)

---
updated-dependencies:
- dependency-name: regex
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-10 09:29:12 +00:00
dependabot[bot] ab16180f15
chore(deps): Bump serde from 1.0.167 to 1.0.168 (#8194) 
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.167 to 1.0.168.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.167...v1.0.168)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-10 09:24:01 +00:00
dependabot[bot] 12317fee23
chore(deps): Bump async-channel from 1.8.0 to 1.9.0 
Bumps [async-channel](https://github.com/smol-rs/async-channel) from 1.8.0 to 1.9.0.
- [Release notes](https://github.com/smol-rs/async-channel/releases)
- [Changelog](https://github.com/smol-rs/async-channel/blob/master/CHANGELOG.md)
- [Commits](https://github.com/smol-rs/async-channel/compare/v1.8.0...v1.9.0)

---
updated-dependencies:
- dependency-name: async-channel
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-07-10 01:42:26 +00:00
Marko Mikulicic b5faa37152
fix: Plumb tracing header name env/flag to client (#8189) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-07 21:07:29 +00:00
Andrew Lamb 048fc32bd5
feat: add `influxdb_iox debug build-catalog` command (#8067) 
* feat: add `influxdb_iox debug build-catalog` command

* fix: tests

* fix: Use info! logs instead of println for status

* fix: Set partition_hash_id as well

* fix: remove leftover code

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-07 18:32:27 +00:00
dependabot[bot] 18d5b51a78
chore(deps): Bump sysinfo from 0.29.2 to 0.29.4 (#8183) 
Bumps [sysinfo](https://github.com/GuillaumeGomez/sysinfo) from 0.29.2 to 0.29.4.
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GuillaumeGomez/sysinfo/commits)

---
updated-dependencies:
- dependency-name: sysinfo
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-07 09:43:37 +00:00
dependabot[bot] 8b000862e1
chore(deps): Bump pretty_assertions from 1.3.0 to 1.4.0 (#8182) 
Bumps [pretty_assertions](https://github.com/rust-pretty-assertions/rust-pretty-assertions) from 1.3.0 to 1.4.0.
- [Release notes](https://github.com/rust-pretty-assertions/rust-pretty-assertions/releases)
- [Changelog](https://github.com/rust-pretty-assertions/rust-pretty-assertions/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rust-pretty-assertions/rust-pretty-assertions/compare/v1.3.0...v1.4.0)

---
updated-dependencies:
- dependency-name: pretty_assertions
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-07 09:35:18 +00:00
dependabot[bot] 057ee40cb9
chore(deps): Bump thiserror from 1.0.41 to 1.0.43 (#8181) 
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.41 to 1.0.43.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.41...1.0.43)

---
updated-dependencies:
- dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-07 09:25:12 +00:00
dependabot[bot] 9df1186472
chore(deps): Bump is-terminal from 0.4.8 to 0.4.9 (#8180) 
Bumps [is-terminal](https://github.com/sunfishcode/is-terminal) from 0.4.8 to 0.4.9.
- [Commits](https://github.com/sunfishcode/is-terminal/compare/v0.4.8...v0.4.9)

---
updated-dependencies:
- dependency-name: is-terminal
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-07 09:16:43 +00:00
dependabot[bot] 778299112b
chore(deps): Bump serde from 1.0.166 to 1.0.167 (#8179) 
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.166 to 1.0.167.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.166...v1.0.167)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-07 09:05:08 +00:00
Joe-Blount 9f522bfd30
Revert "feat(idpe-17789): scheduler job_status() (#8121)" (#8175) 
This reverts commit 5d19fa3635.
 2023-07-06 18:52:25 +00:00
wiedld 5d19fa3635
feat(idpe-17789): scheduler job_status() (#8121) 
This block of work moves into the scheduler some of the specific downstream actions affiliated with compaction outcomes. Which responsibilities stay in the compactor, versus moved to the scheduler, roughly followed the heuristic of whether the action (a) had an impact on global catalog state (a.k.a. commits and partition skipping), (b) whether it's logging affiliated with compactor health (e.g. ParitionDoneSink logging outcomes) versus system health (e.g. logging commits), and (c) reporting to the scheduler on any errors encountered during compaction. This boundary is subject to change as we move forward.

Also, a noted caveat (TODO) on this commit. We have a CompactionJob which is used to track work handed off to each compactor. Currently it still uses the partition_id for tracking, but the followup PR will start moving the compactor to have more CompactionJob uuid awareness.
 2023-07-06 09:15:59 -07:00
Dom a005f344d8
Merge branch 'main' into 7899/wal-disk-metrics 2023-07-06 14:44:11 +01:00
dependabot[bot] b2c1532d3a
chore(deps): Bump regex from 1.8.4 to 1.9.0 (#8160) 
* chore(deps): Bump regex from 1.8.4 to 1.9.0

Bumps [regex](https://github.com/rust-lang/regex) from 1.8.4 to 1.9.0.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.8.4...1.9.0)

---
updated-dependencies:
- dependency-name: regex
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
 2023-07-06 12:48:27 +00:00
dependabot[bot] bbcf3f0556
chore(deps): Bump toml from 0.7.5 to 0.7.6 (#8161) 
Bumps [toml](https://github.com/toml-rs/toml) from 0.7.5 to 0.7.6.
- [Commits](https://github.com/toml-rs/toml/compare/toml-v0.7.5...toml-v0.7.6)

---
updated-dependencies:
- dependency-name: toml
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
 2023-07-06 12:35:47 +00:00
Fraser Savage bea9dbf7ab
test(ingester): Integration test dropping of persisted WAL segments 
This adds an integration test that writes some data to the ingester,
waits for the WAL to be rotated and then ensures that the segment file
has been dropped.
 2023-07-06 12:12:58 +01:00
dependabot[bot] a6df3a4b36
chore(deps): Bump clap from 4.3.10 to 4.3.11 (#8162) 
Bumps [clap](https://github.com/clap-rs/clap) from 4.3.10 to 4.3.11.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.3.10...v4.3.11)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-06 10:15:25 +00:00
dependabot[bot] 26a6113a37
chore(deps): Bump async-trait from 0.1.70 to 0.1.71 (#8163) 
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.70 to 0.1.71.
- [Release notes](https://github.com/dtolnay/async-trait/releases)
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.70...0.1.71)

---
updated-dependencies:
- dependency-name: async-trait
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-06 09:58:51 +00:00
Marco Neumann 1670966119
fix: send flight keep-alives (#8144) 
* fix: send flight keep-alives

This was tested using a local nginx proxy and a small modication to the
IOx source code: insert `make_stream_slow` into `GetStream::new` just
between `let query_results = ...` and
`let innr = FlightDataEncoderBuilder::new()...`.

Fixes https://github.com/influxdata/idpe/issues/17824 .

* fix: typo

* refactor: reduce keep-alive interval to 5s
 2023-07-06 09:53:31 +00:00
dependabot[bot] bc6bf2d8e5
chore(deps): Bump smallvec from 1.10.0 to 1.11.0 (#8164) 
Bumps [smallvec](https://github.com/servo/rust-smallvec) from 1.10.0 to 1.11.0.
- [Release notes](https://github.com/servo/rust-smallvec/releases)
- [Commits](https://github.com/servo/rust-smallvec/compare/v1.10.0...v1.11.0)

---
updated-dependencies:
- dependency-name: smallvec
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-06 09:43:27 +00:00
dependabot[bot] c7dbb0ceb4
chore(deps): Bump regex-syntax from 0.7.2 to 0.7.3 (#8165) 
Bumps [regex-syntax](https://github.com/rust-lang/regex) from 0.7.2 to 0.7.3.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/regex-syntax-0.7.2...regex-syntax-0.7.3)

---
updated-dependencies:
- dependency-name: regex-syntax
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-06 09:28:25 +00:00
dependabot[bot] adddaf5d82
chore(deps): Bump rustix from 0.38.2 to 0.38.3 (#8166) 
Bumps [rustix](https://github.com/bytecodealliance/rustix) from 0.38.2 to 0.38.3.
- [Release notes](https://github.com/bytecodealliance/rustix/releases)
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.38.2...v0.38.3)

---
updated-dependencies:
- dependency-name: rustix
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-06 09:08:38 +00:00
dependabot[bot] a25fd6015d
chore(deps): Bump rustls from 0.21.2 to 0.21.3 (#8167) 
Bumps [rustls](https://github.com/rustls/rustls) from 0.21.2 to 0.21.3.
- [Changelog](https://github.com/rustls/rustls/blob/main/RELEASE_NOTES.md)
- [Commits](https://github.com/rustls/rustls/commits)

---
updated-dependencies:
- dependency-name: rustls
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-06 09:01:12 +00:00
wiedld 36e7f53f9b
Merge branch 'main' into 7899/wal-disk-metrics 2023-07-05 13:52:43 -07:00
dependabot[bot] 3827257f94
chore(deps): Bump thiserror from 1.0.40 to 1.0.41 (#8149) 
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.40 to 1.0.41.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.40...1.0.41)

---
updated-dependencies:
- dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
 2023-07-05 09:25:14 +00:00
dependabot[bot] 7c771ce6f3
chore(deps): Bump serde_json from 1.0.99 to 1.0.100 (#8147) 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.99 to 1.0.100.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.99...v1.0.100)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-05 09:12:11 +00:00
dependabot[bot] b5c9628f0f
chore(deps): Bump async-trait from 0.1.69 to 0.1.70 (#8148) 
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.69 to 0.1.70.
- [Release notes](https://github.com/dtolnay/async-trait/releases)
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.69...0.1.70)

---
updated-dependencies:
- dependency-name: async-trait
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-07-05 09:05:13 +00:00
Marco Neumann 9f6acfa3d9
chore: update `hermit-abi` to 0.3.2 (#8150) 
0.3.1 got yanked.
 2023-07-05 08:59:12 +00:00
Marco Neumann d2658d2e32
chore: update heappy to `1de977a241cdd768acc5b6c82c0728b30c7db7b4` (#8145) 
Removes some redundant deps.
 2023-07-04 14:14:46 +00:00
dependabot[bot] b827b76583
chore(deps): Bump console-subscriber from 0.1.9 to 0.1.10 (#8142) 
Bumps [console-subscriber](https://github.com/tokio-rs/console) from 0.1.9 to 0.1.10.
- [Release notes](https://github.com/tokio-rs/console/releases)
- [Commits](https://github.com/tokio-rs/console/compare/console-subscriber-v0.1.9...console-subscriber-v0.1.10)

---
updated-dependencies:
- dependency-name: console-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-04 12:13:49 +00:00
dependabot[bot] 36c253123a
chore(deps): Bump pprof from 0.11.1 to 0.12.0 (#8141) 
Bumps [pprof](https://github.com/tikv/pprof-rs) from 0.11.1 to 0.12.0.
- [Changelog](https://github.com/tikv/pprof-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tikv/pprof-rs/commits)

---
updated-dependencies:
- dependency-name: pprof
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-04 12:00:59 +00:00
dependabot[bot] 71795f265f
chore(deps): Bump serde from 1.0.164 to 1.0.166 (#8140) 
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.164 to 1.0.166.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.164...v1.0.166)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-04 10:56:56 +00:00
dependabot[bot] 9a03d9c9fe
chore(deps): Bump paste from 1.0.12 to 1.0.13 (#8139) 
Bumps [paste](https://github.com/dtolnay/paste) from 1.0.12 to 1.0.13.
- [Release notes](https://github.com/dtolnay/paste/releases)
- [Commits](https://github.com/dtolnay/paste/compare/1.0.12...1.0.13)

---
updated-dependencies:
- dependency-name: paste
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-04 07:57:41 +00:00
Carol (Nichols || Goulding) 8ebf390d9c
feat: Try to prune ingester partitions by partition key 
This is hacktastic.
 2023-07-03 17:24:58 +02:00
Marco Neumann 7079de9b15
chore: manual `cargo update` (#8134) 
* chore: manual `cargo update`

Sometimes dependabot gets behind or confused with a few depndencies, so
running a manual update from time to time is good:

```console
❯ cargo update
    Updating git repository `https://github.com/alamb/arrow-rs.git`
    Updating crates.io index
    Updating git repository `https://github.com/apache/arrow-datafusion.git`
    Updating git repository `https://github.com/mkmik/heappy`
    Updating async-trait v0.1.68 -> v0.1.69
    Updating bitflags v2.3.2 -> v2.3.3
    Updating cpp_demangle v0.4.1 -> v0.4.2
    Updating fd-lock v3.0.12 -> v3.0.13
    Updating h2 v0.3.19 -> v0.3.20
    Updating half v2.2.1 -> v2.3.1
    Updating ipnet v2.7.2 -> v2.8.0
    Updating is-terminal v0.4.7 -> v0.4.8
    Updating itoa v1.0.6 -> v1.0.7
      Adding linux-raw-sys v0.4.3
    Updating phf v0.11.1 -> v0.11.2
    Updating phf_codegen v0.11.1 -> v0.11.2
    Updating phf_generator v0.11.1 -> v0.11.2
    Updating pin-project-lite v0.2.9 -> v0.2.10
    Updating proc-macro2 v1.0.60 -> v1.0.63
    Updating quote v1.0.28 -> v1.0.29
      Adding rustix v0.38.2
    Updating rustls-pemfile v1.0.2 -> v1.0.3
    Updating syn v2.0.18 -> v2.0.23
    Updating whoami v1.4.0 -> v1.4.1
    Updating windows-targets v0.48.0 -> v0.48.1
```

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2023-07-03 10:50:31 +00:00
dependabot[bot] 5223b1fbaa
chore(deps): Bump clap from 4.3.9 to 4.3.10 
Bumps [clap](https://github.com/clap-rs/clap) from 4.3.9 to 4.3.10.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.3.9...v4.3.10)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-07-03 09:19:21 +00:00
dependabot[bot] ca042e54c7
chore(deps): Bump integer-encoding from 3.0.4 to 4.0.0 (#8130) 
Bumps [integer-encoding](https://github.com/dermesser/integer-encoding-rs) from 3.0.4 to 4.0.0.
- [Commits](https://github.com/dermesser/integer-encoding-rs/compare/v3.0.4...v4.0.0)

---
updated-dependencies:
- dependency-name: integer-encoding
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-03 09:17:34 +00:00
dependabot[bot] 9f00c9c4ef
chore(deps): Bump pin-project from 1.1.1 to 1.1.2 (#8129) 
Bumps [pin-project](https://github.com/taiki-e/pin-project) from 1.1.1 to 1.1.2.
- [Release notes](https://github.com/taiki-e/pin-project/releases)
- [Changelog](https://github.com/taiki-e/pin-project/blob/main/CHANGELOG.md)
- [Commits](https://github.com/taiki-e/pin-project/compare/v1.1.1...v1.1.2)

---
updated-dependencies:
- dependency-name: pin-project
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-03 09:10:28 +00:00
Marco Neumann ce6a2fb613
refactor: remove `QueryChunk::column_values` (#8111) 
Similar to #8109.

This was once implemented by the RUB but as it stands right now, no
chunk implements this anymore.

If we ever want to bring this back, we should use the output of
`QueryChunk::data` instead (i.e. use a data-based implementation instead
of a per-chunk one).

Closes #8096.
 2023-07-03 09:03:21 +00:00
dependabot[bot] 7ca456c668
chore(deps): Bump rustix from 0.37.21 to 0.37.22 (#8132) 
Bumps [rustix](https://github.com/bytecodealliance/rustix) from 0.37.21 to 0.37.22.
- [Release notes](https://github.com/bytecodealliance/rustix/releases)
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.37.21...v0.37.22)

---
updated-dependencies:
- dependency-name: rustix
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-03 08:56:38 +00:00
wiedld d64a908823
Merge branch 'main' into 7899/wal-disk-metrics 2023-06-30 18:59:49 -07:00
Andrew Lamb 4a1f8db254
chore: Update datafusion + arrow/arrow-flight/parquet to patched version `42.0.0` (#8113) 
* Revert "Revert "chore: Update datafusion + arrow/arrow-flight/parquet to version `42.0.0` (#8036)" (#8049)"

This reverts commit fb0674fc01.

* chore: Update Cargo and hakari

* chore: Update to patched version

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-30 12:59:31 +00:00
dependabot[bot] b05848c998
chore(deps): Bump rustix from 0.37.20 to 0.37.21 (#8120) 
Bumps [rustix](https://github.com/bytecodealliance/rustix) from 0.37.20 to 0.37.21.
- [Release notes](https://github.com/bytecodealliance/rustix/releases)
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.37.20...v0.37.21)

---
updated-dependencies:
- dependency-name: rustix
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-30 10:00:52 +00:00
dependabot[bot] a30ba2ae28
chore(deps): Bump backtrace from 0.3.67 to 0.3.68 (#8119) 
Bumps [backtrace](https://github.com/rust-lang/backtrace-rs) from 0.3.67 to 0.3.68.
- [Release notes](https://github.com/rust-lang/backtrace-rs/releases)
- [Commits](https://github.com/rust-lang/backtrace-rs/compare/0.3.67...0.3.68)

---
updated-dependencies:
- dependency-name: backtrace
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-30 09:43:25 +00:00
dependabot[bot] 9381c348ae
chore(deps): Bump tokio from 1.29.0 to 1.29.1 (#8117) 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.29.0 to 1.29.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.29.0...tokio-1.29.1)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-30 09:25:57 +00:00
dependabot[bot] ede8e32804
chore(deps): Bump pin-project from 1.1.0 to 1.1.1 (#8118) 
Bumps [pin-project](https://github.com/taiki-e/pin-project) from 1.1.0 to 1.1.1.
- [Release notes](https://github.com/taiki-e/pin-project/releases)
- [Changelog](https://github.com/taiki-e/pin-project/blob/main/CHANGELOG.md)
- [Commits](https://github.com/taiki-e/pin-project/compare/v1.1.0...v1.1.1)

---
updated-dependencies:
- dependency-name: pin-project
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-30 08:51:56 +00:00
dependabot[bot] 0cf7d06b9d
chore(deps): Bump num_cpus from 1.15.0 to 1.16.0 (#8115) 
Bumps [num_cpus](https://github.com/seanmonstar/num_cpus) from 1.15.0 to 1.16.0.
- [Release notes](https://github.com/seanmonstar/num_cpus/releases)
- [Changelog](https://github.com/seanmonstar/num_cpus/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/num_cpus/compare/v1.15.0...v1.16.0)

---
updated-dependencies:
- dependency-name: num_cpus
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-30 08:32:43 +00:00
Marco Neumann 1b8b3ae4c3
refactor: bundle projection schema calculation (#8108) 
* refactor: convert projection mask earlier

* refactor: bundle projection schema calculation

Same as #8102 but for the projected schema. This now has a nice side
effect:

1. there is no longer a per chunk cache lookup
2. there is no longer ANY per chunk async computation
3. we no longer need an early pruning stage for the chunks (we've used
   to do that so we can throw away chunks before doing the more
   expensive part of the chunk creation)

This nicely streamlines and simplifies the code.

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-30 08:27:30 +00:00
Marco Neumann ca31c1eade
feat: hook up tokio metrics (#8050) 
* feat: metrics for main tokio runtime

* feat: instrument executor tokio runtime

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-29 11:11:44 +00:00
Marco Neumann 9874b283b7
feat: tokio metrics bridge (#8091) 2023-06-29 08:43:57 +00:00
dependabot[bot] a91ea08f4c
chore(deps): Bump clap from 4.3.8 to 4.3.9 (#8105) 
Bumps [clap](https://github.com/clap-rs/clap) from 4.3.8 to 4.3.9.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.3.8...v4.3.9)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-29 08:19:29 +00:00
dependabot[bot] 89f460c5f8
chore(deps): Bump chrono-tz from 0.8.2 to 0.8.3 (#8104) 
Bumps [chrono-tz](https://github.com/chronotope/chrono-tz) from 0.8.2 to 0.8.3.
- [Release notes](https://github.com/chronotope/chrono-tz/releases)
- [Changelog](https://github.com/chronotope/chrono-tz/blob/main/CHANGELOG.md)
- [Commits](https://github.com/chronotope/chrono-tz/commits)

---
updated-dependencies:
- dependency-name: chrono-tz
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-29 07:58:18 +00:00
wiedld 3a8a8a153e
feat(idpe 17789): provide scheduler interface (#8057) 
* feat: provide convenience methods to create Scheduler, and keep the scheduler implementations crate private. External crates can only create a Scheduler based upon configs.

* feat: provide Scheduler as a component to compactor. Specifically, the scheduler configs are present within the compactor run config, and the scheduler in created within the compactor hardcoded components.

* feat: within the compactor ScheduledPartitionsSource, utilize the dyn Scheduler and Scheduler.get_jobs()

* feat: CompactionJob should be per partition, and have a uniqueness characteristic independent of the partition

* feat: keep compactor_scheduler separate from clap_blocks. Only interface is within ioxd_compactor where the CLI configs are transformed into ShardConfig and PartitionsSourceConfig.

* chore: make IdOnlyPartitionFilter into only pub(crate)

* chore: update scheduler display to include any report information (a.k.a. shard_config, if present)
 2023-06-28 15:04:00 -07:00
dependabot[bot] b15c6062a9
chore(deps): Bump tokio from 1.28.2 to 1.29.0 (#8100) 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.28.2 to 1.29.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.28.2...tokio-1.29.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 13:18:08 +00:00
dependabot[bot] bb6f481de8
chore(deps): Bump uuid from 1.3.4 to 1.4.0 (#8099) 
Bumps [uuid](https://github.com/uuid-rs/uuid) from 1.3.4 to 1.4.0.
- [Release notes](https://github.com/uuid-rs/uuid/releases)
- [Commits](https://github.com/uuid-rs/uuid/compare/1.3.4...1.4.0)

---
updated-dependencies:
- dependency-name: uuid
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 10:26:53 +00:00
dependabot[bot] 647541fc12
chore(deps): Bump croaring from 0.8.1 to 0.9.0 (#8088) 
Bumps [croaring](https://github.com/saulius/croaring-rs) from 0.8.1 to 0.9.0.
- [Release notes](https://github.com/saulius/croaring-rs/releases)
- [Commits](https://github.com/saulius/croaring-rs/commits)

---
updated-dependencies:
- dependency-name: croaring
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-27 08:10:39 +00:00
dependabot[bot] db1d977380
chore(deps): Bump hyper from 0.14.26 to 0.14.27 (#8087) 
Bumps [hyper](https://github.com/hyperium/hyper) from 0.14.26 to 0.14.27.
- [Release notes](https://github.com/hyperium/hyper/releases)
- [Changelog](https://github.com/hyperium/hyper/blob/v0.14.27/CHANGELOG.md)
- [Commits](https://github.com/hyperium/hyper/compare/v0.14.26...v0.14.27)

---
updated-dependencies:
- dependency-name: hyper
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-27 07:44:15 +00:00
dependabot[bot] 7e30c91ceb
chore(deps): Bump libc from 0.2.146 to 0.2.147 (#8079) 
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.146 to 0.2.147.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.146...0.2.147)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-26 10:18:55 +00:00
dependabot[bot] 54aaafd496
chore(deps): Bump rustyline from 11.0.0 to 12.0.0 (#8078) 
Bumps [rustyline](https://github.com/kkawakam/rustyline) from 11.0.0 to 12.0.0.
- [Release notes](https://github.com/kkawakam/rustyline/releases)
- [Changelog](https://github.com/kkawakam/rustyline/blob/master/History.md)
- [Commits](https://github.com/kkawakam/rustyline/compare/v11.0.0...v12.0.0)

---
updated-dependencies:
- dependency-name: rustyline
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-26 10:05:29 +00:00
dependabot[bot] a0f382302a
chore(deps): Bump clap from 4.3.5 to 4.3.8 (#8077) 
Bumps [clap](https://github.com/clap-rs/clap) from 4.3.5 to 4.3.8.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.3.5...v4.3.8)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-26 09:37:26 +00:00
dependabot[bot] 24ca0fef30
chore(deps): Bump serde_json from 1.0.97 to 1.0.99 (#8076) 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.97 to 1.0.99.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.97...v1.0.99)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-26 09:25:38 +00:00
dependabot[bot] 5c7dd8bd88
chore(deps): Bump toml from 0.7.4 to 0.7.5 (#8075) 
Bumps [toml](https://github.com/toml-rs/toml) from 0.7.4 to 0.7.5.
- [Commits](https://github.com/toml-rs/toml/compare/toml-v0.7.4...toml-v0.7.5)

---
updated-dependencies:
- dependency-name: toml
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-26 09:17:16 +00:00
dependabot[bot] 37be773890
chore(deps): Bump phf_shared from 0.11.1 to 0.11.2 (#8074) 
Bumps [phf_shared](https://github.com/rust-phf/rust-phf) from 0.11.1 to 0.11.2.
- [Release notes](https://github.com/rust-phf/rust-phf/releases)
- [Changelog](https://github.com/rust-phf/rust-phf/blob/master/RELEASE_PROCESS.md)
- [Commits](https://github.com/rust-phf/rust-phf/compare/phf_shared-v0.11.1...phf_shared-v0.11.2)

---
updated-dependencies:
- dependency-name: phf_shared
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-26 09:04:56 +00:00
dependabot[bot] 990044dcb2
chore(deps): Bump indexmap from 1.9.3 to 2.0.0 (#8073) 
* chore(deps): Bump indexmap from 1.9.3 to 2.0.0

Bumps [indexmap](https://github.com/bluss/indexmap) from 1.9.3 to 2.0.0.
- [Changelog](https://github.com/bluss/indexmap/blob/master/RELEASES.md)
- [Commits](https://github.com/bluss/indexmap/compare/1.9.3...2.0.0)

---
updated-dependencies:
- dependency-name: indexmap
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-26 08:52:51 +00:00
dependabot[bot] cf560d18f1
chore(deps): Bump sqlparser from 0.34.0 to 0.35.0 (#8071) 
* chore(deps): Bump sqlparser from 0.34.0 to 0.35.0

Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs) from 0.34.0 to 0.35.0.
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.34.0...v0.35.0)

---
updated-dependencies:
- dependency-name: sqlparser
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2023-06-26 08:40:48 +00:00
dependabot[bot] 74a48a8f63
chore(deps): Bump itertools from 0.10.5 to 0.11.0 (#8060) 
* chore(deps): Bump itertools from 0.10.5 to 0.11.0

Bumps [itertools](https://github.com/rust-itertools/itertools) from 0.10.5 to 0.11.0.
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-itertools/itertools/compare/v0.10.5...v0.11.0)

---
updated-dependencies:
- dependency-name: itertools
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-23 08:11:56 +00:00
dependabot[bot] 6e7b838b52
chore(deps): Bump insta from 1.29.0 to 1.30.0 (#8059) 
Bumps [insta](https://github.com/mitsuhiko/insta) from 1.29.0 to 1.30.0.
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md)
- [Commits](https://github.com/mitsuhiko/insta/compare/1.29.0...1.30.0)

---
updated-dependencies:
- dependency-name: insta
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-23 07:45:41 +00:00
wiedld 62251e2323
refactor(idpe-17789): delineate btwn partitions_source within the scheduler versus compactor (#8028) 
This is purely a movement of code, and not any definition of the interface methods yet. At best, it further solidifying the boundary of what partitions_source implementations are within the scheduler -- versus within the compactor.
 2023-06-22 11:48:08 -07:00
Carol (Nichols || Goulding) d991e12fbb
feat: Send PartitionHashId from ingesters to queriers 2023-06-22 09:01:22 -04:00
Carol (Nichols || Goulding) 62ba18171a
feat: Add a new hash column on the partition and parquet file tables 
This will hold the deterministic ID for partitions.

Until all existing partitions have this value, this is optional/nullable.

The row ID still exists and is used as the main foreign key in the
parquet_file and skipped_compaction tables.

The hash_id has a unique index so that we can look up records based on
it (if it's available).

If the parquet file record has a partition_hash_id value, use that to
generate the object storage path instead of the partition_id.
 2023-06-22 09:01:22 -04:00
Andrew Lamb fb0674fc01
Revert "chore: Update datafusion + arrow/arrow-flight/parquet to version `42.0.0` (#8036)" (#8049) 
This reverts commit 70ffedadc7.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-22 11:03:25 +00:00
Fraser Savage 43d6cb6eb1
Merge branch 'main' into savage/replace-dml-operation-for-ingester-rpc-write 2023-06-22 10:14:38 +01:00
Marco Neumann e72566e0e5
refactor: clean up querier server interface (#8045) 
Move all the gRPC assembly into one single place: `ioxd_querier`. This
way `querier` no longer depends on `service_*` (except for
`service_common` which doesn't really implement gRPC but only the
namespace/database entry point).
 2023-06-22 08:57:24 +00:00
Marco Neumann c9349a685f
refactor: remove pointless handler abstraction (#8044) 
If your abstraction has one implementation, it ain't an abstraction.
 2023-06-22 08:30:42 +00:00
wiedld fd881ea82e chore: add disk protection to the wal directory 2023-06-21 22:12:58 -07:00
Stuart Carnie 17bbb47a78
Merge branch 'main' into sgc/issue/7600_moving_average 2023-06-22 14:05:30 +10:00
Stuart Carnie 195d5aa0be
chore: Update pest as 2.6.1 was yanked (#8043) 
See: https://crates.io/crates/pest/2.6.1
 2023-06-22 04:04:48 +00:00
Stuart Carnie 13726c2a76
Merge branch 'main' into sgc/issue/7600_moving_average 2023-06-22 10:10:22 +10:00
wiedld f75736891a
refactor(idpe-17789): move mod id_only_partition_filter (#8027) 
* chore: delineate scheduler logic boundary in code comments
* refactor: move id_only_partition_filter mod into local scheduler
* chore: add docs for each IdOnlyPartitionFilter implementation
 2023-06-21 10:29:38 -07:00
wiedld 09654542ec Revert "chore(7899): add InstrumentedDiskProtection to the WAL" 
This reverts commit f632e4f023.
 2023-06-21 09:51:37 -07:00
Andrew Lamb 70ffedadc7
chore: Update datafusion + arrow/arrow-flight/parquet to version `42.0.0` (#8036) 
* chore: Update datafusion + arrow/arrow-flight/parquet to version `42.0.0`

* chore: Update for new APIs

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-21 16:11:36 +00:00
Fraser Savage 4f489b9d9d
refactor(ingester): Add dml_payload::encode mod for encode to `DatabaseBatch` 
This enables the whole of the RPC write path to use the new
`dml_payload::IngestOp` instead of the `DmlOperation` type. The
implementation of `From<&WriteOperation> for DmlWrite` has been removed
to complete the switch.
 2023-06-21 11:13:44 +01:00
dependabot[bot] 1e22d7a25c
chore(deps): Bump clap from 4.3.4 to 4.3.5 (#8030) 
Bumps [clap](https://github.com/clap-rs/clap) from 4.3.4 to 4.3.5.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.3.4...v4.3.5)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-21 09:09:02 +00:00
dependabot[bot] 27a98d7586
chore(deps): Bump tower-http from 0.4.0 to 0.4.1 (#8029) 
Bumps [tower-http](https://github.com/tower-rs/tower-http) from 0.4.0 to 0.4.1.
- [Release notes](https://github.com/tower-rs/tower-http/releases)
- [Commits](https://github.com/tower-rs/tower-http/compare/tower-http-0.4.0...tower-http-0.4.1)

---
updated-dependencies:
- dependency-name: tower-http
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-21 08:17:40 +00:00
Stuart Carnie edaac28498
Merge branch 'main' into sgc/issue/7600_moving_average 2023-06-21 11:39:06 +10:00
wiedld e29b453e0d
refactor: move PartitionsSourceConfig into local scheduler (#8026) 2023-06-20 16:05:59 -07:00
wiedld 34b5fadde0
refactor: move scheduler related configs to compactor_scheduler (#8013) 2023-06-20 09:55:35 -07:00
Dom cdbde3044e
Merge branch 'main' into dom/perf-sparse-reup 2023-06-20 14:14:47 +01:00
dependabot[bot] 123ba32fb9
chore(deps): Bump serde_json from 1.0.96 to 1.0.97 (#8021) 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.96 to 1.0.97.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.96...v1.0.97)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-20 08:53:15 +00:00
dependabot[bot] 6f3fb2e551
chore(deps): Bump mockito from 1.0.2 to 1.1.0 (#8020) 
Bumps [mockito](https://github.com/lipanski/mockito) from 1.0.2 to 1.1.0.
- [Release notes](https://github.com/lipanski/mockito/releases)
- [Commits](https://github.com/lipanski/mockito/compare/1.0.2...1.1.0)

---
updated-dependencies:
- dependency-name: mockito
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-20 08:44:52 +00:00
dependabot[bot] 7d878b21bd
chore(deps): Bump comfy-table from 7.0.0 to 7.0.1 (#8019) 
Bumps [comfy-table](https://github.com/nukesor/comfy-table) from 7.0.0 to 7.0.1.
- [Release notes](https://github.com/nukesor/comfy-table/releases)
- [Changelog](https://github.com/Nukesor/comfy-table/blob/main/CHANGELOG.md)
- [Commits](https://github.com/nukesor/comfy-table/compare/v7.0.0...v7.0.1)

---
updated-dependencies:
- dependency-name: comfy-table
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-20 08:32:33 +00:00
Stuart Carnie 8670b28445
Merge branch 'main' into sgc/issue/7600_moving_average 2023-06-18 09:41:19 +10:00
wiedld f632e4f023 chore(7899): add InstrumentedDiskProtection to the WAL 
* this requires providing the metrics registry to the wal
 2023-06-16 20:42:24 -07:00
wiedld cd24e49a48 feat(7899): add InstrumentedDiskProtection sealed abstraction for reuse across codebase 2023-06-16 20:42:10 -07:00
wiedld 8b7ef69f6f
refactor: move partitions_source to scheduler (#8010) 
* refactor: make compactor_scheduler crate

* refactor: move PartitionsSource into the compactor_scheduler

    The compactor currently uses PartitionsSource in two ways:
       * for the preparation of PartitionIds prior to the compactor pipeline.
       * for the abstraction which utilize the PartitionIds during the IO pipeline.
    This commit is a refactoring to enable us to delineate between these two utilizations.
    The former (preparation) utilization will now be done in the compactor_scheduler.
    Since the compactor is dependent on the compactor_scheduler, it made sense to move the trait to the scheduler.
 2023-06-16 10:02:13 -07:00
Dom 27977299ad
Merge branch 'main' into dom/perf-sparse-reup 2023-06-16 15:55:21 +01:00
Stuart Carnie e10b8c93c8
chore: Update DataFusion and other dependencies (#8014) 
* chore: Update DataFusion pin

* chore: Update API changes

* chore: Don't use deprecated API

* chore: Run cargo hakari tasks

* chore: Update tests due to changes in logical plan nodes from DF update

* chore: Fix broken links in docs

* chore: Adjust changes to expected output

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
 2023-06-16 10:39:36 +00:00
Andrew Lamb 5889c96501
chore: Update `datafusion` and other dependencies (#7981) 
* chore: Update DatFaFusion pin

* chore: Update other dependencies

* chore: Update hakari

* fix: Update for API changes

* fix: Update explain plan

* fix: Update influxql plans

* fix: rustdoc links

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-16 09:48:55 +00:00
dependabot[bot] e346ec9286
chore(deps): Bump sha2 from 0.10.6 to 0.10.7 (#8015) 
Bumps [sha2](https://github.com/RustCrypto/hashes) from 0.10.6 to 0.10.7.
- [Commits](https://github.com/RustCrypto/hashes/compare/sha2-v0.10.6...sha2-v0.10.7)

---
updated-dependencies:
- dependency-name: sha2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-16 08:40:01 +00:00
Stuart Carnie 22baf5096e
feat: define place-holder UDFs for `moving_average` and `difference` 2023-06-16 13:10:47 +10:00
Dom Dwyer 0a2a315b91
chore: limit chrono features 
See https://rustsec.org/advisories/RUSTSEC-2020-0071
 2023-06-15 16:41:20 +02:00
Dom Dwyer 5388e49734
test: router partition handler 
Asserts the partitioning code within the router (that drives the
low-level partitioning logic) generates partitions with rows with
timestamps that belong in those partitions.
 2023-06-15 14:54:46 +02:00
dependabot[bot] 88c50bb486
chore(deps): Bump rustls from 0.21.1 to 0.21.2 (#7999) 
Bumps [rustls](https://github.com/rustls/rustls) from 0.21.1 to 0.21.2.
- [Changelog](https://github.com/rustls/rustls/blob/main/RELEASE_NOTES.md)
- [Commits](https://github.com/rustls/rustls/compare/v/0.21.1...v/0.21.2)

---
updated-dependencies:
- dependency-name: rustls
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-15 12:20:02 +00:00
Marco Neumann 64f573c13f
feat: cache partition template in querier (#7987) 
* feat: impl `Eq` for `TablePartitionTemplateOverride`

* feat: `TablePartitionTemplateOverride::size`

* feat: cache partition template in querier

Required for #7974.
 2023-06-15 10:30:56 +00:00
dependabot[bot] 38d7e0e0de
chore(deps): Bump clap from 4.3.3 to 4.3.4 (#7998) 
Bumps [clap](https://github.com/clap-rs/clap) from 4.3.3 to 4.3.4.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.3.3...v4.3.4)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-15 08:07:39 +00:00
Phil Bracikowski e34ec77e8d
feat(garbage-collector): batch parquet existence checks to catalog (#7964) 
* feat(garbage-collector): batch parquet existence checks to catalog

The core feature of this PR is batching the existence checks of parquet
files in object store against the catalog. Before, there was 1 catalog
query per each parquet file in object store. This can be a lot of
requests.

This PR can perform one query of at most 100 parquet file uuids against
the catalog in one query. A hundred seems like a decent starting place.

The batch may not reach 100 because there is also a timeout on receiving
object store meta objects from the object store lister thread. That
timeout is set to 100 milliseconds. If more than 100 are received, they
are batched into 100 for the catalog.

Additionally, this PR includes surrounding code changes to make it more
idiomatic (but not perfect). It follows up some suggested work from
 #7652 for watching for shutdown on the threads.

* fixes #7784

* use hashset instead of vec to test for contains
* chore: add test for db failure path
* remove ParquetFileExistsByOSID and other single field structs that are
  just for sql deserialization; map to uuid explicitly
* fix the sqlite query by using a blob literal X'<hex>' for uuids
* comment clarifications
* adjust loggings to warn from debug for expected rare events

Many thanks to Carol for help implementing this!
 2023-06-14 07:59:00 -07:00
Marco Neumann 335d9f7357
chore: minimize proptest features (#7993) 2023-06-14 12:28:18 +00:00
Marco Neumann 1762172321
refactor: remove `procspawn` (#7990) 
`procspawn` indirectly pulls in many outdated dependencies. It was
introduced in #7850 to work around a flaky test. Isolating the test into
its own binary has the same affect and requires less dependencies.
 2023-06-14 11:09:37 +00:00
Marco Neumann c4d475cb86
chore: prune mockito dependencies (#7991) 
We don't really need the color feature. Removing it has several
benifits:

- less dependencies
- one less MPL license
- no `atty` (only used by tests but has a security bug)
 2023-06-14 10:33:30 +00:00
dependabot[bot] 98a2e852db
chore(deps): Bump uuid from 1.3.3 to 1.3.4 (#7985) 
Bumps [uuid](https://github.com/uuid-rs/uuid) from 1.3.3 to 1.3.4.
- [Release notes](https://github.com/uuid-rs/uuid/releases)
- [Commits](https://github.com/uuid-rs/uuid/compare/1.3.3...1.3.4)

---
updated-dependencies:
- dependency-name: uuid
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-14 07:58:52 +00:00
Dom 3b7c2bc3e5
Merge branch 'main' into dom/partition-key-truncate 2023-06-13 14:02:14 +01:00
dependabot[bot] 2ffa9f3cda
chore(deps): Bump crossbeam-utils from 0.8.15 to 0.8.16 
Bumps [crossbeam-utils](https://github.com/crossbeam-rs/crossbeam) from 0.8.15 to 0.8.16.
- [Release notes](https://github.com/crossbeam-rs/crossbeam/releases)
- [Changelog](https://github.com/crossbeam-rs/crossbeam/blob/master/CHANGELOG.md)
- [Commits](https://github.com/crossbeam-rs/crossbeam/compare/crossbeam-utils-0.8.15...crossbeam-utils-0.8.16)

---
updated-dependencies:
- dependency-name: crossbeam-utils
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-06-13 02:00:14 +00:00
kodiakhq[bot] c2c614c765
Merge branch 'main' into savage/inspect-wal-contents 2023-06-12 14:22:05 +00:00
dependabot[bot] 792b991778
chore(deps): Bump rustix from 0.37.19 to 0.37.20 (#7969) 
Bumps [rustix](https://github.com/bytecodealliance/rustix) from 0.37.19 to 0.37.20.
- [Release notes](https://github.com/bytecodealliance/rustix/releases)
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.37.19...v0.37.20)

---
updated-dependencies:
- dependency-name: rustix
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-12 11:37:11 +00:00
dependabot[bot] 41386078fe
chore(deps): Bump chrono from 0.4.24 to 0.4.26 (#7971) 
* chore(deps): Bump chrono from 0.4.24 to 0.4.26

Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.24 to 0.4.26.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md)
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.24...v0.4.26)

---
updated-dependencies:
- dependency-name: chrono
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
 2023-06-12 09:12:50 +00:00
dependabot[bot] 5db690f214
chore(deps): Bump clap from 4.3.2 to 4.3.3 (#7968) 
Bumps [clap](https://github.com/clap-rs/clap) from 4.3.2 to 4.3.3.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.3.2...v4.3.3)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-12 08:57:03 +00:00
dependabot[bot] 19b0fbc81c
chore(deps): Bump log from 0.4.18 to 0.4.19 (#7966) 
Bumps [log](https://github.com/rust-lang/log) from 0.4.18 to 0.4.19.
- [Release notes](https://github.com/rust-lang/log/releases)
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/log/compare/0.4.18...0.4.19)

---
updated-dependencies:
- dependency-name: log
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-12 08:38:44 +00:00
Fraser Savage 73c0c28bd0
feat(cli): Add `influxdb_iox debug wal inspect` command 
This commit adds an `inspect` command to read through the sequenced
operations in a WAL file and debug pretty print their contents to
stdout, optionally filtering by a sequence number range.
 2023-06-09 18:16:57 +01:00
Dom Dwyer 050093df1e
feat: truncate partition key parts at 200 bytes 
This commit ensures all partition key parts are less than or equal to
200 bytes long.

If a string exceeds the 200 byte limit, it is truncated (avoiding
splitting unicode code-points or graphemes) and then a single "#"
sentinel value is appended. When reversed from the string, these column
values are indicated to be suitable for prefix-matching only - a
property that is encoded into the type system.

This commit takes a conservative approach of not splitting graphemes as
outlined in the module documentation, but this could be relaxed in the
future if needed.
 2023-06-09 13:44:32 +02:00
dependabot[bot] 5acca21f12
chore(deps): Bump serde from 1.0.163 to 1.0.164 (#7951) 
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.163 to 1.0.164.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.163...v1.0.164)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
 2023-06-08 09:20:19 +00:00
CircleCI[bot] 01732fd7b0 chore: Run cargo hakari tasks 2023-06-08 08:45:49 +00:00
dependabot[bot] 4531be115d
chore(deps): Bump hashbrown from 0.13.2 to 0.14.0 
Bumps [hashbrown](https://github.com/rust-lang/hashbrown) from 0.13.2 to 0.14.0.
- [Changelog](https://github.com/rust-lang/hashbrown/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/hashbrown/compare/v0.13.2...v0.14.0)

---
updated-dependencies:
- dependency-name: hashbrown
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-06-08 08:44:40 +00:00
Dom be75ba23e0
Merge branch 'main' into dependabot/cargo/hashbrown-0.14.0 2023-06-08 09:32:01 +01:00
Andrew Lamb 17c0d837b3
chore: Update DataFusion, arrow, object_store pins (#7942) 
* chore: Update DataFusion, arrow, object_store pins

* chore: Update for hakari

* chore: Update for new APIs

* fix: update test

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-07 17:08:31 +00:00
dependabot[bot] fcd9d9e3e6
chore(deps): Bump hashbrown from 0.13.2 to 0.14.0 
Bumps [hashbrown](https://github.com/rust-lang/hashbrown) from 0.13.2 to 0.14.0.
- [Changelog](https://github.com/rust-lang/hashbrown/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/hashbrown/compare/v0.13.2...v0.14.0)

---
updated-dependencies:
- dependency-name: hashbrown
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-06-07 11:52:16 -04:00
Carol (Nichols || Goulding) d0db1194e2
feat: Validate custom partition templates on their creation 
Make sure custom partition templates have:

- At least one part
- No more than 8 parts
- Only nonempty, valid strftime formats
 2023-06-07 11:38:12 -04:00
dependabot[bot] ccd11b52b0
chore(deps): Bump nu-ansi-term from 0.47.0 to 0.48.0 (#7936) 
Bumps [nu-ansi-term](https://github.com/nushell/nu-ansi-term) from 0.47.0 to 0.48.0.
- [Release notes](https://github.com/nushell/nu-ansi-term/releases)
- [Changelog](https://github.com/nushell/nu-ansi-term/blob/main/CHANGELOG.md)
- [Commits](https://github.com/nushell/nu-ansi-term/compare/v0.47.0...v0.48.0)

---
updated-dependencies:
- dependency-name: nu-ansi-term
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-07 09:02:33 +00:00
dependabot[bot] f7f0151fd6
chore(deps): Bump getrandom from 0.2.9 to 0.2.10 (#7945) 
Bumps [getrandom](https://github.com/rust-random/getrandom) from 0.2.9 to 0.2.10.
- [Changelog](https://github.com/rust-random/getrandom/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-random/getrandom/compare/v0.2.9...v0.2.10)

---
updated-dependencies:
- dependency-name: getrandom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
 2023-06-07 08:56:40 +00:00
dependabot[bot] 7b6efae62c
chore(deps): Bump tempfile from 3.5.0 to 3.6.0 
Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.5.0 to 3.6.0.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.5.0...v3.6.0)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-06-07 08:21:40 +00:00
dependabot[bot] f7d84a8438
chore(deps): Bump comfy-table from 6.2.0 to 7.0.0 (#7944) 
Bumps [comfy-table](https://github.com/nukesor/comfy-table) from 6.2.0 to 7.0.0.
- [Release notes](https://github.com/nukesor/comfy-table/releases)
- [Changelog](https://github.com/Nukesor/comfy-table/blob/main/CHANGELOG.md)
- [Commits](https://github.com/nukesor/comfy-table/compare/v6.2.0...v7.0.0)

---
updated-dependencies:
- dependency-name: comfy-table
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
 2023-06-07 08:19:53 +00:00
dependabot[bot] 67956efb8e
chore(deps): Bump libc from 0.2.145 to 0.2.146 
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.145 to 0.2.146.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.145...0.2.146)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-06-07 02:00:09 +00:00
dependabot[bot] c0a683b9f2
chore(deps): Bump lock_api from 0.4.9 to 0.4.10 (#7932) 
* chore(deps): Bump lock_api from 0.4.9 to 0.4.10

Bumps [lock_api](https://github.com/Amanieu/parking_lot) from 0.4.9 to 0.4.10.
- [Changelog](https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Amanieu/parking_lot/compare/lock_api-0.4.9...lock_api-0.4.10)

---
updated-dependencies:
- dependency-name: lock_api
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-06 12:45:25 +00:00
dependabot[bot] 8a06483e3b
chore(deps): Bump clap from 4.3.1 to 4.3.2 
Bumps [clap](https://github.com/clap-rs/clap) from 4.3.1 to 4.3.2.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/clap_complete-v4.3.1...v4.3.2)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-06-06 02:02:37 +00:00
Andrew Lamb f571aeb445
chore: Update DataFusion pin (#7916) 
* chore: Update DataFusion pin

* chore: Update cargo

* fix: update for API changes

* fix: Update plans

* chore: Update for new api

* fix: Update plans

* chore: Update for API changes more

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-05 18:38:59 +00:00
dependabot[bot] a0686c5f95
chore(deps): Bump regex from 1.8.3 to 1.8.4 (#7928) 
Bumps [regex](https://github.com/rust-lang/regex) from 1.8.3 to 1.8.4.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.8.3...1.8.4)

---
updated-dependencies:
- dependency-name: regex
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-05 15:50:26 +00:00
dependabot[bot] 3f2960f0c7
chore(deps): Bump url from 2.3.1 to 2.4.0 (#7925) 
* chore(deps): Bump url from 2.3.1 to 2.4.0

Bumps [url](https://github.com/servo/rust-url) from 2.3.1 to 2.4.0.
- [Release notes](https://github.com/servo/rust-url/releases)
- [Commits](https://github.com/servo/rust-url/compare/v2.3.1...v2.4.0)

---
updated-dependencies:
- dependency-name: url
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-05 14:24:42 +00:00
dependabot[bot] ee61e954bf
chore(deps): Bump flatbuffers from 23.1.21 to 23.5.26 (#7922) 
Bumps [flatbuffers](https://github.com/google/flatbuffers) from 23.1.21 to 23.5.26.
- [Release notes](https://github.com/google/flatbuffers/releases)
- [Changelog](https://github.com/google/flatbuffers/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/flatbuffers/compare/v23.1.21...v23.5.26)

---
updated-dependencies:
- dependency-name: flatbuffers
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-06-05 09:41:08 +00:00
dependabot[bot] 9bf7dee628
chore(deps): Bump libc from 0.2.144 to 0.2.145 (#7921) 
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.144 to 0.2.145.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.144...0.2.145)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
 2023-06-05 09:34:15 +00:00
dependabot[bot] d8b06c59c4
chore(deps): Bump once_cell from 1.17.2 to 1.18.0 
Bumps [once_cell](https://github.com/matklad/once_cell) from 1.17.2 to 1.18.0.
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md)
- [Commits](https://github.com/matklad/once_cell/compare/v1.17.2...v1.18.0)

---
updated-dependencies:
- dependency-name: once_cell
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-06-05 02:03:15 +00:00
dependabot[bot] 00f579086d
chore(deps): Bump clap from 4.3.0 to 4.3.1 
Bumps [clap](https://github.com/clap-rs/clap) from 4.3.0 to 4.3.1.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/clap_complete-v4.3.0...clap_complete-v4.3.1)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-06-02 15:24:33 +00:00
kodiakhq[bot] 1d6fd83a9a
Merge branch 'main' into savage/wal-regenerate-lp-catalog-support 2023-06-02 14:23:55 +00:00
Fraser Savage e9b5708c70
refactor(cli): Perform `regenerate-lp` using a sorted output comparison 
Query the ingester directly through the test cluster to allow for less
brittle assertion of results.
 2023-06-02 13:43:44 +01:00
Fraser Savage 50797b6967
test(cli): Assert writing `regenerate-lp` output produces same query results 
This changes the e2e test to delete the WAL segment file, restart the
ingester and ensure the results returned by an ingester query after
feeding the regenerated line proto in are the same as those before.
 2023-06-02 12:45:52 +01:00
Dom 9d445cc935
Merge branch 'main' into dom/strftime-panic 2023-06-02 10:03:39 +01:00
Stuart Carnie d743f071b5
Merge branch 'main' into sgc/issue/7829_time_bounds_2 2023-06-02 14:00:05 +10:00
Stuart Carnie d5719f9be2
refactor: Moved simplification of time range expressions to parser 2023-06-02 09:50:01 +10:00
Dom Dwyer 47214ec9a0
fix: prevent panics in partitioning logic 
Changes the partitioning logic to be fallible. This prevents an invalid
partition template from causing a panic, previously possible through two
known code paths:

    * TagValue formatter referencing a non-tag column
    * Time formatter using an invalid strftime format string

If either occurs, the write attempt is now aborted and an error returned
to the user with a HTTP 500 status code.

Additionally unexpected partitioner errors now map to a catch-all error
instead of panicking.
 2023-06-01 17:44:44 +02:00
Dom 4b0753a800
Merge branch 'main' into dom/reversible-partition-key 2023-05-31 16:04:16 +01:00
Andrew Lamb a48f681e56
feat(parquet): reduce and limit buffering when writing parquet files (#7880) 
* feat: limit buffering when writing parquet files ("combined solution")

* chore: Run cargo hakari tasks

---------

Co-authored-by: Raphael Taylor-Davies <r.taylordavies@googlemail.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-05-31 13:27:32 +00:00
Marco Neumann e1c1908a0b
refactor: add `parquet_file` PG index for querier (#7842) 
* refactor: add `parquet_file` PG index for querier

Currently the `list_by_table_not_to_delete` catalog query is somewhat
expensive:

```text
iox_catalog_prod=> select table_id, sum((to_delete is NULL)::int) as n from parquet_file group by table_id order by n desc limit 5;
 table_id |  n
----------+------
  1489038 | 7221
  1489037 | 7019
  1491534 | 5793
  1491951 | 5522
  1513377 | 5339
(5 rows)

iox_catalog_prod=> EXPLAIN ANALYZE SELECT id, namespace_id, table_id, partition_id, object_store_id,
       min_time, max_time, to_delete, file_size_bytes,
       row_count, compaction_level, created_at, column_set, max_l0_created_at
FROM parquet_file
WHERE table_id = 1489038 AND to_delete IS NULL;
                                                                          QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on parquet_file  (cost=46050.91..47179.26 rows=283 width=200) (actual time=464.368..472.514 rows=7221 loops=1)
   Recheck Cond: ((table_id = 1489038) AND (to_delete IS NULL))
   Heap Blocks: exact=7152
   ->  BitmapAnd  (cost=46050.91..46050.91 rows=283 width=0) (actual time=463.341..463.343 rows=0 loops=1)
         ->  Bitmap Index Scan on parquet_file_table_idx  (cost=0.00..321.65 rows=22545 width=0) (actual time=1.674..1.674 rows=7221 loops=1)
               Index Cond: (table_id = 1489038)
         ->  Bitmap Index Scan on parquet_file_deleted_at_idx  (cost=0.00..45728.86 rows=1525373 width=0) (actual time=460.717..460.717 rows=4772117 loops=1)
               Index Cond: (to_delete IS NULL)
 Planning Time: 0.092 ms
 Execution Time: 472.907 ms
(10 rows)
```

I think this may also be because PostgreSQL kinda chooses the wrong
strategy, because it could just look at the existing index and filter
from there:

```text
iox_catalog_prod=> EXPLAIN ANALYZE SELECT id, namespace_id, table_id, partition_id, object_store_id,
       min_time, max_time, to_delete, file_size_bytes,
       row_count, compaction_level, created_at, column_set, max_l0_created_at
FROM parquet_file
WHERE table_id = 1489038;
                                                                    QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------
 Index Scan using parquet_file_table_idx on parquet_file  (cost=0.57..86237.78 rows=22545 width=200) (actual time=0.057..6.994 rows=7221 loops=1)
   Index Cond: (table_id = 1489038)
 Planning Time: 0.094 ms
 Execution Time: 7.297 ms
(4 rows)
```

However PostgreSQL doesn't know the cardinalities well enough. So
let's add a dedicated index to make the querier faster.

* feat: new migration system

* docs: explain dirty migrations
 2023-05-31 10:56:32 +00:00
Fraser Savage 24f0dca838
test(cli): Add test to ensure `regenerate-lp` continues on minor errors 
This adds a test to the `wal regenerate-lp` command to ensure that
non-fatal errors do not block regeneration of any other recoverable
entries.
 2023-05-30 18:01:28 +01:00
Dom Dwyer 37bb5e0585
test: arbitrary reversible partition keys 
This test constructs a partition key from an arbitrary selection of
pre-defined parts, and uses the resulting template to partition a write
containing an arbitrary selection of pre-defined tag columns.

Once a partition key is derived, the test asserts build_column_values()
reverses it into the original set of tag (column_name, value) tuples
present in the write.
 2023-05-30 15:58:26 +02:00
Dom Dwyer 27bef292a3
feat: unambiguously reversible partition keys 
This commit changes the format of partition keys when generated with
non-default partition key templates ONLY. A prior fixture test is
unchanged by this commit, ensuring the default partition keys remain
the same.

When a custom partition key template is provided, it may specify one or
more parts, with the TagValue template causing values extracted from tag
columns to appear in the derived partition key.

This commit changes the generated partition key in the following ways:

    * The delimiter of multi-part partition keys; the character used to
      delimit partition key parts is changed from "/" to "|" (the pipe
      character) as it is less likely to occur in user-provided input,
      reducing the encoding overhead.

    * The format of the extracted TagValue values (see below).

Building on the work of custom partition key overrides, where an
immutable partition template is resolved and set at table creation time,
the changes in this PR enable the derived partition key to be
unambiguously reversed into the set of tag (column_name, column_value)
tuples it was generated from for use in query pruning logic. This is
implemented by the build_column_values() method in this commit, which
requires both the template, and the derived partition key.

Prior to this commit, a partition key value extracted from a tag column
was in the form "tagname_x" where "x" is the value and "tagname" is the
name of the tag column it was extracted from. After this commit, the
partition key value is in the form "x"; the column name is removed from
the derived string to reduce the catalog storage overhead (a key driver
of COGS). In the case of a NULL tag value, the sentinel value "!" is
inserted instead of the prior "tagname_" marker. In the case of an empty
string tag value (""), the sentinel "^" value is inserted instead of the
"tagname_-" marker, ensuring the distinction between an empty value and
a not-present tag is preserved.

Additionally tag values utilise percent encoding to encode reserved
characters (part delimiter, empty sentinel character, % itself) to
eliminate deserialisation ambiguity.

Examples of how this has changed derived partition keys, for a template
of [Time(YYYY-MM-DD), TagValue(region), TagValue(bananas)]:

    Write: time=1970-01-01,region=west,other=ignored
        Old: "1970-01-01-region_west-bananas"
        New: "1970-01-01|west|!"

    Write: time=1970-01-01,other=ignored
        Old: "1970-01-01-region-bananas"
        New: "1970-01-01|!|!"
 2023-05-30 15:58:25 +02:00
dependabot[bot] 840692e1f3
chore(deps): Bump once_cell from 1.17.1 to 1.17.2 
Bumps [once_cell](https://github.com/matklad/once_cell) from 1.17.1 to 1.17.2.
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md)
- [Commits](https://github.com/matklad/once_cell/compare/v1.17.1...v1.17.2)

---
updated-dependencies:
- dependency-name: once_cell
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-05-30 02:58:12 +00:00
dependabot[bot] bfc395a30d
chore(deps): Bump comfy-table from 6.1.4 to 6.2.0 (#7883) 
Bumps [comfy-table](https://github.com/nukesor/comfy-table) from 6.1.4 to 6.2.0.
- [Release notes](https://github.com/nukesor/comfy-table/releases)
- [Changelog](https://github.com/Nukesor/comfy-table/blob/main/CHANGELOG.md)
- [Commits](https://github.com/nukesor/comfy-table/compare/v6.1.4...v6.2.0)

---
updated-dependencies:
- dependency-name: comfy-table
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-05-29 09:22:56 +00:00
dependabot[bot] edbecd53af
chore(deps): Bump log from 0.4.17 to 0.4.18 (#7884) 
Bumps [log](https://github.com/rust-lang/log) from 0.4.17 to 0.4.18.
- [Release notes](https://github.com/rust-lang/log/releases)
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/log/compare/0.4.17...0.4.18)

---
updated-dependencies:
- dependency-name: log
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-05-29 09:17:08 +00:00
dependabot[bot] e0720db138
chore(deps): Bump tokio from 1.28.1 to 1.28.2 (#7885) 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.28.1 to 1.28.2.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.28.1...tokio-1.28.2)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-05-29 09:11:24 +00:00
dependabot[bot] e2b9beffad
chore(deps): Bump criterion from 0.5.0 to 0.5.1 
Bumps [criterion](https://github.com/bheisler/criterion.rs) from 0.5.0 to 0.5.1.
- [Changelog](https://github.com/bheisler/criterion.rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/bheisler/criterion.rs/compare/0.5.0...0.5.1)

---
updated-dependencies:
- dependency-name: criterion
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-05-29 02:02:12 +00:00
Christopher M. Wolff 2a07b53879
feat: add more tag predicate rewrite logic for InfluxQL (#7869) 
* feat: add more tag predicate rewrite logic for InfluxQL

* chore: cargo fmt

* chore: fmt

* test: add more tests

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-05-26 21:53:52 +00:00
Fraser Savage 51d59f8216
refactor(`wal_inspect`): Make `LineProtoWriter` namespace unaware 
Instead, the type responsible for initialising it handles namespaced
`Write` initialisation and management, as well as the failure paths that
may need handling. This commit introduces a `NamespaceDemultiplexer`
type with a generic implementation allowing fallible `async` lazy init
of any type from a given `NamespaceId`. This paves the way for catalog-aware
initialisation of `LineProtoWriter`s.
 2023-05-26 17:12:35 +01:00
dependabot[bot] eb0d77f354
chore(deps): Bump regex from 1.8.2 to 1.8.3 (#7873) 
Bumps [regex](https://github.com/rust-lang/regex) from 1.8.2 to 1.8.3.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.8.2...1.8.3)

---
updated-dependencies:
- dependency-name: regex
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-05-26 09:34:25 +00:00
dependabot[bot] ececd0ada7
chore(deps): Bump base64 from 0.21.1 to 0.21.2 (#7874) 
Bumps [base64](https://github.com/marshallpierce/rust-base64) from 0.21.1 to 0.21.2.
- [Changelog](https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/marshallpierce/rust-base64/compare/v0.21.1...v0.21.2)

---
updated-dependencies:
- dependency-name: base64
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
 2023-05-26 09:28:43 +00:00
kodiakhq[bot] c4eca5fecf
Merge branch 'main' into cn/table-create-grpc-api 2023-05-25 18:53:12 +00:00
Andrew Lamb 138b14e0db
chore: Update DataFusion and arrow to `40.0.0` (#7864) 
* chore: Update DataFusion and arrow to `40.0.0`

* chore: Run cargo hakari tasks

* fix: update for API

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-05-25 17:34:51 +00:00
Carol (Nichols || Goulding) 17219d71fe
feat: Use the table service in the router 2023-05-25 10:44:57 -04:00
Carol (Nichols || Goulding) e1a93252c5
feat: Add a new table service crate 2023-05-25 10:44:57 -04:00
dependabot[bot] 89d8207784
chore(deps): Bump io-lifetimes from 1.0.10 to 1.0.11 (#7865) 
Bumps [io-lifetimes](https://github.com/sunfishcode/io-lifetimes) from 1.0.10 to 1.0.11.
- [Commits](https://github.com/sunfishcode/io-lifetimes/compare/v1.0.10...v1.0.11)

---
updated-dependencies:
- dependency-name: io-lifetimes
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-05-25 07:49:53 +00:00
Carol (Nichols || Goulding) c8712bbc90
fix: Add a fixture test encoding and documenting default partition template assumptions 2023-05-24 10:36:52 -04:00
Carol (Nichols || Goulding) 604bab9508
fix: Make Table create_or_get be only create 2023-05-24 10:34:30 -04:00
Carol (Nichols || Goulding) afb3838437
feat: Optionally supply the namespace partition template when creating a namespace 2023-05-24 10:10:34 -04:00
dependabot[bot] 24a4f36d24
chore(deps): Bump proptest from 1.1.0 to 1.2.0 (#7857) 
Bumps [proptest](https://github.com/proptest-rs/proptest) from 1.1.0 to 1.2.0.
- [Release notes](https://github.com/proptest-rs/proptest/releases)
- [Changelog](https://github.com/proptest-rs/proptest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/proptest-rs/proptest/compare/v1.1.0...v1.2.0)

---
updated-dependencies:
- dependency-name: proptest
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-05-24 09:21:32 +00:00
dependabot[bot] b7fbfa6fb2
chore(deps): Bump criterion from 0.4.0 to 0.5.0 (#7856) 
Bumps [criterion](https://github.com/bheisler/criterion.rs) from 0.4.0 to 0.5.0.
- [Changelog](https://github.com/bheisler/criterion.rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/bheisler/criterion.rs/compare/0.4.0...0.5.0)

---
updated-dependencies:
- dependency-name: criterion
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-05-24 09:08:37 +00:00
Dom Dwyer 928a4d163e
build: remove unused dependencies from crates 
This commit fixes loads of crates (47!) had unused dependencies, or
mis-configured dependencies (test deps as normal deps).

I added the "unused_crate_dependencies" to all crates to help prevent
this mess from growing again!

    https://doc.rust-lang.org/beta/nightly-rustc/rustc_lint_defs/builtin/static.UNUSED_CRATE_DEPENDENCIES.html

This has the minor downside of false-positives when specifying
dev-dependencies for test/bench binaries - these are files in /test or
/benches (not normal tests). This commit includes a workaround,
importing them in lib.rs (gated by a feature flag). I think the
trade-off of better dependency management is worth it!
 2023-05-23 14:55:43 +02:00
dependabot[bot] ef0bfb6186
chore(deps): Bump regex from 1.8.1 to 1.8.2 (#7845) 
Bumps [regex](https://github.com/rust-lang/regex) from 1.8.1 to 1.8.2.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.8.1...1.8.2)

---
updated-dependencies:
- dependency-name: regex
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-05-23 12:37:03 +00:00
Marco Neumann c5ee3250f7
fix: flaky `service::tests::test_log_on_panic` (#7850) 
* refactor: remove line number from test

* refactor: clean up code

* fix: flaky `service::tests::test_log_on_panic`

This seems to be a really rare issue. So provoking it requires a lot of
tests. This can be done (using `fish`):

```console
$ while cargo with "parallel -n0 {bin} {args} ::: $(seq 10)" -- test -p service_grpc_influxrpc --lib -- --test-threads 20; echo; end
```

Fixes #7838.

* fix: panic handler construction

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-05-23 12:17:49 +00:00
kodiakhq[bot] b9bcaf1aa0
Merge branch 'main' into savage/wal-regenerate-lp-cli-command 2023-05-22 16:18:44 +00:00
Dom Dwyer 0719928800
chore: remove unused deps 
The wal crate imports a bunch of stuff it never uses!
 2023-05-22 13:38:49 +02:00
dependabot[bot] f3f1a5edef
chore(deps): Bump sqlparser from 0.33.0 to 0.34.0 (#7834) 
* chore(deps): Bump sqlparser from 0.33.0 to 0.34.0

Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs) from 0.33.0 to 0.34.0.
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.33.0...v0.34.0)

---
updated-dependencies:
- dependency-name: sqlparser
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

* fix: sqlparser compat

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
 2023-05-22 10:16:38 +00:00
dependabot[bot] 6cb7619d83
chore(deps): Bump base64 from 0.21.0 to 0.21.1 (#7832) 
Bumps [base64](https://github.com/marshallpierce/rust-base64) from 0.21.0 to 0.21.1.
- [Changelog](https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/marshallpierce/rust-base64/commits)

---
updated-dependencies:
- dependency-name: base64
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-05-22 09:50:06 +00:00
dependabot[bot] 63b3279e33
chore(deps): Bump digest from 0.10.6 to 0.10.7 (#7835) 
Bumps [digest](https://github.com/RustCrypto/traits) from 0.10.6 to 0.10.7.
- [Commits](https://github.com/RustCrypto/traits/compare/digest-v0.10.6...digest-v0.10.7)

---
updated-dependencies:
- dependency-name: digest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-05-22 08:48:17 +00:00
dependabot[bot] bf847584d4
chore(deps): Bump clap from 4.2.7 to 4.3.0 (#7833) 
Bumps [clap](https://github.com/clap-rs/clap) from 4.2.7 to 4.3.0.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.2.7...clap_complete-v4.3.0)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-05-22 08:33:05 +00:00
dependabot[bot] 8cd96f7238
chore(deps): Bump toml from 0.7.3 to 0.7.4 (#7827) 
Bumps [toml](https://github.com/toml-rs/toml) from 0.7.3 to 0.7.4.
- [Commits](https://github.com/toml-rs/toml/compare/toml-v0.7.3...toml-v0.7.4)

---
updated-dependencies:
- dependency-name: toml
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
 2023-05-19 09:25:49 +00:00
Christopher M. Wolff 90a25a3ff0
chore: update DataFusion (#7825) 
* chore: update DataFusion

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2023-05-18 17:51:16 +00:00
Andrew Lamb e97c48b388
feat: download catalog objects as part of `remote store get-table` (#7779) 
* feat: download catalog objects as part of remote store get-table

* fix: fix future_not_send

* fix: reorder import order

* chore: reduce repetition of `Error` in naming

* fix: improve the partition filtering API

* fix: clarify export comment

* fix: make index printing consistent

* fix: improve skip message

* chore: add extra clippy lints to inmport_export crate

* fix: comments

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-05-18 16:52:43 +00:00
Fraser Savage c263585d94
feat(cli): Add table ID `debug wal regenerate-lp` command implementation 
This adds a command to `influxdb_iox` that can take a WAL segment file
and regenerate all write operation entries, writing to stdout or namespaced
files within a target directory, using table ID as the measurement name
in the case where there is no catalog access at point of regeneration.
 2023-05-18 17:20:41 +01:00
dependabot[bot] 43a1f7c0b2
chore(deps): Bump rustls from 0.21.0 to 0.21.1 (#7819) 
Bumps [rustls](https://github.com/rustls/rustls) from 0.21.0 to 0.21.1.
- [Changelog](https://github.com/rustls/rustls/blob/main/RELEASE_NOTES.md)
- [Commits](https://github.com/rustls/rustls/compare/v/0.21.0...v/0.21.1)

---
updated-dependencies:
- dependency-name: rustls
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
 2023-05-18 09:29:26 +00:00
Marco Neumann 62fed73bcd
refactor: upgrade DataFusion to `19b03240920ad63cac916b42951754c0337bdac8#19b03240920ad63cac916b42951754c0337bdac8` (#7813) 
I need:

- https://github.com/apache/arrow-datafusion/pull/6226.

Changes in code due to:

- https://github.com/apache/arrow-datafusion/pull/6332

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-05-17 13:57:12 +00:00
dependabot[bot] e51ca3bc2d
chore(deps): Bump reqwest from 0.11.17 to 0.11.18 (#7806) 
* chore(deps): Bump reqwest from 0.11.17 to 0.11.18

Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.17 to 0.11.18.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.17...v0.11.18)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-05-17 06:15:24 +00:00
wiedld 506bd80f6f
Merge branch 'main' into chore/router-metrics-for-auth-v2 2023-05-16 10:24:42 -07:00
Andrew Lamb c077dec0e0
refactor: rename `import` to `iox_import_export` (#7799) 
* refactor: rename `import` crate to `import_export`

* chore: Consolidate tsm import/export

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-05-16 17:10:17 +00:00
Chunchun Ye 2bb6445668
chore: update DataFusion and arrow / arrow-flight / parquet to `39.0.0` (#7793) 
* chore: update DataFusion and arrow/parquet/arrow-flight to 39.0.0

* chore: update DataFusion and arrow/parquet/arrow-flight to 39.0.0 in workspace-hack/Cargo.toml

* chore: Run cargo hakari tasks

* chore: fix CI test and lint

* chore: update csv schema

* refactor: remove type-annotate for `Arc`

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-05-16 13:42:26 +00:00
kodiakhq[bot] 6bfa269896
Merge branch 'main' into savage/wal-inspect-library-refactor 2023-05-16 11:07:47 +00:00
dependabot[bot] 3462e29859
chore(deps): Bump uuid from 1.3.2 to 1.3.3 
Bumps [uuid](https://github.com/uuid-rs/uuid) from 1.3.2 to 1.3.3.
- [Release notes](https://github.com/uuid-rs/uuid/releases)
- [Commits](https://github.com/uuid-rs/uuid/compare/1.3.2...1.3.3)

---
updated-dependencies:
- dependency-name: uuid
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-05-16 02:00:24 +00:00
wiedld de3ff7aa2d chore: update MockAuthorizer to use return call pattern 2023-05-15 09:26:09 -07:00
wiedld db76a85a4d chore: create duration metric decorator for Authorizer 2023-05-15 08:36:24 -07:00
wiedld 867fd39dbf
Merge branch 'main' into authz/refactor-interface 2023-05-15 08:03:10 -07:00
kodiakhq[bot] e14bc925db
Merge branch 'main' into cn/move-peas 2023-05-15 14:36:18 +00:00
Andrew Lamb 7735e7c95b
chore: Update DataFusion again (#7777) 
* chore: Update datafusion again

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-05-15 12:38:45 +00:00
Dom 6aa634c1b9
Merge branch 'main' into cn/move-peas 2023-05-15 13:29:42 +01:00
dependabot[bot] fba9836f2a
chore(deps): Bump pin-project from 1.0.12 to 1.1.0 
Bumps [pin-project](https://github.com/taiki-e/pin-project) from 1.0.12 to 1.1.0.
- [Release notes](https://github.com/taiki-e/pin-project/releases)
- [Changelog](https://github.com/taiki-e/pin-project/blob/main/CHANGELOG.md)
- [Commits](https://github.com/taiki-e/pin-project/compare/v1.0.12...v1.1.0)

---
updated-dependencies:
- dependency-name: pin-project
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-05-15 02:02:32 +00:00
wiedld 62d83b9219 test: add tests for IoxAuthorizer contract 2023-05-12 15:31:12 -05:00
wiedld ea3029ee3f refactor: break out Authorizer trait versus IoxAuthorizer struct. 
* make the boundaries more evident
* also make explicit what actions are tied to the IoxAuthorizer client (a.k.a. the client connection & request)
 2023-05-12 15:28:58 -05:00
Carol (Nichols || Goulding) 14007808bd
fix: Move remaining conversions between data types and proto into data_types 
And have data_types depend on generated_types rather than vice versa.
 2023-05-12 13:31:04 -04:00
Carol (Nichols || Goulding) 1770d0f4d8
fix: Move ingester-querier gRPC communication to its own crate 2023-05-12 13:28:30 -04:00
dependabot[bot] ac0ea99cde
chore(deps): Bump tracing-core from 0.1.30 to 0.1.31 (#7781) 
Bumps [tracing-core](https://github.com/tokio-rs/tracing) from 0.1.30 to 0.1.31.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-core-0.1.30...tracing-core-0.1.31)

---
updated-dependencies:
- dependency-name: tracing-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
 2023-05-12 13:40:04 +00:00
dependabot[bot] 6c185bdec8
chore(deps): Bump handlebars from 4.3.6 to 4.3.7 
Bumps [handlebars](https://github.com/sunng87/handlebars-rust) from 4.3.6 to 4.3.7.
- [Release notes](https://github.com/sunng87/handlebars-rust/releases)
- [Changelog](https://github.com/sunng87/handlebars-rust/blob/v4.3.7/CHANGELOG.md)
- [Commits](https://github.com/sunng87/handlebars-rust/compare/v4.3.6...v4.3.7)

---
updated-dependencies:
- dependency-name: handlebars
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-05-12 13:11:56 +00:00
dependabot[bot] bd1d1f648c
chore(deps): Bump serde from 1.0.162 to 1.0.163 (#7780) 
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.162 to 1.0.163.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.162...v1.0.163)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-05-12 13:07:56 +00:00
dependabot[bot] 36c1d23e19
chore(deps): Bump tokio from 1.28.0 to 1.28.1 (#7776) 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.28.0 to 1.28.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.28.0...tokio-1.28.1)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-05-11 19:46:58 +00:00
dependabot[bot] d99444fb2e
chore(deps): Bump console-subscriber from 0.1.8 to 0.1.9 
Bumps [console-subscriber](https://github.com/tokio-rs/console) from 0.1.8 to 0.1.9.
- [Release notes](https://github.com/tokio-rs/console/releases)
- [Commits](https://github.com/tokio-rs/console/compare/console-subscriber-v0.1.8...console-subscriber-v0.1.9)

---
updated-dependencies:
- dependency-name: console-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-05-10 01:59:08 +00:00
Carol (Nichols || Goulding) 23c0110b32
feat: Create newtypes for different partition templates 
So that the different kinds aren't mixed up. Also extracts the logic
having to do with which template takes precedence onto the
PartitionTemplate type itself.
 2023-05-09 14:55:02 +02:00
Dom 372ec8ef96
Merge branch 'main' into cn/delete-experiments 2023-05-09 10:17:30 +01:00
Carol (Nichols || Goulding) 9229ce5668
fix: Rename compactor2_test_utils to compactor_test_utils 2023-05-09 11:02:11 +02:00
Carol (Nichols || Goulding) 45493b91e5
fix: Rename ioxd_compactor2 to ioxd_compactor 2023-05-09 11:02:11 +02:00
Carol (Nichols || Goulding) dd9c5d1b13
fix: Rename compactor2 to compactor 2023-05-09 10:58:55 +02:00
Carol (Nichols || Goulding) 6000d1e8d4
fix: Remove unused crates 2023-05-08 20:24:56 -04:00
Carol (Nichols || Goulding) 3d5df5574a
fix: Remove vestiges of shards 2023-05-08 20:24:36 -04:00
Carol (Nichols || Goulding) 0849ce6f2b
fix: Rename ingester2_test_ctx to ingester_test_ctx 2023-05-08 20:23:02 -04:00
Carol (Nichols || Goulding) 19a56bdf4b
fix: Rename ioxd_ingester2 to ioxd_ingester 2023-05-08 20:20:03 -04:00
dependabot[bot] 68117aee0f
chore(deps): Bump libc from 0.2.142 to 0.2.144 (#7766) 
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.142 to 0.2.144.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.142...0.2.144)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-05-08 19:12:58 +00:00
Carol (Nichols || Goulding) 56916cf942
fix: Rename ingester2 to ingester 2023-05-08 12:03:05 -04:00
Andrew Lamb 2860d87fe1
chore: Update DataFusion (#7756) 
* chore: Update DataFusion pin

* chore: Update explain plans

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2023-05-05 18:58:18 +00:00
dependabot[bot] aa70b14fe4
chore(deps): Bump rustix from 0.37.18 to 0.37.19 (#7752) 
Bumps [rustix](https://github.com/bytecodealliance/rustix) from 0.37.18 to 0.37.19.
- [Release notes](https://github.com/bytecodealliance/rustix/releases)
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.37.18...v0.37.19)

---
updated-dependencies:
- dependency-name: rustix
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-05-05 15:47:26 +00:00
dependabot[bot] da6e123e9d
chore(deps): Bump serde from 1.0.160 to 1.0.162 (#7751) 
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.160 to 1.0.162.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.160...1.0.162)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-05-05 13:29:50 +00:00
Andrew Lamb 667da5eea4
chore: Add test for `dsn-file://` catalog urls (#7735) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-05-04 10:48:18 +00:00
Marco Neumann abe5d26d2f
chore: update DataFusion to `2787e7a36a6be83d91201df20827d3695f933300` (#7732) 
Required to get:

- https://github.com/apache/arrow-datafusion/pull/6199
 2023-05-03 16:40:00 +00:00
Fraser Savage b2e5ea2266
refactor(wal): Add test & docs for WriteOpEntryDecoder 
Adds some documentation for the WriteOpEntryDecoder and
a unit test that asserts it skips over non write entries
and can continue to be consumed from.
 2023-05-03 11:51:00 +01:00
Fraser Savage f6dea224e8
feat(wal): Add wal_inspect crate & a write op entry decoder 
This adds a new crate with a type capable of converting
decoded WAL Write Op entries to line protocol and writing
the result to a namespaced destination. The wal crate
now exports a type which reads the sequenced wal ops and
decodes them as namespaced table batch writes.
 2023-05-03 11:50:59 +01:00
dependabot[bot] bd07add78d
chore(deps): Bump clap from 4.2.5 to 4.2.7 (#7726) 
Bumps [clap](https://github.com/clap-rs/clap) from 4.2.5 to 4.2.7.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.2.5...v4.2.7)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-05-03 09:39:10 +00:00
Andrew Lamb 65ca52ce3b
chore: Update DataFusion again (#7722) 
* chore: Update DF again

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2023-05-02 18:34:45 +00:00
Andrew Lamb 2b1f8b56e2
chore: Update DataFusion (#7719) 
* chore: Update DataFusion

* chore: update for API change

* chore: update some tests

* fix: Update plans in optimizer

* chore: Update plans

* chore: Update error messages

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2023-05-02 17:55:04 +00:00
Utkarsh Gupta 18b02a03cf
chore: move to the smaller, cargo-team maintained `home` crate (#7716) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-05-02 16:10:22 +00:00
Carol (Nichols || Goulding) 0c4a86cbaa
fix: Use humantime rather than chrono-english 2023-05-01 11:31:42 -04:00
Carol (Nichols || Goulding) cd4d961db9
fix: garbage_collector no longer uses chrono-english so it can use the workspace hack crate 2023-05-01 11:31:42 -04:00
dependabot[bot] 67957d6fb5
chore(deps): Bump rustix from 0.37.15 to 0.37.18 (#7701) 
Bumps [rustix](https://github.com/bytecodealliance/rustix) from 0.37.15 to 0.37.18.
- [Release notes](https://github.com/bytecodealliance/rustix/releases)
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.37.15...v0.37.18)

---
updated-dependencies:
- dependency-name: rustix
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-05-01 15:11:37 +00:00
dependabot[bot] b7c3711282
chore(deps): Bump uuid from 1.3.1 to 1.3.2 (#7702) 
Bumps [uuid](https://github.com/uuid-rs/uuid) from 1.3.1 to 1.3.2.
- [Release notes](https://github.com/uuid-rs/uuid/releases)
- [Commits](https://github.com/uuid-rs/uuid/compare/1.3.1...1.3.2)

---
updated-dependencies:
- dependency-name: uuid
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-05-01 14:54:28 +00:00
dependabot[bot] 9146306015
chore(deps): Bump flate2 from 1.0.25 to 1.0.26 (#7703) 
Bumps [flate2](https://github.com/rust-lang/flate2-rs) from 1.0.25 to 1.0.26.
- [Release notes](https://github.com/rust-lang/flate2-rs/releases)
- [Commits](https://github.com/rust-lang/flate2-rs/commits)

---
updated-dependencies:
- dependency-name: flate2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-05-01 14:44:06 +00:00
Andrew Lamb 3bbeb99c9f
fix: Fix cargo_audit using tracing-subscriber (#7710) 2023-05-01 14:09:21 +00:00
dependabot[bot] ea7872760d
chore(deps): Bump reqwest from 0.11.16 to 0.11.17 (#7700) 
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.16 to 0.11.17.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.16...v0.11.17)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-05-01 11:02:23 +00:00
CircleCI[bot] bc8512d74c chore: Run cargo hakari tasks 2023-04-28 09:45:56 +00:00
Dom Dwyer 411c9f6c1c
chore(clap_blocks): remove unused dep 
Config & logic are now decoupled!
 2023-04-28 11:40:46 +02:00
dependabot[bot] f575370dc4
chore(deps): Bump clap from 4.2.4 to 4.2.5 
Bumps [clap](https://github.com/clap-rs/clap) from 4.2.4 to 4.2.5.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.2.4...v4.2.5)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-04-28 02:00:45 +00:00
Phil Bracikowski 8609015821
chore(garbage collector): backoff first connect to objectstore (#7652) 
* chore(garbage collector): backoff first connect to objectstore

This pr replaces the initial sleep added in #7562 with expential backoff
retry of connecting to objectstore. This avoids the issue of waiting the
configured sleep which can be quite long in a world where the service is
getting redeployed often.

* for #7562

* chore: rewrite lister::perform based on pr feedback

This commit redoes perform as a do..while loop, putting the call to list
from the object store at the top of the loop so the infinite backoff
retry and be used for each loop iteration - it might fail on more than
the first time!

There are 3 selects as there are 3 wait stages and each needs to check
for shutdown: os list, processing the list, and sleeping on the poll
interval.

* chore: hoist cancellation check higher; limit listing to 1000 files

Responding to PR feedback.

* chore: add error info message

* chore: make build. :|

* chore: linter
 2023-04-27 17:51:52 +00:00
Marco Neumann e67ed1fd0b
feat: classify panics (#7681) 
We are currently aware that Arrow may panic for overlarge string
columns. Howeve we would still like to get distinct alerts for unknown
panics. Hence we shall classify the panic both on our log messages and
our metrics.
 2023-04-27 13:12:35 +00:00
dependabot[bot] 35b5d57353
chore(deps): Bump rustix from 0.37.14 to 0.37.15 (#7677) 
Bumps [rustix](https://github.com/bytecodealliance/rustix) from 0.37.14 to 0.37.15.
- [Release notes](https://github.com/bytecodealliance/rustix/releases)
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.37.14...v0.37.15)

---
updated-dependencies:
- dependency-name: rustix
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-04-27 08:08:36 +00:00
Dom be3256d1a7
Merge branch 'main' into dom/proptest-cache 2023-04-26 14:55:58 +01:00
Martin Hilton 4b24c988ad
feat(service_grpc_flight): JDBC compatible Handshake (#7660) 
* refactor(authz): move extract_header_token into authz

Move the extract_header_token method into the authz package so that
it can be shared by the query path. The method is renamed to reflect
the fact that it can now also extract a token from gRPC metadata.

The extract_token function is now a little more generic to allow
it to be used with HTTP header values and gRPC metadata values.

* feat(service_grpc_flight): JDBC compatible Handshake

While testing some JDBC based clients we found that some, Tableau
in this case,  cannot be configured with authoriztion tokens. In
these cases we need to be able to support username/password. The
approach taken is to ignore the username and make the token the
password. This is the same approach being taken throughout the
product.

To facilitate this the Flight RPC Handshake command has been extended
to look for Basic authorization credentials and respond with the
appropriate Bearer authorization header.

While adding end-to-end tests the subprocess commands were causing
a deadlock. These have been changed to using the tonic::process
module.

There are also some small changes to the JDBC test application where
the hardcoded values were clashing with the authorization parameters.

* fix: lint

* chore: apply suggestions from code review

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

* chore: review suggestion

---------

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-04-26 13:52:49 +00:00
Dom Dwyer 9cc58eb4e4
test: property test namespace schema merging 
This commit adds a randomised property test, that compares the results
of the new namespace cache schema merging (#7555) with a known-good
stdlib HashSet union (the cache implementation is effectively a more
specialised set union operation).

This property test also validates the "last writer wins" semantics for
other, non-schema data within the namespace.

Additionally the ChangeSet values returned over a pair of updates are
asserted to reflect the actual values added to the cache (but not each
call individually) to ensure accurate metrics are reported.
 2023-04-26 15:45:54 +02:00