36503b04cf
Merge pull request #8541 from influxdata/dom/gossip-schema
...
feat: reusable schema notifications via gossip_schema crate
2023-08-23 11:35:38 +01:00
64ca381775
Merge branch 'main' into dom/gossip-schema
2023-08-23 11:15:21 +01:00
10fd966721
chore(deps): Bump sqlparser from 0.36.1 to 0.37.0 ( #8547 )
...
* chore(deps): Bump sqlparser from 0.36.1 to 0.37.0
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs ) from 0.36.1 to 0.37.0.
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md )
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.36.1...v0.37.0 )
---
updated-dependencies:
- dependency-name: sqlparser
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-23 10:00:35 +00:00
a458173866
chore(deps): Bump backtrace from 0.3.68 to 0.3.69 ( #8546 )
...
Bumps [backtrace](https://github.com/rust-lang/backtrace-rs ) from 0.3.68 to 0.3.69.
- [Release notes](https://github.com/rust-lang/backtrace-rs/releases )
- [Commits](https://github.com/rust-lang/backtrace-rs/compare/0.3.68...0.3.69 )
---
updated-dependencies:
- dependency-name: backtrace
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-23 09:54:57 +00:00
0dd9036466
Merge pull request #8548 from influxdata/dependabot/cargo/ordered-float-3.9.1
...
chore(deps): Bump ordered-float from 3.9.0 to 3.9.1
2023-08-23 10:48:59 +01:00
71eda2ab71
chore(deps): Bump ordered-float from 3.9.0 to 3.9.1
...
Bumps [ordered-float](https://github.com/reem/rust-ordered-float ) from 3.9.0 to 3.9.1.
- [Release notes](https://github.com/reem/rust-ordered-float/releases )
- [Commits](https://github.com/reem/rust-ordered-float/compare/v3.9.0...v3.9.1 )
---
updated-dependencies:
- dependency-name: ordered-float
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-08-23 01:23:52 +00:00
8bbfc70b96
Merge branch 'main' into dom/gossip-schema
2023-08-22 17:23:40 +01:00
6ea6f77d9f
ci(gossip_schema): add missing lints
...
Adds the "default" lints to the gossip_schema crate.
2023-08-22 18:22:16 +02:00
1a51f3bd53
build(gossip_schema): remove unused deps
...
Some listed dependencies aren't actually used.
2023-08-22 18:22:15 +02:00
17e1541104
docs: fix typos, add handle module docs
...
More docs more better.
2023-08-22 18:22:14 +02:00
b071967847
chore: upgrade `rustls-webpki` to fix `RUSTSEC-2023-0053` ( #8545 )
...
See <https://rustsec.org/advisories/RUSTSEC-2023-0053 >.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-22 13:48:32 +00:00
50ab9afd5e
feat: add schema+batch serde for ingester->querier V2 ( #8498 )
...
* feat: `PartitionIdentifier` serde
* fix: typo
* refactor: use `Bytes` for V2 protocols
* feat: add schema+batch serde for i->q V2
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-22 13:30:47 +00:00
da3b20270b
docs: doc link fixes
...
Fix bad links!
2023-08-22 12:58:41 +02:00
33af90ef9a
chore: Run cargo hakari tasks
2023-08-22 10:51:04 +00:00
2e77507f7b
feat: implement gossip_schema crate
...
Adds a new gossip_schema crate that provides a high-level interface to
schema change notifications.
This crate layers schema-specific interfaces over the existing low-level
gossip crate. Users can obtain best-effort schema change notifications
by implementing a SchemaEventHandler delegate given to a SchemaRx, or
efficiently dispatch schema change notifications to listening peers
using a SchemaTx.
Schema notifications are sent over the Topic::SchemaChanges topic
(ID=1), which the caller must register as an interest on receiving
gossip nodes.
2023-08-22 12:45:22 +02:00
55631a4f83
refactor: ColumnsByName::from_iter()
...
Allow a ColumnsByName to be constructed by collecting a set of (name,
column_schema) tuples.
2023-08-22 12:45:21 +02:00
53a2ece7b3
Merge pull request #8538 from influxdata/dom/gossip-rename
...
refactor(gossip): rename GossipMessage
2023-08-22 11:41:40 +01:00
93b4bf286e
Merge branch 'main' into dom/gossip-rename
2023-08-22 11:36:21 +01:00
cf0185f2a3
revert: #8491 ( #8540 )
...
This reverts #8491 by nullifying the problematic migration and the
aftermath migrations.
2023-08-22 10:35:52 +00:00
84b9d07f52
Merge branch 'main' into dom/gossip-rename
2023-08-22 11:08:02 +01:00
d35bd48f65
refactor(gossip): rename GossipMessage
...
Now there's a Topic, there's no need for a giant "all message types"
enum.
As part of this shift, the gossip_message::GossipMessage used for schema
gossiping is sounding overly generic. This commit changes the name to
schema_message::SchemaMessage and updates the code.
This is a backwards-compatible change (and if anything goes wrong, the
"old" routers simply log a warning if a message is unreadable).
2023-08-22 12:06:49 +02:00
4fc8b946fe
chore: manual `cargo update` ( #8531 )
...
```text
❯ cargo update
Updating crates.io index
Updating git repository `https://github.com/apache/arrow-datafusion.git `
Updating git repository `https://github.com/mkmik/heappy `
Updating aho-corasick v1.0.2 -> v1.0.4
Updating anstyle-wincon v1.0.1 -> v1.0.2
Updating anyhow v1.0.72 -> v1.0.75
Updating axum v0.6.19 -> v0.6.20
Updating clap v4.3.22 -> v4.3.23
Updating clap_builder v4.3.22 -> v4.3.23
Updating const-oid v0.9.4 -> v0.9.5
Updating cpp_demangle v0.4.2 -> v0.4.3
Updating der v0.7.7 -> v0.7.8
Updating errno v0.3.1 -> v0.3.2
Updating httpdate v1.0.2 -> v1.0.3
Updating inferno v0.11.15 -> v0.11.16
Updating linux-raw-sys v0.4.3 -> v0.4.5
Updating matchit v0.7.0 -> v0.7.2
Updating num-complex v0.4.3 -> v0.4.4
Updating pest v2.7.1 -> v2.7.2
Updating pest_derive v2.7.1 -> v2.7.2
Updating pest_generator v2.7.1 -> v2.7.2
Updating pest_meta v2.7.1 -> v2.7.2
Updating petgraph v0.6.3 -> v0.6.4
Updating quote v1.0.32 -> v1.0.33
Updating rustls-webpki v0.101.2 -> v0.101.3
Updating serde v1.0.183 -> v1.0.185
Updating serde_derive v1.0.183 -> v1.0.185
Updating strum_macros v0.25.1 -> v0.25.2
Updating syn v2.0.28 -> v2.0.29
Updating windows-targets v0.48.1 -> v0.48.5
Updating windows_aarch64_gnullvm v0.48.0 -> v0.48.5
Updating windows_aarch64_msvc v0.48.0 -> v0.48.5
Updating windows_i686_gnu v0.48.0 -> v0.48.5
Updating windows_i686_msvc v0.48.0 -> v0.48.5
Updating windows_x86_64_gnu v0.48.0 -> v0.48.5
Updating windows_x86_64_gnullvm v0.48.0 -> v0.48.5
Updating windows_x86_64_msvc v0.48.0 -> v0.48.5
Updating winnow v0.5.1 -> v0.5.14
```
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-22 10:03:23 +00:00
262f9aba0d
chore(deps): Bump petgraph from 0.6.3 to 0.6.4 ( #8536 )
...
Bumps [petgraph](https://github.com/petgraph/petgraph ) from 0.6.3 to 0.6.4.
- [Changelog](https://github.com/petgraph/petgraph/blob/master/RELEASES.rst )
- [Commits](https://github.com/petgraph/petgraph/compare/petgraph@v0.6.3...petgraph@v0.6.4 )
---
updated-dependencies:
- dependency-name: petgraph
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-22 09:27:35 +00:00
d8447779a2
feat: migrate name-based sort keys to ID based on in PG ( #8491 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-22 09:11:43 +00:00
694756ef6d
chore(deps): Bump serde from 1.0.183 to 1.0.185 ( #8535 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.183 to 1.0.185.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.183...v1.0.185 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-22 09:06:06 +00:00
18783f9462
test: harden `end_to_end_cases::debug::build_catalog` ( #8537 )
...
This seems to fail a lot in CI, try to work around it. A proper fix is
tracked under #8287 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-22 08:56:19 +00:00
2f27007861
chore(deps): Bump clap from 4.3.22 to 4.3.23 ( #8530 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 4.3.22 to 4.3.23.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v4.3.22...v4.3.23 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-22 08:49:33 +00:00
c020ec3813
chore(deps): Bump reqwest from 0.11.18 to 0.11.19 ( #8534 )
...
* chore(deps): Bump reqwest from 0.11.18 to 0.11.19
Bumps [reqwest](https://github.com/seanmonstar/reqwest ) from 0.11.18 to 0.11.19.
- [Release notes](https://github.com/seanmonstar/reqwest/releases )
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md )
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.18...v0.11.19 )
---
updated-dependencies:
- dependency-name: reqwest
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-08-22 07:50:49 +00:00
53915f0653
feat: move vertical splitting & detect non-linear data ( #8506 )
...
* chore: test changes and additions in preparation for functional changes
* feat: move vertical splitting to RoundInfo calculation, align splits to L1 files
* chore: insta test churn
* feat: detect non-linear data distribution in vertical splitting
* chore: add tests for non-linear data distribution
* chore: insta churn
* chore: cleanup & comment additions
* chore: some variable renaming
2023-08-21 18:22:25 +00:00
967aef0e9d
chore: Update datafusion ( #8515 )
...
* chore: Update datafusion
* fix: update for API
* fix: Verify unsupported statements, with tests
* fix: update tests
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-21 17:49:21 +00:00
3e98f7ea5c
feat: fill sort_key_ids when partition is inserted and updated ( #8517 )
...
* feat: read null sort_key_ids
* chore: clearer explanation about test strategy
* chore: Apply suggestions from code review
Co-authored-by: Marco Neumann <marco@crepererum.net>
* test: tests that add partition with NULL sort_key_ids
* feat: set sort_key_ids to empty array {} during partition insertion
* feat: initial step to update sort_key_ids
* chore: address review comments
* chore: remove unecessary comments and tests
* fix: typos
* chore: remove unecessary tests
* feat: continue the work of updating sort_key_ids
* fix: chec duplicates for SortedColumnSet
* test: tests for sort ley ids
* test: fix a test
* chore: remove unused comments
* chore: address first half of review comments and removing tests of tests
* chore: address review commnets for fetching colums in ingester
---------
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-21 14:26:57 +00:00
fbb2460c84
chore(deps): Bump cc from 1.0.82 to 1.0.83 ( #8528 )
...
Bumps [cc](https://github.com/rust-lang/cc-rs ) from 1.0.82 to 1.0.83.
- [Release notes](https://github.com/rust-lang/cc-rs/releases )
- [Commits](https://github.com/rust-lang/cc-rs/compare/1.0.82...1.0.83 )
---
updated-dependencies:
- dependency-name: cc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-21 09:24:58 +00:00
158e2a108d
chore(deps): Bump tempfile from 3.7.1 to 3.8.0 ( #8527 )
...
Bumps [tempfile](https://github.com/Stebalien/tempfile ) from 3.7.1 to 3.8.0.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md )
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.7.1...v3.8.0 )
---
updated-dependencies:
- dependency-name: tempfile
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-21 08:52:19 +00:00
4d0e55fe45
chore(deps): Bump ordered-float from 3.8.0 to 3.9.0 ( #8526 )
...
Bumps [ordered-float](https://github.com/reem/rust-ordered-float ) from 3.8.0 to 3.9.0.
- [Release notes](https://github.com/reem/rust-ordered-float/releases )
- [Commits](https://github.com/reem/rust-ordered-float/compare/v3.8.0...v3.9.0 )
---
updated-dependencies:
- dependency-name: ordered-float
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-21 08:35:00 +00:00
9b0e2cecca
Merge pull request #8524 from influxdata/cn/test-cleanups
...
refactor: Clean up some of the end-to-end CLI tests
2023-08-21 08:29:04 +00:00
8263c4f3cf
feat: implement new ingest structure
...
Define new proto for the structure that gets sent from router to ingester and persisted in the ingester WAL.
Create ingest_structure crate with functions to convert from line protocol to new proto structure while validating schema.
Add function to convert new proto structure to RecordBatch.
2023-08-20 17:31:52 -04:00
0fd5af64bc
refactor: Extract a builder of namespace commands
2023-08-18 15:59:30 -04:00
64eb3be4c2
refactor: Split long strings for readability and rustfmt
...
When there are long strings in a file, rustfmt just gives up and
ignores the whole file.
2023-08-18 15:59:06 -04:00
defd20c7ed
refactor: Reduce duplication and increase consistency in namespace names in tests
...
Create consts outside the test steps when possible to share values that
need to be the same. Call all of these namespace_name to distinguish
from a namespace object or command.
2023-08-18 15:52:38 -04:00
d575873ceb
refactor: Extract CLI test helper functions into their own module
2023-08-18 15:52:37 -04:00
412df8dc4e
fix: Remove debugging printlns from tests
2023-08-18 15:52:37 -04:00
65207b2f9c
fix: Remove unneeded use of lazy_static
...
This can just be a static slice rather than a vec.
2023-08-18 15:52:37 -04:00
3d1e49e57a
refactor: Extract table CLI tests to their own module
2023-08-18 15:52:37 -04:00
22f1d6f469
refactor: Extract namespace CLI tests to their own module
2023-08-18 15:52:37 -04:00
38c3229e35
fix: fix flaky partition tests with a retry ( #8523 )
2023-08-18 18:12:49 +00:00
df0349202f
chore(ingester): Better logs and more comments for Disk Full Protection
...
Extra details, get your extra right here!
Co-authored-by: Dom <dom@itsallbroken.com>
2023-08-18 16:25:44 +01:00
5d17a99dbb
feat: read null sort_key_ids ( #8489 )
...
* feat: read null sort_key_ids
* chore: clearer explanation about test strategy
* chore: Apply suggestions from code review
Co-authored-by: Marco Neumann <marco@crepererum.net>
* test: tests that add partition with NULL sort_key_ids
* chore: address review comments
* chore: remove unecessary comments and tests
* fix: typos
* chore: remove unecessary tests
* fix: chec duplicates for SortedColumnSet
---------
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-18 14:15:27 +00:00
0b736b05af
chore(ingester): Add log for disk full protection task exit
...
Co-authored-by: Dom <dom@itsallbroken.com>
2023-08-18 14:47:42 +01:00
6af0dfb962
Merge pull request #8412 from influxdata/cn/use-bloom-filter
...
feat: Use the bloom filter to reduce ingester catalog queries, Luke
2023-08-18 13:36:43 +00:00
1d6366d2f4
test: Ask ingesters to persist before asking catalog for their partition keys
2023-08-18 09:25:18 -04:00
Dom
dependabot[bot]
Marco Neumann
CircleCI[bot]
Joe-Blount
Andrew Lamb
Nga Tran
kodiakhq[bot]
Paul Dix
Carol (Nichols || Goulding)
Fraser Savage