d34d23c354
refactor: remove `processed_tombstone` table ( #7840 )
...
- the table is unused
- there are no foreign keys or triggers based on this table
- the design is generally not scalable (N*M entries) and tombstones
should rather have
a timestamp so we can check if a parquet file includes that
information or not (or some other form of serialization mechanism)
- it's currently empty in prod (an never was filled w/ data in any
cluster)
2023-05-22 15:56:23 +00:00
31b8813760
feat: hide `system.queries` table from prod by default ( #7810 )
...
Introduce a new header called `iox-debug` which when set enables certain
debug features. The first one will be the `system.queries` table which
is a process-local, namespace-scoped query log. In most prod setups this
is only useful for debugging and will confuse the user a lot because
when multiple queries are deployed then the K8s routing decides which
pod/process the users hits. This leads to an inconsistent view. However
the log is still useful for debugging.
This also wires the "debug header set" flag through the Flight ticket,
because JDBC proved (integration tests FTW!) that headers are only
passed to `GetFlightInfo` but not to `DoGet` and the ticket must encode
all the relevant information.
Closes #7119 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-22 12:29:24 +00:00
6f10fd2383
Merge pull request #7837 from influxdata/dom/deps
...
chore: remove unused deps
2023-05-22 11:46:14 +00:00
0719928800
chore: remove unused deps
...
The wal crate imports a bunch of stuff it never uses!
2023-05-22 13:38:49 +02:00
f3f1a5edef
chore(deps): Bump sqlparser from 0.33.0 to 0.34.0 ( #7834 )
...
* chore(deps): Bump sqlparser from 0.33.0 to 0.34.0
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs ) from 0.33.0 to 0.34.0.
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md )
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.33.0...v0.34.0 )
---
updated-dependencies:
- dependency-name: sqlparser
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
* fix: sqlparser compat
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
2023-05-22 10:16:38 +00:00
6cb7619d83
chore(deps): Bump base64 from 0.21.0 to 0.21.1 ( #7832 )
...
Bumps [base64](https://github.com/marshallpierce/rust-base64 ) from 0.21.0 to 0.21.1.
- [Changelog](https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md )
- [Commits](https://github.com/marshallpierce/rust-base64/commits )
---
updated-dependencies:
- dependency-name: base64
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-05-22 09:50:06 +00:00
63b3279e33
chore(deps): Bump digest from 0.10.6 to 0.10.7 ( #7835 )
...
Bumps [digest](https://github.com/RustCrypto/traits ) from 0.10.6 to 0.10.7.
- [Commits](https://github.com/RustCrypto/traits/compare/digest-v0.10.6...digest-v0.10.7 )
---
updated-dependencies:
- dependency-name: digest
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-22 08:48:17 +00:00
db9fe92981
refactor: consolidate pruning code ( #7815 )
...
Let's have a single chunk pruning implementation in our code, not two.
Also removes a bit of crust from `QueryChunk` since it is technically no
longer responsible for pruning (this part has been pushed into the
querier for early pruning and bits for the `iox_query_influxrpc` for
some RPC shenanigans).
2023-05-22 08:42:20 +00:00
bf847584d4
chore(deps): Bump clap from 4.2.7 to 4.3.0 ( #7833 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 4.2.7 to 4.3.0.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v4.2.7...clap_complete-v4.3.0 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-05-22 08:33:05 +00:00
e9cbe4e589
chore: Fix broken reference
2023-05-22 16:55:41 +10:00
50289ebab7
chore: Update insta snapshots
2023-05-22 16:30:55 +10:00
c9b6695a44
chore: No need to use deref
2023-05-22 16:30:38 +10:00
ed9a16c4ad
chore: Add test to validate compatibility
2023-05-22 16:23:21 +10:00
af76865b2c
feat: complete InfluxQL subquery compatibility
...
Closes #7794
2023-05-22 16:22:54 +10:00
ccd73a0b32
chore: remove duplicate code
2023-05-22 16:20:26 +10:00
0d4af814dc
Merge branch 'main' into sgc/issue/7794_subquery_inconsistency
2023-05-20 10:53:15 +10:00
c082ebac7d
chore: remove config lines to skip failed rules, as it is now default ( #7831 )
2023-05-19 22:27:57 +00:00
44ec835191
docs: formatting fix of doc comment
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2023-05-19 14:02:56 +01:00
a53e2a43f2
docs(README): Update link to datafusion ( #7821 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-19 11:14:59 +00:00
8cd96f7238
chore(deps): Bump toml from 0.7.3 to 0.7.4 ( #7827 )
...
Bumps [toml](https://github.com/toml-rs/toml ) from 0.7.3 to 0.7.4.
- [Commits](https://github.com/toml-rs/toml/compare/toml-v0.7.3...toml-v0.7.4 )
---
updated-dependencies:
- dependency-name: toml
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-05-19 09:25:49 +00:00
5bb2c8cb03
Merge pull request #7828 from influxdata/dependabot/cargo/bitflags-2.3.1
...
chore(deps): Bump bitflags from 1.3.2 to 2.3.1
2023-05-19 10:19:58 +01:00
e88d8befdb
Merge branch 'main' into sgc/issue/7794_subquery_inconsistency
2023-05-19 18:45:58 +10:00
4d069800dd
feat: Generate plan for each table and then union final plans together
...
Closes #7794
2023-05-19 18:45:35 +10:00
fdee86dc91
chore: Run cargo hakari tasks
2023-05-19 02:02:17 +00:00
1820fea85d
chore(deps): Bump bitflags from 1.3.2 to 2.3.1
...
Bumps [bitflags](https://github.com/bitflags/bitflags ) from 1.3.2 to 2.3.1.
- [Release notes](https://github.com/bitflags/bitflags/releases )
- [Changelog](https://github.com/bitflags/bitflags/blob/main/CHANGELOG.md )
- [Commits](https://github.com/bitflags/bitflags/compare/1.3.2...2.3.1 )
---
updated-dependencies:
- dependency-name: bitflags
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-05-19 02:01:02 +00:00
8bb98ab21c
chore: Improve docs
2023-05-19 09:24:33 +10:00
3fb867f1dc
feat: Display the query text when panicking.
...
Helpful for determining the cause of a failure.
2023-05-19 09:24:14 +10:00
90a25a3ff0
chore: update DataFusion ( #7825 )
...
* chore: update DataFusion
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-05-18 17:51:16 +00:00
6344fe8c3f
chore: Add rationale for `clippy::future_not_send` ( #7822 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-18 16:58:56 +00:00
e97c48b388
feat: download catalog objects as part of `remote store get-table` ( #7779 )
...
* feat: download catalog objects as part of remote store get-table
* fix: fix future_not_send
* fix: reorder import order
* chore: reduce repetition of `Error` in naming
* fix: improve the partition filtering API
* fix: clarify export comment
* fix: make index printing consistent
* fix: improve skip message
* chore: add extra clippy lints to inmport_export crate
* fix: comments
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-18 16:52:43 +00:00
c263585d94
feat(cli): Add table ID `debug wal regenerate-lp` command implementation
...
This adds a command to `influxdb_iox` that can take a WAL segment file
and regenerate all write operation entries, writing to stdout or namespaced
files within a target directory, using table ID as the measurement name
in the case where there is no catalog access at point of regeneration.
2023-05-18 17:20:41 +01:00
fd5d5e0758
fix(wal): Assert WriteOpDecoder handles tail-corrupt WAL files correctly
...
WAL read errors encountered by the new WAL WriteOpDecoder were being
discarded and presented as a "happy path" end of file to callers due
to a bug in handling a nested result type. This moves the test for
decoding a tail-corrupt WAL into the crate itself and asserts the
error is reported correctly.
2023-05-18 17:18:19 +01:00
607bc7243d
Merge pull request #7823 from influxdata/dom/replication-help
...
refactor(cli): update replication help text
2023-05-18 14:10:35 +00:00
82500720e4
refactor(cli): update replication help text
...
The replication flag defines the total number of copies of each write -
slightly less confusing than the additional copies it was previously,
and matches with the actual code.
2023-05-18 16:01:12 +02:00
042a6a66d5
refactor: remove old tsm import code ( #7804 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-18 10:51:24 +00:00
b2c9592581
chore(deps): Bump bitflags from 1.3.2 to 2.3.1 ( #7818 )
...
* chore(deps): Bump bitflags from 1.3.2 to 2.3.1
Bumps [bitflags](https://github.com/bitflags/bitflags ) from 1.3.2 to 2.3.1.
- [Release notes](https://github.com/bitflags/bitflags/releases )
- [Changelog](https://github.com/bitflags/bitflags/blob/main/CHANGELOG.md )
- [Commits](https://github.com/bitflags/bitflags/compare/1.3.2...2.3.1 )
---
updated-dependencies:
- dependency-name: bitflags
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-18 09:35:15 +00:00
43a1f7c0b2
chore(deps): Bump rustls from 0.21.0 to 0.21.1 ( #7819 )
...
Bumps [rustls](https://github.com/rustls/rustls ) from 0.21.0 to 0.21.1.
- [Changelog](https://github.com/rustls/rustls/blob/main/RELEASE_NOTES.md )
- [Commits](https://github.com/rustls/rustls/compare/v/0.21.0...v/0.21.1 )
---
updated-dependencies:
- dependency-name: rustls
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-05-18 09:29:26 +00:00
97db01dbc6
Merge pull request #7817 from influxdata/cn/flaky-test
...
test: Add an invalid test configuration checker and fix potentially flaky test setups it detects
2023-05-18 09:04:42 +00:00
04617a4362
Merge branch 'main' into sgc/issue/7794_subquery_inconsistency
2023-05-18 07:24:00 +10:00
9cc2169ce2
fix: Rename Step::WaitForPersisted2 to Step::WaitForPersisted
2023-05-17 17:02:59 -04:00
6785dcfd37
fix: Correct invalid test setups that the detector now detects
2023-05-17 17:00:17 -04:00
45e47af974
test: Add an invalid test configuration checker
...
If the test setup calls `Step::Persist` to persist on-demand, that
means it shouldn't be used with `ChunkStage::Parquet`, which tries to
persist as fast as possible. This will fail the test with a hopefully
helpful message to prevent this.
2023-05-17 16:58:50 -04:00
3ec40ca5c7
Merge pull request #7800 from influxdata/dom/partition-template-rpc
...
feat(proto): partition template type
2023-05-17 15:35:40 +01:00
5fbf2d3d69
Merge branch 'main' into dom/partition-template-rpc
2023-05-17 15:06:17 +01:00
62fed73bcd
refactor: upgrade DataFusion to `19b03240920ad63cac916b42951754c0337bdac8#19b03240920ad63cac916b42951754c0337bdac8` ( #7813 )
...
I need:
- https://github.com/apache/arrow-datafusion/pull/6226 .
Changes in code due to:
- https://github.com/apache/arrow-datafusion/pull/6332
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-17 13:57:12 +00:00
1ff11d0856
refactor: Change catalog configuration so it is entirely dsn based / support end to end testing without postgres ( #7736 )
...
* refactor: Change catalog configuration so it is entirely dsn based / support end to end testing without postgres
Restores code from https://github.com/influxdata/influxdb_iox/pull/7708
Revert "revert: PR #7708"
This reverts commit c9cfe05f8d
2023-05-17 13:36:25 +00:00
63de1a3bc8
refactor(proto): use "tag" instead of "column"
...
I was going back and forth on this, but the MVP is tags only. If we
expand it to be the more general "columns" in the future, we can change
the proto to reflect the more generalised implementation and have a more
descriptive field name now!
2023-05-17 14:03:31 +02:00
442cd1457a
Merge pull request #7811 from influxdata/savage/wal-inspect-without-table-name-index
...
feat(wal_inspect): Make table batch writing a trait, support table ID based line protocol
2023-05-17 10:45:19 +00:00
21d6460453
refactor(wal_inspect): Put the ref inside the option
2023-05-17 11:37:49 +01:00
a56eb6efa6
refactor(wal_inspect): Take generic iterator rather than map as table_batch parameter
...
This allows for callers to use whatever collection implementation they
need, so we can ordered iterate in tests.
2023-05-17 11:31:05 +01:00
Marco Neumann
kodiakhq[bot]
Dom Dwyer
dependabot[bot]
Stuart Carnie
Christopher M. Wolff
Fraser Savage
Andrew Lamb
CircleCI[bot]
Carol (Nichols || Goulding)