Commit Graph

532 Commits (41d93aea4d8e407b772e3e935f1e9e3ddb4dbe88)

Author SHA1 Message Date
kodiakhq[bot] 18405a5919
Merge branch 'main' into cn/shard-tests 2023-03-22 13:36:27 +00:00
Andrew Lamb 779234eb20
chore(flightsql): Add tests that schema and data matches (#7280)
* fix(flightsql): Tests for matching schema in FligthSQL

* fix: Update influxdb_iox/tests/end_to_end_cases/flightsql.rs
2023-03-21 15:40:21 +00:00
Carol (Nichols || Goulding) 635b22b5a9
fix: Explicitly use memory catalog for invalid compactor CLI tests
To avoid the command picking up the catalog from your `.env`.
2023-03-20 16:40:46 -04:00
Carol (Nichols || Goulding) d8bd185fa0
test: Add compactor shard e2e tests 2023-03-20 15:52:47 -04:00
Andrew Lamb 184565b552
feat(flightsql): Implement FlightSQL `GetSqlInfo` endpoint (#7198)
* feat(flightsql): Implement GetSqlInfo endpoint

* chore: Add some comments to clarify the tests intent
2023-03-20 19:34:18 +00:00
Andrew Lamb 795a79a7ad
feat(flightsql): Add `GetTables` metadata endpoint without filters (#7238)
* test: Add getTables jdbc_client example

* feat: add `CommandGetTables` in FlightSqlClient

* feat: add `CommandGetTables` in flightsql cmd and planner

* test: add e2e test for `CommandGetTables`

* chore: clippy

* chore: comment out the test with filters

* test: update jdbc test expected value for tables

---------

Co-authored-by: Chunchun <14298407+appletreeisyellow@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-20 19:20:13 +00:00
Carol (Nichols || Goulding) 528d83e989
test: Re-enable running compactor once in step tests
Putting back the test that I removed in
<https://github.com/influxdata/influxdb_iox/pull/6850>

now that they can use `run compactor2 --compaction-process-once
--compaction-process-all-partitions`

thanks to <https://github.com/influxdata/influxdb_iox/pull/6886>!
2023-03-20 13:20:43 -04:00
Andrew Lamb 2bc495b29e
feat(flightsql): Support `GetTableTypes` metadata API (#7242)
* feat(flightsql): Support `GetTableTypes` metadata API

* chore: comment about possible optimization

* chore: fix logical conflict
2023-03-17 13:12:23 +00:00
Andrew Lamb 3f0073807f
chore(flightsql): `insta`ize flightsql end to end tests (#7236)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-16 14:41:49 +00:00
Andrew Lamb 7dfaa05e8a
chore: Update datafusion again (#7208)
* chore: update datafusion again

* fix: update test

* fix: use table_reference

* fix: clean up import

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-16 14:34:40 +00:00
Marco Neumann d2874f5c02
test: fix chunk boundaries for retention query tests (#7235)
See code comment, found while working on #6098.
2023-03-16 13:14:08 +00:00
Andrew Lamb c819d4f865
feat(flightsql): Implement `GetDbSchemas` metadata API (#7213)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-16 10:13:51 +00:00
Andrew Lamb 6d6fd8f663
feat(flightsql): implement basic `CommandGetCatalogs` support (#7212)
* refactor: reduce redundancy in test

* chore: implement basic get_catalog support

* fix: clippy
2023-03-15 21:52:59 +00:00
Stuart Carnie 2b74f07fe5
feat: Support `GROUP BY` with tags in raw `SELECT` queries (#7109)
* chore: Normalise name of Call expression to lowercase

Simplifies matching functions in planner, as they are guaranteed to be
lowercase.

This also ensures compatibility with InfluxQL when generating column
alias names, which are reflected in updated tests.

* chore: Ensure aggregate functions fail gracefully.

* feat: GROUP BY tag support

* feat: Ensure schema-level metadata is propagated

Requires: https://github.com/apache/arrow-rs/issues/3779

* chore: Add some tests to validate GROUP BY output

* chore: Add clarifying comment

* chore: Declare message in flight.proto

The metadata is public API, so best practice is to encode this in a way
that is most compatible for clients in other languages, and will also
document the history of schema changes.

Added tests to validate the metadata is encoded correctly.

* chore: Placate linters

* chore: Use correct column in test cases

* chore: Add `is_projected` to the TagKeyColumn message

`is_projected` is necessary to inform a client whether it should include
the tag key is used exclusively for the group key (false) or also
projected in the `SELECT` column list.

* refactor: Move constants to `schema` crate per PR feedback

* chore: rustfmt 🙄

* chore: Update docs for InfluxQlMetadata

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

---------

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2023-03-07 22:40:23 +00:00
Andrew Lamb ed0704ac8d
chore: Update datafusion (#7100)
* chore: Update datafusion

* chore: iox_query to compile for API changes + update tests

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-03-06 17:59:24 +00:00
Andrew Lamb e19ce98407
chore: Update datafusion + arrow/arrow-flight/parquet to 34.0.0 (#7084)
* chore: Update datafusion + arrow/arrow-flight/parquet to 34.0.0

* chore: Run cargo hakari tasks

* chore: Update plans

* chore: Update querier expected output

* chore: Update querier tests to use insta

* fix: sort output too

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-01 11:25:01 +00:00
Stuart Carnie 2ed5758ddb
feat: InfluxQL planner learns how to project multiple measurements (#7063)
* feat: Planner learns how to project multiple measurements

Closes #6896

* chore: Update test

* chore: PR feedback
2023-02-27 00:50:06 +00:00
Christopher M. Wolff 0282eb4750
feat: streaming implementation of gap filling (#7037)
* feat: streaming implementation of gap filling

* chore: cargo fmt

* refactor: when gapfilling, concatenate input batches less frequently

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-23 15:42:41 +00:00
Andrew Lamb f93baf7693
chore: Update DataFusion and `arrow` / `arrow-flight` / `parquet` to `33.0.0` (#7045)
* chore: Update DataFusion and arrow/arrow-flight/parquet to 33.0.0

* fix: Update test output

* fix: update more test output

* fix: Update querier test output

* chore: Run cargo hakari tasks

* test: fix formatting

Fix formatting of batch pretty printing.

* test: fix formatting

Fix formatting of batch pretty printing.

* test: fix formatting for selector tests

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: Christopher Wolff <chris.wolff@influxdata.com>
2023-02-22 21:24:20 +00:00
kodiakhq[bot] 48e6cce746
Merge branch 'main' into cn/less-shard-id 2023-02-22 15:05:37 +00:00
Stuart Carnie 929ac9081e
feat: Rewrite logical expression to match InfluxQL behaviour (#7031)
* chore: Move to inline snapshots

* chore: Container for the DataFusion and IOx schema

* chore: Simplify using logical expression helper functions

* feat: Rewrite conditional expressions using InfluxQL rules

* feat: Add tests to validation conditional expression rewriting

* feat: Rewrite column expressions

* chore: Rewrite expression to use false when possible

This allows the planner to optimise away the entire logical plan to an
empty plan in many cases.

* feat: Complete cast postfix operator support

Added `unsigned` postfix operator, as the feature was mostly complete.

Closes #6895

* chore: Remove redundant attribute
2023-02-21 20:01:31 +00:00
Carol (Nichols || Goulding) 65ba208f88
fix: Remove shard_id from the Parquet File protobuf in the catalog service 2023-02-17 13:53:03 -05:00
Andrew Lamb 27890b313f
chore: Update datafusion (#6997)
* chore: Update datafusion

* chore: update the plans

* fix: update some plans

* chore: Update plans and port some explain plans to use insta snapshots

* fix: another plan

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-16 17:03:25 +00:00
Carol (Nichols || Goulding) b11228c72e
test: Write an e2e test of namespace soft-deletion (#7002)
Co-authored-by: Dom <dom@itsallbroken.com>
2023-02-16 09:59:14 +00:00
Carol (Nichols || Goulding) 9e27709c84
test: Add an e2e test of just the ingester's API 2023-02-13 16:36:59 -05:00
Andrew Lamb 2751c77de6
chore: add extra debugging to help track down jdbc_client failures (#6968)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-13 12:14:39 +00:00
Andrew Lamb 143cb26e87
chore: Update datafusion again (#6958)
* chore: Update datafusion pin

* chore: Update plans

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-13 10:28:44 +00:00
Christopher M. Wolff a2510c8343
feat: partial implementation of gap filling operator (#6911)
* feat: partial implementation of gap filling operator

* chore: code review feedback

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-10 19:03:34 +00:00
Carol (Nichols || Goulding) 7b80b03b28
test: Test with multiple ingesters, like in production 2023-02-09 17:31:28 -05:00
Carol (Nichols || Goulding) 1794147b3c
fix: Only test querier panic logging; router2 and ingester2 now exit on panic 2023-02-09 13:56:07 -05:00
Carol (Nichols || Goulding) b5464aa82e
fix: Remove test that compacts once for now 2023-02-09 13:56:07 -05:00
Carol (Nichols || Goulding) 38f8e18357
fix: Only run end-to-end tests against Kafkaless 2023-02-09 13:56:06 -05:00
Carol (Nichols || Goulding) d7c59da46b
fix: Switch more tests over to the kafkaless architecture 2023-02-09 13:56:06 -05:00
Stuart Carnie a2945a77a4
fix: Implement EXPLAIN, ORDER BY and default ordering (#6864)
* chore: Add more tests

* chore: Fix default ordering; implement ORDER BY

* feat: Add EXPLAIN support

* chore: Add additional tests to validate GROUP BY expansion

* chore: More test cases for TZ, and failing log scalar function
2023-02-07 22:18:52 +00:00
Carol (Nichols || Goulding) f1f475d552
fix: Remove compactor generate (#6872)
This debugging tool was more useful in previous situations where it was
harder to get real data as input for the compactor.

It's currently causing a flaky test that isn't worth investigating.

Fixes #6190 by making it moot.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-06 19:19:15 +00:00
Dom Dwyer b8ec022ea6
test(e2e): HTTP error response
Ensure a HTTP error response contains a well-formed JSON structure
containing "code" and "message" fields (for backwards compatibility with
existing InfluxDB versions) and a correct "content-type" header.
2023-02-06 17:10:30 +01:00
Andrew Lamb ae10b33f0c
chore: Update datafusion (#6868)
* chore: Update datafusion

* chore: Run cargo hakari tasks

* fix: Update error message

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-02-06 14:22:45 +00:00
Raphael Taylor-Davies d3601a59f8
chore: update DataFusion, upgrade `arrow` `arrow-flight` and `parquet` to `32.0.0` (#6756)
* chore: update DataFusion

* fix: test

* chore: format

* chore: clippy

* chore: update arrow

* chore: arrow upgrade fallout

* chore: Run cargo hakari tasks

* chore: remove failing warm compaction test

* fix: flight error propagation

* chore: update parquet size

* fix: Update error message

* chore: Update parquet metadata test

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-06 11:35:39 +00:00
Carol (Nichols || Goulding) ca8558e636
chore: Unnest test code out of the module and clean up long lines 2023-02-03 14:20:30 -05:00
Carol (Nichols || Goulding) b17d1c0862
test: Port more query tests to kafkaless 2023-02-03 13:56:01 -05:00
Carol (Nichols || Goulding) 30fea67701
fix: Move variables within format strings. Thanks clippy!
Changes made automatically using `cargo clippy --fix`.
2023-02-03 13:06:17 -05:00
Carol (Nichols || Goulding) b55e4ffa21
test: Port table_schema tests to end-to-end and get rid of query_tests (#6828)
* test: Port table_schema query_tests to end-to-end

* fix: Remove now outdated query_tests crate

---------

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-03 17:41:08 +00:00
Andrew Lamb 673d1019f0
fix: Normalize for flaky tests (#6845) 2023-02-03 17:20:11 +00:00
Andrew Lamb 0d01306fd2
chore: port remote end to end tests to new write path (#6838)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-03 15:33:49 +00:00
Andrew Lamb b9ab09e944
chore: remove kafka full querier end to end tests (#6834)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-03 15:25:24 +00:00
Andrew Lamb ef98b8c348
chore: Port `influxrpc` end to end tests to new writepath (#6839)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-03 14:58:18 +00:00
Andrew Lamb 05ca71f91f
chore: Remove old ingester end to end tests (#6837)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-03 14:43:27 +00:00
Andrew Lamb 755d3586ea
chore: port cli end to end tests to kafkaless write path (#6836)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-03 14:35:05 +00:00
Andrew Lamb 012c35bf3e
chore: Port flightsql end to end tests for new kafkaless writepath (#6835)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-03 14:26:26 +00:00
Andrew Lamb acc258e9aa
chore: Update influxql end to end tests to kafkaless write path (#6833)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-03 14:11:55 +00:00
Stuart Carnie 7ae453fa75
feat: Add support for data-driven InfluxQL tests (#6830)
* feat: Add support for data-driven InfluxQL tests

* chore: Added more tests

* chore: Check if GIT_HASH has changed, to avoid rebuilds

This speeds up the edit-test cycle, when changing data files, as
cargo won't rebuild the test binary 🥳

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2023-02-03 12:56:37 +00:00
Stuart Carnie 63d0a77daf
feat: Updating to new services for all-in-one (#6811)
* feat: Updating to new services for all-in-one

* fix: Use correct shard id for ingester2

* fix: clippy

* fix: use wal directory

* fix: end to end tests

* fix: Update tracing cases for new ingest reality

* fix: update metrics test

* fix: Use rpc mode

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2023-02-02 20:42:29 +00:00
Carol (Nichols || Goulding) c9b6e269d7
test: Port remaining query_tests/src/sql.rs tests to end-to-end (#6807)
* fix: Reading file error reported the wrong path

When the `.expected` SQL file couldn't be found, this error reported
the input file path instead.

* test: Port SQL query_tests to end-to-end tests

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-02 15:35:15 +00:00
Carol (Nichols || Goulding) 7590fa5aaa
fix: Change test data and predicates to get better coverage
These two tests weren't actually ensuring that the combination of these
predicates worked, because the tests would still pass if some of the
predicate parts were removed.
2023-02-01 11:09:10 -05:00
Carol (Nichols || Goulding) 072e228d35
test: Port tag_values query_tests to end-to-end tests 2023-02-01 10:26:03 -05:00
Carol (Nichols || Goulding) 629867c6ac
refactor: Start a new file for tag values tests; move the one existing test 2023-02-01 10:25:48 -05:00
Andrew Lamb 80f0125940
feat: Add number of rows to explain of RecordBatchesExec (#6781)
* feat: Add number of rows to explain of RecordBatchesExec

* fix: Update test output
2023-01-31 14:26:20 +00:00
Carol (Nichols || Goulding) ff32a042b0
test: Port tag_keys query_tests to end-to-end tests (#6760)
* refactor: Start a new file for tag keys tests; move the one existing test

* test: Port tag_keys query_tests to end-to-end tests

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2023-01-31 14:05:22 +00:00
Carol (Nichols || Goulding) d18d2c34e4
test: Port measurement_names/table_names query_tests to end-to-end tests (#6757)
* refactor: Start a new file for measurement names tests; move the one existing test

* fix: Pass on predicate when sending a measurement names request with GrpcRequestBuilder

* feat: Support literal integer queries too

* test: Port measurement_names/table_names query_tests to end-to-end tests

* fix: merge conflict error

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-31 12:55:30 +00:00
Andrew Lamb e8e50df692
chore: Enable last SQL test, retention.sql (#6721) 2023-01-31 12:46:50 +00:00
Carol (Nichols || Goulding) cff422b795
test: Port read_window_aggregate query_tests to end-to-end tests (#6755)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-31 12:15:50 +00:00
Andrew Lamb 5b14caa780
chore: Update DataFusion (#6753)
* chore: Update datafusion

* fix: Update for changes

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-30 14:48:52 +00:00
Andrew Lamb 51e324378c
chore: Add test for `SHOW TABLES` (#6754)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-30 14:05:26 +00:00
Carol (Nichols || Goulding) 2254ee06b7
test: Port read_group query_tests to end-to-end tests (#6743)
* fix: Delete the read filter *file*; last PR only deleted the *contents*

* test: Port read_group query_tests to end-to-end tests

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-30 12:10:39 +00:00
Christopher M. Wolff 55257b46c9
chore: validate ingester URIs on querier CLI (#6740)
* chore: add validate for ingesters on querier CLI

* chore: fix typo and tests

* chore: clippy

* chore: review feedback

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-27 21:13:52 +00:00
Andrew Lamb 4a6b4f78d2
refactor: Make some nicer methods on the GrpcRequestBuilder (#6741)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-27 21:06:22 +00:00
Carol (Nichols || Goulding) 11233e3b3b
test: Translate the filter_on_value test using field ref (#6734) 2023-01-27 20:03:44 +00:00
Carol (Nichols || Goulding) 4f8dd072b3
fix: Translate a test with a predicate of a literal = literal 2023-01-27 10:28:43 -05:00
Carol (Nichols || Goulding) 94f7f015f4
fix: Port a test with a predicate that tag=tag, which is always true 2023-01-27 10:28:43 -05:00
Carol (Nichols || Goulding) a2b67abe54
fix: Remove test cases that aren't valid to port to end-to-end tests 2023-01-27 10:28:43 -05:00
Carol (Nichols || Goulding) 67c430da63
test: Port read_filter query_tests to end-to-end tests 2023-01-27 10:28:43 -05:00
Andrew Lamb 5ef9018f7e
refactor: Move sql script files from query_tests and into end to end query tests (#6708)
* refactor: Move sql script files from query_tests and into end to end query tests

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2023-01-26 19:49:21 +00:00
Andrew Lamb 589fbbf11c
chore: remove unecessary checks for persisted in end to end tests (#6713)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-26 18:24:39 +00:00
Andrew Lamb c100737a81 chore: Do not send dictionary encoded data to clients 2023-01-26 06:35:15 -05:00
Nga Tran b8a80869d4
feat: introduce a new way of max_sequence_number for ingester, compactor and querier (#6692)
* feat: introduce a new way of max_sequence_number for ingester, compactor and querier

* chore: cleanup

* feat: new column max_l0_created_at to order files for deduplication

* chore: cleanup

* chore: debug info for chnaging cpu.parquet

* fix: update test parquet file

Co-authored-by: Marco Neumann <marco@crepererum.net>
2023-01-26 10:52:47 +00:00
Andrew Lamb 7853a19953
feat: JDBC integration tests with FlightSQL (#6693)
* feat: basic JDBC integration test

* fix: do not run test without env set

* docs: add maven link

* refactor: clean up java with switch statement
2023-01-25 22:21:18 +00:00
Carol (Nichols || Goulding) 57b5b639d6
test: Port all field columns query_tests to end-to-end tests (#6707)
* test: Port a test that's not actually supported through the full gRPC API

* test: Port remaining field column/measurement fields tests

* test: Remove unsupported measurement predicate and clarify purposes of tests

Andrew confirmed that the only way to invoke a Measurement Fields
request is with a measurement/table name specified: <0249b5018e/generated_types/protos/influxdata/platform/storage/service.proto (L43)>

so testing with a `_measurement` predicate is not valid.

I thought this test would become redundant with some other tests, but
they're actually still different enough; I took this opportunity to
better highlight the differences in the test names.

* refactor: Move all measurement fields tests to their own file

* test: Remove field columns tests that are now covered in end-to-end measurement fields tests
2023-01-25 19:49:29 +00:00
Carol (Nichols || Goulding) 4658510102
fix: For Ingester2, persist a particular namespace on demand and share MiniClusters
This should hopefully help CI from running out of Postgres
connections 😬

The old architecture will still need to be non-shared and persist
everything.
2023-01-25 10:36:56 -05:00
Carol (Nichols || Goulding) f310e01b1a
test: Start of porting InfluxRpc query_tests
Make a new trait, `InfluxRpcTest`, that types can implement to define
how to run a test on a specific Storage gRPC API. `InfluxRpcTest` takes
care of iterating through the two architectures, running the setups, and
creating the custom test step.

Implementers of the trait can define aspects of the tests that differ
per run, to make the parameters of the test clearer and highlight what
different tests are testing.
2023-01-25 10:27:42 -05:00
Andrew Lamb 0c55a0f257
feat: Implement basic prepared statement support in IOx (#6667)
* feat: allow override of flightsql namespace

* feat: Implement DoAction endpoint

* refactor: Remove try_unpack

* fix: remove unused code / more clone
2023-01-25 12:00:43 +00:00
Andrew Lamb 1b882e0062
fix: `error arrow/ipc: could not read message schema: EOF` (#6668)
* chore: Test for schema from query

* fix: Send schema even for no RecordBatches

* fix: docs
2023-01-23 22:23:34 +00:00
Carol (Nichols || Goulding) 6afd782b3f
fix: Move query_tests2 into influxdb_iox/tests so that the code rebuilds 2023-01-19 16:44:31 -05:00
Andrew Lamb 65c020c9f2
refactor: remove iox_arrow_flight use in `influxdb_iox_client ` and `querier` (#6624)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-19 18:48:23 +00:00
Carol (Nichols || Goulding) f3b5dcaab7
feat: Reimagining query_tests 2023-01-18 10:24:17 -05:00
Dom f7ff877582
Merge branch 'main' into cn/ingester-persist-tick 2023-01-13 12:31:45 +00:00
Carol (Nichols || Goulding) f56123bf30
test: Allow integration tests that should_panic to pass if TEST_INTEGRATION isn't set 2023-01-12 15:31:34 -05:00
Carol (Nichols || Goulding) 1c7ffb95df
test: Write a should_panic test that shows ingester is persisting when I thought it wouldn't 2023-01-12 14:55:28 -05:00
Carol (Nichols || Goulding) b989e0893f
test: Make persist-on-demand test a step test and check the number of parquet files 2023-01-12 11:40:46 -05:00
Carol (Nichols || Goulding) 3a2544a7eb
feat: Define a new gRPC service for ingester persist 2023-01-12 11:03:12 -05:00
Carol (Nichols || Goulding) e9b3efb33d
refactor: Extract a method for making requests to the ingester onto MiniCluster 2023-01-12 11:03:10 -05:00
Dom 1e5b594863
Merge branch 'main' into dom/shutdown-persist 2023-01-12 10:15:34 +00:00
Carol (Nichols || Goulding) be7c312033
fix: Wait for a particular number of Parquet files, not just any change 2023-01-11 12:11:56 -05:00
Carol (Nichols || Goulding) 7e921e6a23
fix: Make recording num parquet files an explicit test step
To support a case where someone calls WriteLineProtocol twice in
a row to simulate two write requests. The test should be able to
record this state before the two write requests and not twice.
2023-01-11 11:51:56 -05:00
Carol (Nichols || Goulding) 6677ae5c61
test: Record number of Parquet files before a write
Fixes #6506.

Also has the pleasant side effect of making this code simpler and less
hacky-- it now checks the number of Parquet files for the whole
namespace, which is useful in cases where the line protocol writes to
several tables.
2023-01-11 11:51:55 -05:00
Dom Dwyer 303c9e4398
test: fix e2e test
This test was relying on a graceful shutdown of the ingester to drive a
WAL replay, restoring the buffered state at startup.

Now the shutdown causes the data to be persisted and not replayed, this
didn't work.
2023-01-11 17:15:04 +01:00
Stuart Carnie 66047f4372
feat: InfluxQL learns how to plan some InfluxQL queries (#6520)
* feat: InfluxQL learns how to plan some queries

Also added a means to test the planner and execution

* chore: Update module docs

* chore: Document the planner functions

* chore: Update end_to_end_cases crate

* chore: Clarify why `SLIMIT` and `SOFFSET` return `NotImplemented`

* chore: Address lint issues

* chore: Fix rustdoc link issue

* chore: Remove InfluxQL tests from query_tests crate

Will follow conventions established by @carols10cents when
new query_tests crate is merged.

* chore: `now` field

`now` is a DataFusion built-in scalar function

* chore: remove unused code

* chore: Add additional arithmetic expression tests

* chore: Establish pattern for identifying and tracking InfluxQL issues

* chore: Add tests for case sensitivity issues

* chore: group tests into modules and functions

This avoids mass rewriting of insta snapshots as new
tests are added to each function. When tests are added in the middle,
existing snapshots are renamed (-N+1, -N+2, etc) resulting in
having to review numerous additional snapshots.
2023-01-11 02:50:49 +00:00
Carol (Nichols || Goulding) afaaedc758
test: Reproduce the test for querier caching 2023-01-04 10:15:35 -05:00
Carol (Nichols || Goulding) 9f7dede433
refactor: Reorganize tests to make deleting half the tests easier
When we switch to RPC write mode, the with_kafka module can be deleted
and the kafkaless_rpc_write module can be unwrapped.
2023-01-04 10:15:35 -05:00
Carol (Nichols || Goulding) c464487bb2
test: Ability to set up a querier2 that doesn't connect to any ingesters 2023-01-04 10:06:57 -05:00
Carol (Nichols || Goulding) 08ceb4ee48
test: Check catalog for new Parquet files to know when data is persisted 2023-01-04 10:06:57 -05:00
Carol (Nichols || Goulding) e49bee0c26
test: Make test ingester2 instances either persist very quickly or not at all 2023-01-04 10:06:57 -05:00
Carol (Nichols || Goulding) 96029654ab
test: Add a shared MiniCluster for version 2 services 2023-01-04 10:06:56 -05:00
Andrew Lamb dbe52f1ca1
chore: Upgrade datafusion (#6467)
* chore: Update datafusion

* fix: Update for new apis

* chore: Update expected plan

* fix: Update for new config construction

* chore: update clippy

* fix: Fix error codes

* fix: update another test

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-03 15:29:11 +00:00
Nga Tran d27e137c39
chore: add debug info for the investigation (#6472) 2022-12-29 23:49:29 +00:00
Carol (Nichols || Goulding) 7c6ccdb6d7
fix: Use keys and values functions. Thanks clippy! 2022-12-21 14:32:35 -05:00
Andrew Lamb e1059a9009
feat: FlightSQL Milestone 2 basic FlightSQL client and FlightSQL server implementation and plumbing (#6398)
* feat: Add basic Flight and FlightSQL client into IOx codebase

Basic flight end to end test

* fix: Apply suggestions from code review

Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>

Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-20 17:34:00 +00:00
Andrew Lamb 9b22ede3f0
refactor: Make arrow flight client return `futures::Streams` (#6438)
* refactor: Make arrow flight client use futures::Streams

* refactor: concision
2022-12-19 17:09:26 +00:00
kodiakhq[bot] c0f2ba09ee
Merge branch 'main' into cn/compactor2 2022-12-19 14:22:56 +00:00
Andrew Lamb 94c2f94ea1
refactor: Extract common ArrowFlight client into iox_arrow_flight (#6427)
* refactor: Extract common ArrowFlight client into iox_arrow_flight

* chore: Run cargo hakari tasks

* fix: clarify intent of iox_arrow_flight crate

* refactor: Apply suggestions from code review

Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>

* fix: loop --> while let

* fix: REmove make_tonic_error in favor of From impl

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-19 11:35:20 +00:00
Carol (Nichols || Goulding) d7e75d43ea
fix: Make shard ID optional for compactor queries in RPC write mode 2022-12-16 17:28:53 -05:00
Carol (Nichols || Goulding) 2406cdb24b
feat: Create a compactor2 cli 2022-12-16 17:22:06 -05:00
Carol (Nichols || Goulding) 2a1e540ee3
fix: Rename INFLUXDB_IOX_MODE to INFLUXDB_IOX_RPC_MODE 2022-12-15 14:13:01 -05:00
Carol (Nichols || Goulding) 7d216ba1fd
feat: Error if you run the wrong command with the wrong env var set
Connects to #6402.
2022-12-15 14:06:59 -05:00
Carol (Nichols || Goulding) aec98015d7
fix: Remove the rpc_write feature flag and use INFLUXDB_IOX_MODE env var instead
And standardize on ingester2 and router2 for consistency.

Connects to #6402.
2022-12-15 14:06:59 -05:00
Carol (Nichols || Goulding) 1c7f322a4e
feat: Keep track of and report number of Parquet files persisted
Per partition and starting over each time the ingester restarts.

Fixes #6334.
2022-12-12 11:45:00 -05:00
Carol (Nichols || Goulding) c3a7575d46
feat: Enable rpc_write on the inner command if it's enabled for tests
And only run rpc_write specific tests if the feature is enabled when
running the tests.
2022-12-08 17:45:30 -05:00
Carol (Nichols || Goulding) 2fd2d05ef6
feat: Identify each run of an ingester with a Uuid
And send that UUID in the Flight response for queries to that ingester
run.

Fixes #6333.
2022-12-08 17:22:52 -05:00
Marco Neumann 4ded68de62
test: "not found" end2end tests for querier (#6352)
I couldn't find any end2end tests for these cases and I was kinda
worried that our error codes were wrong. Turns out they are correct, but
let's have some nice tests for this behavior.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-08 18:17:53 +00:00
Marco Neumann f62b270852
fix: gRPC errors regarding group cols (#6314)
* fix: gRPC errors regarding group cols

- missing group col prev. produced an "internal error" but should be
  "invalid argument"
- duplicate group cols produced a panic but should also be "invalid
  argument"

* docs: clarify
2022-12-06 07:36:32 +00:00
Marco Neumann cd6a8a1a82
refactor: DF-driven on-demand mem limit instead of ahead-of-time heuristics (#6313)
* refactor: DF-driven on-demand mem limit instead of ahead-of-time heuristics

Closes #6310.

* refactor: rename and tune default exec mem limits

* fix: ingester2 bits after rebase
2022-12-05 12:38:28 +00:00
Stuart Carnie 2306c383f3
feat: Introduce InfluxQL to Flight (#6166)
* feat: Introduce InfluxQL to Flight

All InfluxQL queries will fail with an error

* chore: Temper protobuf lint

* chore: Finalize flight.proto changes; fix tests

* chore: Add tests for InfluxQL planner

* chore: Update docs

* chore: Update docs

* chore: Rename back to original

* chore: Use .into() rather than cast

* chore: Use function rather than field

* chore: Improved InfluxQL planner name

* chore: Restore `impl Into<String>` argument

* chore: Add a comment that Go clients are unable to execute InfluxQL

* chore: Add a test for the `--lang` argument and InfluxQL
2022-11-23 00:33:49 +00:00
Marco Neumann 67af105bde
feat: add `process_start_time_seconds` metric (#6187)
This is helpful to reason about IOx versions and changes/alerts during
deployments.
2022-11-21 12:27:04 +00:00
Nga Tran 49a9565240
feat: gRPC that creates namespace (#6103)
* feat: create namespace API call in router

Co-authored-by: Nga Tran <nga-tran@live.com>

* chore: treat retention as ns except in CLI

* fix: overflow in nanosecond calc

* fix: retention test after changing it from hours to ns

* chore: comment clarification in cli; better response type for error in ns API

* fix: correct some rebase mistakes

* chore: merge namespace create & create_with_retention; renamed ns create test helper fn & const

* fix: ns autocreation test was wrong after rebase

* fix: mem catalog has default 1hr retention, accidently removed in rebase

* chore: remove mem catalogs default 1hr retention; make it settable in sets & router

Co-authored-by: Luke Bond <luke.n.bond@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-18 13:02:12 +00:00
Nga Tran 6f7b1e2e26
feat: reject writes that are outside the retention period (#6148)
* feat: reject writes that are outside the retention period

* feat: add retention validator into handler stack

* chore: Apply suggestions from code review

Co-authored-by: Dom <dom@itsallbroken.com>

* refactor: address review comments

* test: unit tests fot retention validation

* chore: address review comments

* test: more unit tests and integration tests

* refactor: make time inside retention period for emphemeral_mode test

* fix: 2 hours

Co-authored-by: Dom <dom@itsallbroken.com>
2022-11-17 20:55:58 +00:00
Marco Neumann 62851afc27
feat: add querier->ingester circuit breaker (#6147)
* feat: add log ingester memory pressure persist

* feat: add querier->ingester circuit breaker

Closes #4608.

* docs: explain high-level circuit breaker

* docs: improve

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

* test: add additional test assertion

* refactor: upgrade info to warning log

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2022-11-16 10:50:33 +00:00
Nga Tran 9c4266c503
refactor: first step to remove unused retention_duration (#6113)
* refactor: first step to remove unused retention_duration

* refactor: remove retenion_duration from update catalog

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-11 15:21:06 +00:00
Carol (Nichols || Goulding) 43687a86d2
fix: Remove lots of needless borrows that Clippy can now identify
Except for in generated code that we don't control.
2022-11-09 10:54:18 -05:00
Dom d9c97795fc
feat: use IDs in ingester query API (#6093)
* refactor: NS+table ID (instead of name) in querier<>ingester

* feat(ingester): use IDs for query API

Changes the ingester to utilise the ID fields (instead of names) sent
over the query wire message wrapped within the Flight API.

BREAKING: this changes the "query-ingester" CLI command arguments which
now expects the namespace & table IDs, rather than their names.

* refactor(ingester): add more query logging context

Updates the log messages during query execution to include more context
fields.

* style: remove unused import

Co-authored-by: Marco Neumann <marco@crepererum.net>
2022-11-09 11:25:13 +00:00
Marco Neumann 903f7bafa7
refactor: expose `ParquetExec` directly to DataFusion phys. plan (#6072)
* refactor: expose `ParquetExec` directly to DataFusion phys. plan

Closes #5897.

* fix: update tracing tests

* refactor: use `EmptyExec`

* refactor: use `target_partitions`

* refactor: improve UUID normalization in query tests

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-11-08 12:19:28 +00:00
Nga Tran 9356f2a1b9
feat: grpc for updating namespace retention period (#6041)
* refactor: make namespace folder for all namesapce's commands

* feat: WIP for add command to set retention period

* feat: more on updating retention period

* feat: grpc for update namespace retention period

* test: end to end test fpr namespace retention

* fix: lint proto

* chore: cleanup

* chore: kick CI run again

* fix: command hierachy

* chore: fix comments
2022-11-04 20:58:11 +00:00
Andrew Lamb 3ba0458653
feat: Add object_store handler to querier so `remote get-table` works (#6014)
* feat: Add object_store handler to querier

* test: end to end test for get-table from querier

* fix: doc links

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-02 14:20:26 +00:00
Nga Tran fba4408d05
refactor: move `influxdb_iox debug namespace` command to `influxdb_iox namespace` (#6031)
* refactor: move  command to

* docs: update the doc accordingly

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-02 10:57:58 +00:00
Carol (Nichols || Goulding) 69a2e6b871
feat: Last 2 bonus features of remote store get-table (#5991)
* feat: Only get files that aren't already on disk with the reported size

* feat: Stream Parquet file bytes to file on disk

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-28 11:03:08 +00:00
Carol (Nichols || Goulding) ace497d47c
fix: Rename database to namespace in the commands I just added 2022-10-27 10:40:39 -04:00
Carol (Nichols || Goulding) d65a6a86dd
fix: Make error output less repetitive/wordy 2022-10-27 10:30:58 -04:00
Carol (Nichols || Goulding) 47faca6843
feat: Allow specifying output dir for get-table 2022-10-27 10:30:57 -04:00
Carol (Nichols || Goulding) de2ae6f557
feat: MVP of remote store get-table command 2022-10-26 13:50:03 -04:00
Carol (Nichols || Goulding) 8697ef4967
feat: Set up CLI args for new get-table command 2022-10-26 11:19:00 -04:00
Carol (Nichols || Goulding) 71770486af
refactor: Extract influxdb_iox remote CLI tests to their own file 2022-10-26 11:19:00 -04:00
Marco Neumann 1d440ddb2d
refactor: `IOxReadFilterNode` can always accumulate statistics (#5954)
* refactor: `IOxReadFilterNode` can always accumulate statistics

`IOxReadFilterNode` used to not emit statistics if one chunk has
duplicates or delete predicates. This is wrong (or at least overly
conservative), because the node itself (or the chunks themselves) do NOT
perform dedup or delete predicate filtering. Instead this is done is
done by parent nodes (`DeduplicateExec` and `FilterExec`) and its their
job to propagate statistics correctly.

Helps w/ #5897.

* test: explain setup

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2022-10-24 13:34:22 +00:00
Carol (Nichols || Goulding) 59e1c1d5b9
feat: Pass trace id through Flight requests from querier to ingester
Fixes #5723.
2022-10-20 08:55:30 -04:00
Andrew Lamb 82d6fc3bda
feat: support queries via influxrpc with periods in field names (#5919)
* feat: support queries via influxrpc with periods in field names

* fix: update comments

* fix: more tests

* fix: more tests
2022-10-19 20:09:55 +00:00
Carol (Nichols || Goulding) c28ac4a3c3
fix: Return an error for unsupported SQL queries (#5876)
* test: Failing tests for unsupported queries

* fix: Catch unsupported SQL operations and error rather than return nothing

* test: Document a few more error messages that come through DataFusion

* refactor: Extract a Step to make query error tests nicer to read and write

* fix: update tests for new error codes

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-18 19:27:29 +00:00
Dom Dwyer c4f542bbe2 refactor(ingester): remove tombstone support
This commit removes tombstone support from the ingester, and deletes
associated code/helpers/tests. This commit does NOT remove tombstone
support from any other service, but MAY include removing overlapping
test coverage.

This also removes the tombstone support from the Ingester -> Querier RPC
response message.

This has the nice side effect of removing a whole lot of thread spawning
in the ingester tests for the Executor, speeding everything up!
2022-10-11 13:10:04 +02:00
Marco Neumann c4c83e0840
fix: query error propagation (#5801)
- treat OOM protection as "resource exhausted"
- use `DataFusionError` in more places instead of opaque `Box<dyn Error>`
- improve conversion from/into `DataFusionError` to preserve more
  semantics

Overall, this improves our error handling. DF can now return errors like
"resource exhausted" and gRPC should now automatically generate a
sensible status code for it.

Fixes #5799.
2022-10-06 08:54:01 +00:00
Andrew Lamb 82d5c7f336
feat: support parallel, chunked upload via `influxdb_iox write` of line protocol, gzip'd line protocol, and parquet (#5757)
* feat: Upload in small chunks and in parallel

* fix: doclink

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: Update influxdb_iox_client/src/client/write.rs

* fix: fixup error handling and fmt

* fix: Make default chunk sizes the same and add docs

* fix: clippy

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2022-09-30 21:02:38 +00:00
Nga Tran d171697fd7
feat: always pick cold partitions in next cycle even if it has been pa… (#5772)
* fix: always pick cold partitions in next cycle even if it has been partially compacted recently

* fix: comment

* fix: test output

* refactor: using var instead of literal

* fix: consider deleted L0s for recent writes

* chore: cleanup

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-30 15:54:00 +00:00
Carol (Nichols || Goulding) 39a810ec3a
feat: Generate parquet files instead of line protocol 2022-09-29 16:22:37 -04:00
Andrew Lamb 56a1c579a1
refactor: Change influxdb_iox client to use http rather than grpc for write (#5756)
* refactor: Change influxdb_iox client to use http rather than grpc for write

* refactor: remove custom variants

* refactor: consolidate more
2022-09-29 11:12:51 +00:00
Andrew Lamb 13ed1c089a
feat: use /api/v2 upload for write command rather than grpc `write_service` (#5749)
* feat: use /api/v2 upload for write command rather than grpc service

* fix: Update influxdb_iox/src/commands/write.rs
2022-09-28 11:16:51 +00:00