Commit Graph

321 Commits (eb15e2e8d9f71a45faf3037f2719cf35596f87ac)

Author SHA1 Message Date
Carol (Nichols || Goulding) f1f475d552
fix: Remove compactor generate (#6872)
This debugging tool was more useful in previous situations where it was
harder to get real data as input for the compactor.

It's currently causing a flaky test that isn't worth investigating.

Fixes #6190 by making it moot.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-06 19:19:15 +00:00
Dom Dwyer b8ec022ea6
test(e2e): HTTP error response
Ensure a HTTP error response contains a well-formed JSON structure
containing "code" and "message" fields (for backwards compatibility with
existing InfluxDB versions) and a correct "content-type" header.
2023-02-06 17:10:30 +01:00
Raphael Taylor-Davies d3601a59f8
chore: update DataFusion, upgrade `arrow` `arrow-flight` and `parquet` to `32.0.0` (#6756)
* chore: update DataFusion

* fix: test

* chore: format

* chore: clippy

* chore: update arrow

* chore: arrow upgrade fallout

* chore: Run cargo hakari tasks

* chore: remove failing warm compaction test

* fix: flight error propagation

* chore: update parquet size

* fix: Update error message

* chore: Update parquet metadata test

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-06 11:35:39 +00:00
Carol (Nichols || Goulding) ca8558e636
chore: Unnest test code out of the module and clean up long lines 2023-02-03 14:20:30 -05:00
Carol (Nichols || Goulding) b17d1c0862
test: Port more query tests to kafkaless 2023-02-03 13:56:01 -05:00
Carol (Nichols || Goulding) 30fea67701
fix: Move variables within format strings. Thanks clippy!
Changes made automatically using `cargo clippy --fix`.
2023-02-03 13:06:17 -05:00
Carol (Nichols || Goulding) b55e4ffa21
test: Port table_schema tests to end-to-end and get rid of query_tests (#6828)
* test: Port table_schema query_tests to end-to-end

* fix: Remove now outdated query_tests crate

---------

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-03 17:41:08 +00:00
Andrew Lamb 0d01306fd2
chore: port remote end to end tests to new write path (#6838)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-03 15:33:49 +00:00
Andrew Lamb b9ab09e944
chore: remove kafka full querier end to end tests (#6834)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-03 15:25:24 +00:00
Andrew Lamb ef98b8c348
chore: Port `influxrpc` end to end tests to new writepath (#6839)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-03 14:58:18 +00:00
Andrew Lamb 05ca71f91f
chore: Remove old ingester end to end tests (#6837)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-03 14:43:27 +00:00
Andrew Lamb 755d3586ea
chore: port cli end to end tests to kafkaless write path (#6836)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-03 14:35:05 +00:00
Andrew Lamb 012c35bf3e
chore: Port flightsql end to end tests for new kafkaless writepath (#6835)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-03 14:26:26 +00:00
Andrew Lamb acc258e9aa
chore: Update influxql end to end tests to kafkaless write path (#6833)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-03 14:11:55 +00:00
Stuart Carnie 63d0a77daf
feat: Updating to new services for all-in-one (#6811)
* feat: Updating to new services for all-in-one

* fix: Use correct shard id for ingester2

* fix: clippy

* fix: use wal directory

* fix: end to end tests

* fix: Update tracing cases for new ingest reality

* fix: update metrics test

* fix: Use rpc mode

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2023-02-02 20:42:29 +00:00
Carol (Nichols || Goulding) 7590fa5aaa
fix: Change test data and predicates to get better coverage
These two tests weren't actually ensuring that the combination of these
predicates worked, because the tests would still pass if some of the
predicate parts were removed.
2023-02-01 11:09:10 -05:00
Carol (Nichols || Goulding) 072e228d35
test: Port tag_values query_tests to end-to-end tests 2023-02-01 10:26:03 -05:00
Carol (Nichols || Goulding) 629867c6ac
refactor: Start a new file for tag values tests; move the one existing test 2023-02-01 10:25:48 -05:00
Carol (Nichols || Goulding) ff32a042b0
test: Port tag_keys query_tests to end-to-end tests (#6760)
* refactor: Start a new file for tag keys tests; move the one existing test

* test: Port tag_keys query_tests to end-to-end tests

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2023-01-31 14:05:22 +00:00
Carol (Nichols || Goulding) d18d2c34e4
test: Port measurement_names/table_names query_tests to end-to-end tests (#6757)
* refactor: Start a new file for measurement names tests; move the one existing test

* fix: Pass on predicate when sending a measurement names request with GrpcRequestBuilder

* feat: Support literal integer queries too

* test: Port measurement_names/table_names query_tests to end-to-end tests

* fix: merge conflict error

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-31 12:55:30 +00:00
Carol (Nichols || Goulding) cff422b795
test: Port read_window_aggregate query_tests to end-to-end tests (#6755)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-31 12:15:50 +00:00
Carol (Nichols || Goulding) 2254ee06b7
test: Port read_group query_tests to end-to-end tests (#6743)
* fix: Delete the read filter *file*; last PR only deleted the *contents*

* test: Port read_group query_tests to end-to-end tests

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-30 12:10:39 +00:00
Christopher M. Wolff 55257b46c9
chore: validate ingester URIs on querier CLI (#6740)
* chore: add validate for ingesters on querier CLI

* chore: fix typo and tests

* chore: clippy

* chore: review feedback

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-27 21:13:52 +00:00
Andrew Lamb 4a6b4f78d2
refactor: Make some nicer methods on the GrpcRequestBuilder (#6741)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-27 21:06:22 +00:00
Carol (Nichols || Goulding) 11233e3b3b
test: Translate the filter_on_value test using field ref (#6734) 2023-01-27 20:03:44 +00:00
Carol (Nichols || Goulding) 4f8dd072b3
fix: Translate a test with a predicate of a literal = literal 2023-01-27 10:28:43 -05:00
Carol (Nichols || Goulding) 94f7f015f4
fix: Port a test with a predicate that tag=tag, which is always true 2023-01-27 10:28:43 -05:00
Carol (Nichols || Goulding) a2b67abe54
fix: Remove test cases that aren't valid to port to end-to-end tests 2023-01-27 10:28:43 -05:00
Carol (Nichols || Goulding) 67c430da63
test: Port read_filter query_tests to end-to-end tests 2023-01-27 10:28:43 -05:00
Andrew Lamb 589fbbf11c
chore: remove unecessary checks for persisted in end to end tests (#6713)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-26 18:24:39 +00:00
Andrew Lamb c100737a81 chore: Do not send dictionary encoded data to clients 2023-01-26 06:35:15 -05:00
Nga Tran b8a80869d4
feat: introduce a new way of max_sequence_number for ingester, compactor and querier (#6692)
* feat: introduce a new way of max_sequence_number for ingester, compactor and querier

* chore: cleanup

* feat: new column max_l0_created_at to order files for deduplication

* chore: cleanup

* chore: debug info for chnaging cpu.parquet

* fix: update test parquet file

Co-authored-by: Marco Neumann <marco@crepererum.net>
2023-01-26 10:52:47 +00:00
Andrew Lamb 7853a19953
feat: JDBC integration tests with FlightSQL (#6693)
* feat: basic JDBC integration test

* fix: do not run test without env set

* docs: add maven link

* refactor: clean up java with switch statement
2023-01-25 22:21:18 +00:00
Carol (Nichols || Goulding) 57b5b639d6
test: Port all field columns query_tests to end-to-end tests (#6707)
* test: Port a test that's not actually supported through the full gRPC API

* test: Port remaining field column/measurement fields tests

* test: Remove unsupported measurement predicate and clarify purposes of tests

Andrew confirmed that the only way to invoke a Measurement Fields
request is with a measurement/table name specified: <0249b5018e/generated_types/protos/influxdata/platform/storage/service.proto (L43)>

so testing with a `_measurement` predicate is not valid.

I thought this test would become redundant with some other tests, but
they're actually still different enough; I took this opportunity to
better highlight the differences in the test names.

* refactor: Move all measurement fields tests to their own file

* test: Remove field columns tests that are now covered in end-to-end measurement fields tests
2023-01-25 19:49:29 +00:00
Carol (Nichols || Goulding) 4658510102
fix: For Ingester2, persist a particular namespace on demand and share MiniClusters
This should hopefully help CI from running out of Postgres
connections 😬

The old architecture will still need to be non-shared and persist
everything.
2023-01-25 10:36:56 -05:00
Carol (Nichols || Goulding) f310e01b1a
test: Start of porting InfluxRpc query_tests
Make a new trait, `InfluxRpcTest`, that types can implement to define
how to run a test on a specific Storage gRPC API. `InfluxRpcTest` takes
care of iterating through the two architectures, running the setups, and
creating the custom test step.

Implementers of the trait can define aspects of the tests that differ
per run, to make the parameters of the test clearer and highlight what
different tests are testing.
2023-01-25 10:27:42 -05:00
Andrew Lamb 0c55a0f257
feat: Implement basic prepared statement support in IOx (#6667)
* feat: allow override of flightsql namespace

* feat: Implement DoAction endpoint

* refactor: Remove try_unpack

* fix: remove unused code / more clone
2023-01-25 12:00:43 +00:00
Andrew Lamb 1b882e0062
fix: `error arrow/ipc: could not read message schema: EOF` (#6668)
* chore: Test for schema from query

* fix: Send schema even for no RecordBatches

* fix: docs
2023-01-23 22:23:34 +00:00
Andrew Lamb 65c020c9f2
refactor: remove iox_arrow_flight use in `influxdb_iox_client ` and `querier` (#6624)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-19 18:48:23 +00:00
Carol (Nichols || Goulding) f3b5dcaab7
feat: Reimagining query_tests 2023-01-18 10:24:17 -05:00
Dom f7ff877582
Merge branch 'main' into cn/ingester-persist-tick 2023-01-13 12:31:45 +00:00
Carol (Nichols || Goulding) f56123bf30
test: Allow integration tests that should_panic to pass if TEST_INTEGRATION isn't set 2023-01-12 15:31:34 -05:00
Carol (Nichols || Goulding) 1c7ffb95df
test: Write a should_panic test that shows ingester is persisting when I thought it wouldn't 2023-01-12 14:55:28 -05:00
Carol (Nichols || Goulding) b989e0893f
test: Make persist-on-demand test a step test and check the number of parquet files 2023-01-12 11:40:46 -05:00
Carol (Nichols || Goulding) 3a2544a7eb
feat: Define a new gRPC service for ingester persist 2023-01-12 11:03:12 -05:00
Carol (Nichols || Goulding) e9b3efb33d
refactor: Extract a method for making requests to the ingester onto MiniCluster 2023-01-12 11:03:10 -05:00
Dom 1e5b594863
Merge branch 'main' into dom/shutdown-persist 2023-01-12 10:15:34 +00:00
Carol (Nichols || Goulding) be7c312033
fix: Wait for a particular number of Parquet files, not just any change 2023-01-11 12:11:56 -05:00
Carol (Nichols || Goulding) 7e921e6a23
fix: Make recording num parquet files an explicit test step
To support a case where someone calls WriteLineProtocol twice in
a row to simulate two write requests. The test should be able to
record this state before the two write requests and not twice.
2023-01-11 11:51:56 -05:00
Carol (Nichols || Goulding) 6677ae5c61
test: Record number of Parquet files before a write
Fixes #6506.

Also has the pleasant side effect of making this code simpler and less
hacky-- it now checks the number of Parquet files for the whole
namespace, which is useful in cases where the line protocol writes to
several tables.
2023-01-11 11:51:55 -05:00