Commit Graph

20 Commits (ba53e71935c29043ba789015064bfce98553bbe2)

Author SHA1 Message Date
Nga Tran fba4408d05
refactor: move `influxdb_iox debug namespace` command to `influxdb_iox namespace` (#6031)
* refactor: move  command to

* docs: update the doc accordingly

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-02 10:57:58 +00:00
Carol (Nichols || Goulding) 71770486af
refactor: Extract influxdb_iox remote CLI tests to their own file 2022-10-26 11:19:00 -04:00
Carol (Nichols || Goulding) c28ac4a3c3
fix: Return an error for unsupported SQL queries (#5876)
* test: Failing tests for unsupported queries

* fix: Catch unsupported SQL operations and error rather than return nothing

* test: Document a few more error messages that come through DataFusion

* refactor: Extract a Step to make query error tests nicer to read and write

* fix: update tests for new error codes

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-18 19:27:29 +00:00
Andrew Lamb 82d5c7f336
feat: support parallel, chunked upload via `influxdb_iox write` of line protocol, gzip'd line protocol, and parquet (#5757)
* feat: Upload in small chunks and in parallel

* fix: doclink

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: Update influxdb_iox_client/src/client/write.rs

* fix: fixup error handling and fmt

* fix: Make default chunk sizes the same and add docs

* fix: clippy

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2022-09-30 21:02:38 +00:00
Nga Tran d171697fd7
feat: always pick cold partitions in next cycle even if it has been pa… (#5772)
* fix: always pick cold partitions in next cycle even if it has been partially compacted recently

* fix: comment

* fix: test output

* refactor: using var instead of literal

* fix: consider deleted L0s for recent writes

* chore: cleanup

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-30 15:54:00 +00:00
Andrew Lamb 13ed1c089a
feat: use /api/v2 upload for write command rather than grpc `write_service` (#5749)
* feat: use /api/v2 upload for write command rather than grpc service

* fix: Update influxdb_iox/src/commands/write.rs
2022-09-28 11:16:51 +00:00
Andrew Lamb 65f1550126
feat: Implement `debug parquet_to_lp` command to convert parquet to line protocol (#5734)
* feat: add `influxdb_iox debug parquet_to_lp` command

* chore: Run cargo hakari tasks

* fix: update command description

* fix: remove unecessary Result import

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2022-09-26 14:17:27 +00:00
Carol (Nichols || Goulding) 3cea67fe36
test: Compaction of 1 file now upgrades the level, so the ID doesn't change 2022-09-12 13:13:29 -04:00
Marco Neumann 267a53a9e8
chore: update `tracing-subscriber`, fix trogging, fix CLI test port allocation (#5581)
* test: use dedicated ports for CLI tests

* chore: update `tracing-subscriber`

* fix: work around tracing-subscriber weirdness

It seems that trogging with tracing-subscriber >= 0.3.14 does not
produce any output at all. I suspect we are hitting
<https://github.com/tokio-rs/tracing/issues/2265>. Let's change the
construct to not use multiple optional layers but a single dyn-dispatch
layer. Logging shouldn't have such a high throughput that his makes any
difference, esp. because the dyn-dispatch happens AFTER the filter.
2022-09-08 09:37:37 +00:00
Carol (Nichols || Goulding) 3aa3ae2ba5
docs: Add more comments about why to use ShardIndex or ShardId 2022-08-29 14:07:20 -04:00
Carol (Nichols || Goulding) 74c9529062
fix: Rename KafkaPartition to ShardIndex 2022-08-29 14:07:18 -04:00
Carol (Nichols || Goulding) 9321c96aaf
test: Add a step for running compaction in e2e tests 2022-08-10 11:30:22 -04:00
Carol (Nichols || Goulding) 45f8e567ed
fix: Revert adjustment of e2e test to expect compaction
This was part of
"feat: Different branch to hook up new compaction algorithm (#5194)"

and will be added back in a new test for compaction specifically in the
next commit.

This reverts part of commit 69640c0ba5.
2022-08-10 11:28:50 -04:00
Nga Tran 69640c0ba5
feat: Different branch to hook up new compaction algorithm (#5194)
* chore: cherry pick the first 3 commits of branch cn/connect-new-compaction

* fix: modify the test to work correctly with compactor running

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-22 19:29:47 +00:00
Sam Arnold 9438570ba1
test: document how to run tests (#4982)
* test: document how to run tests

Also fix a few issues for local runs.

* docs: add back one-liner for running end to end tests

* docs: add comment for clap_blocks test

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

* docs: add comment in influxdb_iox/tests/end_to_end_cases/cli.rs

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-06-30 14:01:35 +00:00
Dom Dwyer 1caeb04869 test(e2e): do not mangle prod database
Unset the all env vars for the following CLI e2e tests:

    * default_mode_is_run_all_in_one
    * default_run_mode_is_all_in_one

This prevents them from executing against the "prod" catalog, running
migrations and inserting values to the prod database specified in the
prod DSN env (INFLUXDB_IOX_CATALOG_DSN).
2022-06-01 17:12:12 +01:00
Dom Dwyer 60de97ac26 test(e2e): ensure "partition pull" writes files
Adds a test case covering the "remote partition pull" command configured
with file-based object storage.
2022-06-01 16:41:57 +01:00
Dom Dwyer 6d647fb7a9 refactor: warn for silly object store configs
Warn when downloading files to an in-memory object store.

The "remote partition pull" command downloads parquet files from an
object store via a router, and saves them locally. It's pretty unlikely
the user intends to download those files to memory of the CLI process
which then exits when the pull is complete, throwing away the downloaded
files, but this is the default.
2022-06-01 16:41:57 +01:00
Marco Neumann 2029bd16ba
feat: enable debugging of failed querier->ingester requests (#4659)
* feat: enable debugging of failed querier->ingester requests

- extend `query-ingester` CLI to allow usage of predicates
- on failed requests: log all information that required for the CLI
- test the "ingester fails" scenario

* test: explain

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

* docs: improve

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

* refactor: move b64 pred. serde into a single crate

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-05-23 15:37:31 +00:00
Carol (Nichols || Goulding) 30e53fd09c
fix: Rename end-to-end NG tests to not contain NG
Connects to #4450.
2022-05-12 16:09:07 -04:00