Commit Graph

419 Commits (30d1c77d363e5d0b730800aa113f3defbed6e9c6)

Author SHA1 Message Date
Carol (Nichols || Goulding) c891f3c4f2
fix: Print an error with sample env var variable if unset 2022-03-09 09:55:43 -05:00
Carol (Nichols || Goulding) 8af2f60b59
fix: Run catalog setup as part of end-to-end test setup 2022-03-09 09:55:43 -05:00
Carol (Nichols || Goulding) 93b0cdbcc4
fix: Create the test database as part of ng server fixture startup 2022-03-09 09:55:43 -05:00
Carol (Nichols || Goulding) e4fb227c6e
feat: Set the catalog URL explicitly in the test config 2022-03-09 09:55:43 -05:00
Carol (Nichols || Goulding) 465fb0272d
fix: Remove unneeded server id const 2022-03-09 09:55:43 -05:00
Carol (Nichols || Goulding) 6d086705a8
fix: Remove ng create_shared server fixture method for now 2022-03-09 09:55:42 -05:00
Carol (Nichols || Goulding) e315012fe3
fix: Switch NG tests to use TEST_INFLUXDB_IOX_CATALOG_DSN too 2022-03-09 09:55:42 -05:00
Carol (Nichols || Goulding) a14e642f39
refactor: Extract NG end-to-end tests from OG end-to-end tests 2022-03-09 09:55:42 -05:00
Carol (Nichols || Goulding) 1536bdeca0
test: Add an end-to-end NG test 2022-03-09 09:55:42 -05:00
Carol (Nichols || Goulding) 66a5649258
test: Print out the server type with server log messages
So that when you have more than one server running in a test, it's
easier to see which one is saying what
2022-03-09 09:55:42 -05:00
Carol (Nichols || Goulding) ae45d9f750
test: Add Router2 as a supported ServerFixture type 2022-03-09 09:55:42 -05:00
Andrew Lamb d2c0acdd46
refactor: Remove serving readiness gate (#3986)
* refactor: Remove serving_readiness

* fix: remove more

* fix: remove test

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-09 12:17:44 +00:00
Nga Tran 09fba1d2c0
feat: NG Compactor - main function for finding and compacting parquet files (#3973)
* feat: main function for finding and compacting parquet files

* chore: Apply suggestions from code review

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

* refactor: rename file and struct

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-08 16:34:43 +00:00
Andrew Lamb 357ea52d8d
refactor: decouple router2 setup and creation logic from the config structs, switch to `Arc<dyn ServerType>` (#3969)
* refactor: Extract router server instantiation into `influxdb_ioxd`

* refactor: complete dyn ServerType

* fix: remove some leftovers

* fix: clippy

* fix: restore startup message order

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-08 14:06:18 +00:00
Andrew Lamb 7fa17ef1d7
refactor: remove unused error enums (#3970)
* refactor: remove unused error enums

* fix: fmt

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-07 21:40:55 +00:00
Paul Dix 337e432a0f
feat: ingester persists on cold partitions (#3942)
Add configuration and lifecycle to trigger partition persistence if it hasn't received a write in a given number of secods.

Fixes #3869
2022-03-07 18:55:56 +00:00
Dom Dwyer aaec1c7828 build: propagate feature flags to influxdb_ioxd
When you did a "cargo build --no-default-features" before this commit,
the influxdb_iox crate was compiled without the default features
(namely jemalloc stuff) but the influxdb_ioxd crate was compiled with
jemalloc (it was not possible to compile without jemalloc support).

This commit fixes propagation of the jemalloc flag
("jemalloc_replacing_malloc") into the influxdb_ioxd crate.
2022-03-07 15:44:13 +00:00
Raphael Taylor-Davies 80fb75d90b
feat: add a flag to enable per-partition tracing (#3928)
* feat: add a flag to enable per-partition tracing

* chore: rename constant

* feat: use BooleanFlag and cache result
2022-03-07 13:49:23 +00:00
kodiakhq[bot] e47bfc40be
Merge branch 'main' into dom/router-grpc-write 2022-03-07 11:13:19 +00:00
dependabot[bot] 48908054d1
chore(deps): Bump once_cell from 1.9.0 to 1.10.0 (#3955)
* chore(deps): Bump once_cell from 1.9.0 to 1.10.0

Bumps [once_cell](https://github.com/matklad/once_cell) from 1.9.0 to 1.10.0.
- [Release notes](https://github.com/matklad/once_cell/releases)
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md)
- [Commits](https://github.com/matklad/once_cell/compare/v1.9.0...v1.10.0)

---
updated-dependencies:
- dependency-name: once_cell
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-07 08:54:20 +00:00
Dom Dwyer 7c5ba34d44 refactor: enable gRPC handler
Plumbs the gRPC write handler into the existing router2 server.
2022-03-04 14:51:43 +00:00
Raphael Taylor-Davies 636381c805
fix: fix read-window-aggregate CLI defaults (#3930) 2022-03-04 12:09:07 +00:00
Dom Dwyer bb9b140f4b refactor: sequencer metrics
Records per-sequencer (kafka partition) enqueue latency / counts broken
down by operation success/error.
2022-03-03 23:40:13 +00:00
Raphael Taylor-Davies e304613546
feat: include trace ID in query log (#3912) (#3923)
* feat: include trace ID in query log (#3912)

* chore: fmt

* chore: lint
2022-03-03 17:50:49 +00:00
Andrew Lamb d6afda0227
refactor: split `influxdb_ioxd`, `clap_blocks` and `serving_readiness` from influxdb_iox crate (#3908)
* refactor: split influxdb_ioxd, clap_blocks, and serving_readiness out of influxdb_iox

split out serving readiness, get compiling

* fix: hakari

* fix: hakari again

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-03 16:48:30 +00:00
Raphael Taylor-Davies 82aa314659
feat: make wait-server-initialized wait for database init (#3915)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-03 15:37:22 +00:00
Edd Robinson ea32bc366a feat: add read_group tracing 2022-03-03 14:27:01 +00:00
Edd Robinson 32baaa1ee7 feat: add tracing to field_columns 2022-03-03 14:27:01 +00:00
Edd Robinson 787a848bf5 feat: add tracing for tag_values 2022-03-03 14:27:01 +00:00
Edd Robinson 6a6fbf73ae feat: add tracing support tag_keys 2022-03-03 14:27:01 +00:00
kodiakhq[bot] 04a7a957fe
Merge branch 'main' into dom/rustls 2022-03-03 11:59:40 +00:00
kodiakhq[bot] c0ebe5a2f1
Merge branch 'main' into dom/fix-feature-flags 2022-03-03 11:48:35 +00:00
Dom Dwyer 6204a9046d fix: jemalloc feature flag
In #2435 we moved the jemalloc metrics around, and missed a few feature
gates to avoid pulling in jemalloc unless it is requested.
2022-03-03 11:48:24 +00:00
Raphael Taylor-Davies 0660b514e3
feat: allow persisting all partitions and/or all tables from CLI (#3874)
* feat: allow persisting all partitions and/or all tables

* chore: use BTreeSet

* feat: non-zero exit code on error continue

* test: test continue-on-error

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-03 11:17:34 +00:00
Dom Dwyer 46bb107be4 refactor: use rustls
Removes openssl as a dependency, switching to rustls[1] as the TLS
implementation throughout.

It is important to note that this change brings with it a significant
behavioural difference - rustls does not currently support IP SANs in
certificates (instead only supporting fully-qualified names / DNS) and
this will manifest as a failure to connect to IP endpoints over TLS.
This might be a blocker that prevents us using rustls exclusively, but
there's noe asy way to know without trying it. Fortunately the rustls
project has received funding to work on IP SAN support[2].

[1]: https://github.com/rustls/rustls
[2]: https://www.abetterinternet.org/post/preparing-rustls-for-wider-adoption/
2022-03-03 11:05:20 +00:00
kodiakhq[bot] caba3e9fd2
Merge branch 'main' into cn/querier-flight-request 2022-03-02 20:30:00 +00:00
Carol (Nichols || Goulding) 806a43eb6c
fix: Temporarily remove ingester end-to-end test case
This now needs more setup because without any data, the flight query
request returns an error. The setup is proving to be nontrivial.
2022-03-02 15:29:48 -05:00
Edd Robinson 3d047073b9
feat: add tracing down to the chunk level (#3804)
* refactor: wire exectution context to Deduplicator

* feat: example trace to chunk read_filter

* refactor: make execution context required

* refactor: expose metadata API

* refactor: more span context for chunk read_filter

* refactor: fix build

* refactor: push context into result stream

* refactor: make executor optional
2022-03-02 19:08:22 +00:00
Carol (Nichols || Goulding) 2a90841715
refactor: Move IngesterQueryRequest to data_types2
So that querier doesn't need to depend on ingester.
2022-03-02 13:52:13 -05:00
Carol (Nichols || Goulding) 8f3e44bf76
refactor: Extract a crate for shared data types in the new design 2022-03-02 12:16:15 -05:00
Carol (Nichols || Goulding) 16d86ed05b
feat: Deserialize metadata to get max_sequencer_number
And add an end-to-end test for the flight request to the ingester.
2022-03-02 11:50:47 -05:00
Carol (Nichols || Goulding) 141a6087d0
feat: Querier able to send Flight queries to Ingester
Fixes #3773.
2022-03-02 11:50:45 -05:00
Andrew Lamb 286d5f7b2b
feat: add `success` column to system.queries (#3891)
* feat: add `success` column to system.queries

* refactor: Remove lifetime from QueryCompletedToken and thread through flight

* test: update test to make incomplete query clearer

* refactor: use better patter to set complete

* fix: logical merge conflict
2022-03-02 15:05:06 +00:00
Dom Dwyer bd64f55658 feat: http ingest metrics
Records LP line count, field count & request body size (decompressed,
byte size) for writes, and request body byte size for deletes.
2022-03-02 13:05:55 +00:00
Marco Neumann af57664f53
feat: wire up flight into the querier (#3889) 2022-03-02 09:20:26 +00:00
Marko Mikulicic 39e39f92a8
fix: remove rskafka from valid options for INFLUXDB_IOX_WRITE_BUFFER_TYPE (#3890)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-01 17:29:02 +00:00
Marco Neumann 01f4d3ed3b
fix: check arrow instead of storage gRPC health for SQL REPL (#3886)
There is no reason a query pod should support the storage API. Note that
some features like the observer mode or `show databases;` still need the
management API. We'll probably need to fix that for NG at some point.
2022-03-01 15:46:02 +00:00
Marco Neumann ace4af1b66
feat: `DedicatedExecutor` async `join` and job `detach`. (#3835)
* feat: detach dedicated exec jobs

* feat: async `DedicatedExecutor::join`

Now `DedicatedExecutor` follows the system we use for other server
components:

- `shutdown`:  a quick sync call that signals the shutdown but doesn't
  drop
- `join`: async awaits until the executor has finished shutdown
- `drop`: warn but still try to shut down

* test: irmpove `detach_receiver` test

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-03-01 15:25:31 +00:00
Marco Neumann e5c45aeab6
refactor: generic top-level flight implementation (#3882)
This allows us to implement flight for the NG querier by just
implementing a few traits and reuse all the existing glue code and
optimizations (like dictionary handling).
2022-03-01 14:33:08 +00:00
Marco Neumann 48722783f9
feat: offer metrics for in-mem catalog (#3876)
This can be quite helpful to test certain caching behavior w/o writing
yet-another abstraction layer.
2022-03-01 11:33:54 +00:00