Commit Graph

125 Commits (37746173d94aa0fb14f6850c718274b778845e06)

Author SHA1 Message Date
Raphael Taylor-Davies 5c9dd68bf8
feat: MVP Management CLI (#907)
* feat: MVP CLI implementation

* feat: multiline database description
2021-03-03 17:37:55 +00:00
Raphael Taylor-Davies ffc20fa821
feat: add basic gRPC health service (#862)
* feat: add basic gRPC health service

* feat: update README.md add /health to HTTP API

* feat: add health client to influxdb_iox_client

feat: end-to-end test health check service

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-25 13:24:12 +00:00
Jake Goulding 2bd09612bd refactor: Enable parking lot for dependencies 2021-02-22 13:37:35 -05:00
Jake Goulding 6603ecd758 refactor: Unify on once_cell
This is closest in API to what is likely to be added to the standard
library at some point, so let's use it consistently.
2021-02-22 13:37:35 -05:00
Raphael Taylor-Davies 6d3c21b952 feat: Use Bytes instead of Vec<u8> in prost generated code
Also add google.rpc.* Protobuf definitions
2021-02-12 17:10:58 +00:00
Jake Goulding 699f4a577f feat: Add an optional Flight client to the IOx client library 2021-02-10 10:30:05 -05:00
Raphael Taylor-Davies 143488fae9 feat: add WAL metadata endpoint (#724) 2021-02-08 16:21:34 +00:00
Carol (Nichols || Goulding) fbf776c6b3
chore: Clean up Cargo.tomls (#754)
* fix: test_helpers crate should only be a dev-dep

* fix: object_store no longer has a build script, so no longer needs a build dep

* chore: Alphabetize all Cargo.tomls
2021-02-04 18:56:02 -05:00
Raphael Taylor-Davies 29314a6118 feat: consistent global error handling and logging 2021-02-04 13:15:17 +00:00
Andrew Lamb 485a59b2f8
feat: Implement logfmt (Heroku) formatted log output (#716)
* feat: add option to output logs formatted via logfmt

* refactor: Apply suggestions from code review

Co-authored-by: Edd Robinson <me@edd.io>

* fix: add tests for span inclusion

* feat: Also log spans

* fix: bug in normalizer

Co-authored-by: Edd Robinson <me@edd.io>
2021-02-01 16:43:01 -05:00
Andrew Lamb f3bd8bd0e3
chore: update deps (tokio 1.0 and ecosystem) (#707)
* chore: Update arrow + tokio deps

* chore: Use bleeding edge azure

* chore: Update aws + other deps

* fix: fmt

* fix: Switch to in-house version of routerify

* fix: Upgrade to hyper 0.14

The hyper::error module is now private; hyper::Error is the public
re-export

* fix: Upgrade cloud storage to get tokio upgrade

* fix: Upgrade open_telemetry

* fix: Do not call `panic::set_hook` during another panic

Doing so leads to a double panic which aborts the process.

* fix: new h2 error who dis

Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@integer32.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2021-01-29 16:11:55 -05:00
Dom 5647bfeb6f refactor: error handling & typed errors
Refactors the API method errors.

The user of the API client needs to be able to distinguish between various error
states when an API request fails. The most ergonomic way of exposing this
information is by returning an error enum that is specific to each API method
(or at least the important ones with well defined failure modes) - currently
only the `create_database()` method has significant error states, so this is
the only one with a specific error type in this impl.

This change defines a bunch of API error codes in the API client, adds them to
the IOx API error response body, and maps them in the API client. Due to error
wrapping the error code mapping in the IOx server is less exhaustive than I had
hoped however.
2021-01-13 17:32:12 +00:00
Dom f5586a9708 build: remove duplicate items from cargo workspace 2021-01-13 17:32:07 +00:00
Dom 62349edb94 feat: create IOx API client
Initialises a new library crate and implements a basic IOx API client.

The API client supports:
	- ping
	- create database

Care has been taken to abstract away the underlying HTTP client used
(reqwest) and avoid leaking it into the public API (error types is a
common leak!) This makes updating the HTTP client and/or swapping it for
something else a backwards compatible change for end users of the crate.

Outstanding items:
	- move shared API types into a sensible location
	- discriminate between various IOx error responses

The former doesn't need doing until we publish the crate and will likely
be rather invasive / conlict prone so aiming to merge this PR and then
move things around in a follow-up.

The latter would allow us to expose error conditions to the user such
that they can take actions to remidy the situation / know if the request
can or should be retried / etc. Currently we expose a string error
message when requests fail, requiring string matching and/or passing the
string higher in the stack (and thus punting the problem to the caller).
It would be very nice to have typed errors, but a detail I have left for
later.
2021-01-12 16:38:33 +00:00
Dom bdc832d040 refactor: replace config system with structopt
Replaces the hand-rolled config system with a StructOpt managed config struct.

I've got most of it ported across, but the interaction between all the logging
config bits is complex! I've left what is there and hooked in the value from
the config struct (which directly replaces the env var in usage, as it also
sources from the env).
2021-01-11 18:43:14 +00:00
Carol (Nichols || Goulding) b66ad643d5 refactor: Extract panic logging to its own crate for ease of reuse 2021-01-08 12:36:56 -05:00
Carol (Nichols || Goulding) 6f3b5a6a9e refactor: Alphabetize workspace members to match dir listing 2021-01-08 12:20:31 -05:00
Andrew Lamb 9f0ff678f1
feat: Formalizes the config system for IOx, including tests (#608)
* feat: Create configuration system, port IOx to use it

* docs: Apply suggestions from code review

Co-authored-by: Paul Dix <paul@influxdata.com>

* fix: fix test for setting values

Co-authored-by: Paul Dix <paul@influxdata.com>
2020-12-31 07:02:31 -05:00
Paul Dix 1d200c5c77 chore: move http API over to Routerify
This moves the HTTP API over to Routerify, which has the basic route parsing logic that will enable the API design for IOx.

I had a little trouble with the error handling in Routerify so I ended up creating a macro for constructing error responses in the HTTP API. I'm not sure what I think of this pattern so I'm interested in what others think. Another option would be to have two functions for each API endpoint. One which is x_handler with a Routerify function signature. Then another which is just x that has the Result<Response<Body>, ApplicationError> return type, which would make using the ? operator work in those functions. That would eliminate the need for the return_err macro.

I'm happy to refactor to that if people prefer it.
2020-12-24 16:45:20 -06:00
Edd Robinson 199ba68769 refactor: rename segment_store crate to read_buffer 2020-12-22 21:26:04 +00:00
Andrew Lamb 48c43b136c
refactor: rename write_buffer --> mutable_buffer (#595)
* refactor: git mv write_buffer mutable_buffer

* refactor: update crate name references

* refactor: update some more references
2020-12-22 10:49:53 -05:00
Dom 41f5099691 refactor: compile out trace! level for release builds
Configures the IOx tracing to compile out trace!() level events in the release
binary. This effectively gives contributors three levels of output:

* Important to the user (info & friends)
* Not important for regular running, but needed to debug
* Only useful to devs in a specific part of the system, never seen by user

Documents this behaviour (and general usage guidelines) for contributors.
2020-12-14 12:06:53 +00:00
Dom 60ee7e1dbb chore(deps): remove unused env_logger 2020-12-14 12:06:53 +00:00
Dom 9d7389dec2 feat(tracing): add Jaeger tracing sink
Adds telemetry / tracing with support for a Jaeger backend, and changes the
logger from env_logger to a tracing subscriber to collect the log entries.

Events are batched and then emitted asynchronosuly via UDP to the Jaeger
collector using the tokio runtime. There's a bunch of settings (env
vars) related to batch sizes and flush frequency etc - they're all using
their default values at the moment (if it ain't broke...) See the docs
for more info:

    https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/sdk-environment-variables.md#opentelemetry-environment-variable-specification

This is only part 1 of telemetry - it does NOT propagate traces across RPC
boundaries as we're still defining how all this should work. I've created #541
to track this.

Closes #202 and closes #203.
2020-12-14 12:06:52 +00:00
Paul Dix fa3ecbd4ed
feat: Implement write buffer to Parquet snapshotting (#526)
* feat: Implement write buffer to Parquet snapshotting

This introduces snapshot to the server packages to manage snapshotting. It also introduces a new trait for representing a Partition. There is a very crude API wired up in http_routes for testing purposes. Follow on work will bring the server package into http_routes and rework the snapshot API.
2020-12-08 14:20:43 -05:00
Edd Robinson f3af86ccb4 chore: add debug symbols to benchmarks 2020-12-04 16:36:05 +00:00
Dom c3a0e893ae test: use flate2 2020-12-01 11:01:10 +00:00
Dom 867aba847a perf(convert): use flate2 for gzip decompression
Switches from `libflate` to `flate2` for the top-level commands (specifically
TSM conversion).
2020-11-30 15:18:25 +00:00
Andrew Lamb cdb26e60e4
refactor: rename `storage` crate to `query` to better reflect what it is (#475)
* refactor: rename storage --> query

* refactor: update a few more referenes
2020-11-24 14:19:29 -05:00
Paul Dix 59e2024d01 chore: rename cluster crate to server 2020-11-20 12:14:12 -05:00
Andrew Lamb fe663c3534
feat: add cpu_feature_check (#458)
* feat: add cpu_feature_check

* fix: clarify output
2020-11-17 13:28:23 -05:00
Edd Robinson 0958849956 chore: rename the segment store crate 2020-11-10 16:35:17 +00:00
Andrew Lamb 5bb530ccc6
refactor: rename tsm --> influxdb_tsm (#418) 2020-11-05 14:35:38 -05:00
Andrew Lamb b745a180a4
refactor: rename delorean --> InfluDB IOx (#417) 2020-11-05 13:51:04 -05:00
Andrew Lamb a52e0001c5
refactor: rename all crates that start with`delorean_` in preparation for rename (#415)
* refactor: rename delorean_cluster --> cluster

* refactor: rebane delorean_generated_types --> generated_types

* refactor: rename delorean_write_buffer --> write_buffer

* refactor: rename delorean_ingest --> ingest

* refactor: rename delorean_storage --> storage

* refactor: rename delorean_tsm --> tsm

* refactor: rename delorean_test_helpers --> test_helpers

* refactor: rename delorean_arrow --> arrow_deps

* refactor: rename delorean_line_parser --> influxdb_line_protocol
2020-11-05 13:44:36 -05:00
Andrew Lamb 9df6c24493
refactor: rename delorean_mem_qe --> mem_qe (#414) 2020-11-05 09:36:46 -05:00
Andrew Lamb 4f348836fe
refactor: remove delorean_parquet by compining with delorean_ingest (#412) 2020-11-05 09:29:59 -05:00
Andrew Lamb ff824a5477
refactor: rename delorean_wal --> wal, conslidate wal_writer (#411) 2020-11-05 09:25:29 -05:00
Andrew Lamb a3b88d5506
refactor: rename delorean_object_store --> object_store (#413) 2020-11-05 08:56:30 -05:00
Andrew Lamb 8399d2a159
refactor: rename delorean_table to packers (#409) 2020-11-05 08:52:22 -05:00
Andrew Lamb 075ba0d8d1
refactor: remove delorean_table_schema crate and fold it into data_types (#408) 2020-11-05 06:17:20 -05:00
Andrew Lamb bf0c58698e
refactor: rename delorean_data_types crate to data_type (#407)
* refactor: rename delorean_data_types crate to data_type - #401

* fix: fmt
2020-11-04 12:33:41 -05:00
Andrew Lamb ef501871bb
feat: remove partition_store (#387) 2020-10-26 14:39:38 -04:00
Paul Dix a80eb0fed3 feat: Store replicated writes
This commit refactors the flatbuffers data types from the wal to a new crate where they can be used by storage, write buffer, and cluster. It also refactors cluster to move the configuration types out to the data types crate so they can be used across storage and elsewhere.

Finally, it adds a new method to store replicated writes on a database in the database trait and implements it.
2020-10-11 15:45:08 -04:00
Edd Robinson a2287acb7c
Merge pull request #330 from influxdata/er/feat/segment-store-shell
feat: Segment Store shell
2020-10-01 14:01:45 +01:00
Paul Dix fdc86fd186
feat: add some initial framework for clustering (#329) 2020-09-30 14:41:42 -04:00
Edd Robinson 2470bdb975 feat: segment store shell 2020-09-30 11:25:59 +01:00
Edd Robinson ec1aaa3a47 chore: update dependencies 2020-09-25 17:22:48 +01:00
Edd Robinson c42d2dcd79 refactor: rebase with delorean_arrow 2020-09-25 10:12:46 +01:00
Edd Robinson aba02cb731 feat: basic store 2020-09-25 10:12:46 +01:00