Commit Graph

255 Commits (706115178b080c674e342c8c9d209eafb92ea381)

Author SHA1 Message Date
Andrew Lamb 746373a687
refactor: Remove mutable_buffer crate dependency on query crate (#927) 2021-03-05 11:34:27 +00:00
Andrew Lamb 3abfb5f089
chore: Update arrow deps, turn off optional datafusion features (#930)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-04 22:41:54 +00:00
Raphael Taylor-Davies 5c9dd68bf8
feat: MVP Management CLI (#907)
* feat: MVP CLI implementation

* feat: multiline database description
2021-03-03 17:37:55 +00:00
Nga Tran 957e05ef25 chore: use newly added Arrow's Expr::is_not_null function 2021-03-03 11:46:49 -05:00
Raphael Taylor-Davies 51981c92f5
feat: implement gRPC API and migrate influxdb_iox_client to use it (#853)
* feat: implement gRPC management API

* feat: migrate influxdb_iox_client to use gRPC API

* fix: review comments

* refactor: separate influxdb_iox_client error types

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-02 17:51:46 +00:00
Andrew Lamb 86c82be7f1
chore: Update arrow deps, remove custom JsonArrayWriter (#888)
* chore: update dependencies

* refactor: Use Arrow json::ArrayWriter
2021-02-27 10:18:43 +00:00
kodiakhq[bot] 76921bdef9
Merge branch 'main' into alamb/update_deps 2021-02-25 13:24:32 +00:00
Raphael Taylor-Davies ffc20fa821
feat: add basic gRPC health service (#862)
* feat: add basic gRPC health service

* feat: update README.md add /health to HTTP API

* feat: add health client to influxdb_iox_client

feat: end-to-end test health check service

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-25 13:24:12 +00:00
Andrew Lamb d29d7efa8c chore: Update arrow/datafusion deps again 2021-02-25 07:39:01 -05:00
kodiakhq[bot] 3502f96ab9
Merge branch 'main' into cn/google-list-with-delimiter 2021-02-22 19:42:49 +00:00
Jake Goulding 6e6cc616a0 refactor: Switch to parking_lot::Mutex 2021-02-22 13:51:31 -05:00
Jake Goulding 2bd09612bd refactor: Enable parking lot for dependencies 2021-02-22 13:37:35 -05:00
Jake Goulding 6603ecd758 refactor: Unify on once_cell
This is closest in API to what is likely to be added to the standard
library at some point, so let's use it consistently.
2021-02-22 13:37:35 -05:00
Raphael Taylor-Davies dd8b41cdb0
feat: encoding of standard gRPC error details payloads (#846)
* feat: encoding of standard gRPC error details payloads

* feat: return unknown error on failure to encode error details

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-22 18:03:17 +00:00
Carol (Nichols || Goulding) cff12da3a1 fix: Upgrade to released version of cloud_storage
Fixes #801.
2021-02-22 13:01:06 -05:00
Carol (Nichols || Goulding) a42103f436 Merge remote-tracking branch 'origin/main' into cn/google-list-with-delimiter 2021-02-22 12:53:46 -05:00
Carol (Nichols || Goulding) 57942b51b7 feat: Update to latest Azure sdk to get delimiter support
Needed these PRs:
  - https://github.com/Azure/azure-sdk-for-rust/pull/176
  - https://github.com/Azure/azure-sdk-for-rust/pull/179

Also needed to enable the queue feature to get the azure_storage crate
compiling; at the moment, the code is still being reorganized and the
features aren't independent yet:
https://github.com/Azure/azure-sdk-for-rust/issues/177
2021-02-18 14:59:06 -05:00
Marko Mikulicic 536c1724bd feat: Allow to put streams of unknown length to objectstore
Addresses the API aspect of #818

Adds a utility module that helps computing the length of a stream while buffering it
for later replay (in-memory or spilling it in a temporary file).
2021-02-18 16:49:18 +00:00
NGA TRAN 213094f8f7 chore: update Arrow dependencies 2021-02-18 10:02:57 -05:00
Carol (Nichols || Goulding) ef54131afb feat: Gets google cloud list_with_delimiter tests passing 2021-02-17 14:23:33 -05:00
Andrew Lamb 071b13b939
chore: Update dependencies (#821)
* chore: Update dependencies

* fix: update udf implementation for DataFusion update

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-16 23:27:36 +00:00
Andrew Lamb 150baec84c chore: Update arrow dependencies 2021-02-14 05:47:23 -05:00
Raphael Taylor-Davies 6d3c21b952 feat: Use Bytes instead of Vec<u8> in prost generated code
Also add google.rpc.* Protobuf definitions
2021-02-12 17:10:58 +00:00
Andrew Lamb b8f85967dd
feat: Enable/Disable logging in tests via RUST_LOG environment variable (#793)
* feat: Enable/Disable logging in tests via RUST_LOG environment variable

* docs: Add section to contributing

* docs: tweak readme

* fix: Use same logging system in tests as in influxdb_ioxd
2021-02-12 13:43:12 +00:00
Andrew Lamb a03598dfe2
feat: Implement Cross Chunk Schema / RecordBatch merging at query time (#783)
* feat: feat: Implement Cross Chunk Schema / RecordBatch merging at query time

* docs: update comments about NullArray::new_with-type

* docs: Update comments based on code review

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-11 18:26:38 +00:00
kodiakhq[bot] a242fc39aa
Merge branch 'main' into jg/flight-client 2021-02-11 14:29:54 +00:00
Raphael Taylor-Davies 7debe94ee6 feat: add background task tracking (#655) 2021-02-11 10:30:19 +00:00
Jake Goulding 699f4a577f feat: Add an optional Flight client to the IOx client library 2021-02-10 10:30:05 -05:00
Raphael Taylor-Davies 143488fae9 feat: add WAL metadata endpoint (#724) 2021-02-08 16:21:34 +00:00
Raphael Taylor-Davies 29314a6118 feat: consistent global error handling and logging 2021-02-04 13:15:17 +00:00
Andrew Lamb d5ebf9c3da
chore: Update deps again (#738) 2021-02-04 06:02:05 -05:00
Jake Goulding a5e09366b0 feat: Export arrow-flight from arrow-deps 2021-02-03 09:56:56 -05:00
Carol (Nichols || Goulding) 0f8ef9c7d5
Merge branch 'main' into cn+jg/osp-types 2021-02-03 09:09:04 -05:00
Andrew Lamb abc26a33c1
chore: Update dependencies (again) (#718)
* chore: Update dependencies (again)

* refactor: update for changes in DataFusion API

* fix: fmt

* fix: clippy
2021-02-02 18:33:01 -05:00
Andrew Lamb 485a59b2f8
feat: Implement logfmt (Heroku) formatted log output (#716)
* feat: add option to output logs formatted via logfmt

* refactor: Apply suggestions from code review

Co-authored-by: Edd Robinson <me@edd.io>

* fix: add tests for span inclusion

* feat: Also log spans

* fix: bug in normalizer

Co-authored-by: Edd Robinson <me@edd.io>
2021-02-01 16:43:01 -05:00
Carol (Nichols || Goulding) ff6955a433 refactor: Extract a trait for ObjectStoreApi with associated path
This is the promised cleanup. This structure gets rid of a lot of
intermediate structures and encodes through associated types how the
object stores and path types are related.

The enums are still necessary to avoid having generics leak all over
the place, but the object store variants and path variants should always
match because they'll always come from the object store trait
implementations that use the associated types.
2021-02-01 14:56:47 -05:00
Andrew Lamb f3bd8bd0e3
chore: update deps (tokio 1.0 and ecosystem) (#707)
* chore: Update arrow + tokio deps

* chore: Use bleeding edge azure

* chore: Update aws + other deps

* fix: fmt

* fix: Switch to in-house version of routerify

* fix: Upgrade to hyper 0.14

The hyper::error module is now private; hyper::Error is the public
re-export

* fix: Upgrade cloud storage to get tokio upgrade

* fix: Upgrade open_telemetry

* fix: Do not call `panic::set_hook` during another panic

Doing so leads to a double panic which aborts the process.

* fix: new h2 error who dis

Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@integer32.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2021-01-29 16:11:55 -05:00
Andrew Lamb 8308fad188
chore: update arrow deps again (#699) 2021-01-26 07:55:30 -05:00
Andrew Lamb 9b6fbae7f5 chore: Bump arrow deps 2021-01-23 08:09:46 -05:00
Andrew Lamb a967e2f1dd
fix: disallow control characters in Database names (#684) 2021-01-21 17:55:55 -05:00
Andrew Lamb c50f9b1baf
Merge branch 'main' into alamb/underscore_in_bucket_names_2 2021-01-21 15:53:27 -05:00
Andrew Lamb 747b96d801
chore: Upgrade arrow dependencies, reduce duplication with upstream (#676) 2021-01-21 08:58:11 -05:00
Andrew Lamb a6b9ff9c91 fix: allow arbitrary characters in org/bucket names 2021-01-20 17:58:15 -05:00
Andrew Lamb 7969808f09
feat: Chunk Migration APIs and query data in the read buffer via SQL (#668)
* feat: Chunk Migration APIs and query data in the read buffer via SQL

* fix: Make code more consistent

* fix: fmt / clippy

* chore: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* refactor: Remove unecessary Result and make chunks() infallable

* chore: Apply more suggestions from code review

Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Edd Robinson <me@edd.io>
2021-01-19 13:28:26 -05:00
Andrew Lamb 71627120b9
refactor: consolidate line protocol schema creation into data_types and port code to use it (#663)
* refactor: consolidate line protocol schema creation into data_types, and port code to use it

refactor: Port mutable buffer to use SchemaBuilder

* fix: doctest

* refactor: remove unecessary clippyisms

* docs: Improve comments via suggestions from code review

Co-authored-by: Edd Robinson <me@edd.io>

* refactor: use more idomatic try_ naming and TryInto trait

* docs: Change from line protocol data model to InfluxDB data model

* refactor: rename LP --> Influx in code

* feat: add support for UInteger type

Co-authored-by: Edd Robinson <me@edd.io>
2021-01-15 17:29:30 -05:00
Carol (Nichols || Goulding) 813092649d fix: Make file behave the same as other object stores with paths 2021-01-15 10:25:05 -05:00
Dom a2c0434554
Merge branch 'main' into dom/iox-api-client 2021-01-14 14:35:17 +00:00
Paul Dix 9377dc1943 chore: address pr feedback 2021-01-13 18:15:35 -05:00
Paul Dix 3b1f045f44 feat: add serialization to wal buffer segments
Adds serialization with compression and checksum for WAL buffer segments.

This required a weird structure where the flatbuffer bytes of ReplicatedWrite were kept as a raw payload. I did this because otherwise each of the replicated writes would have been rebuilt in the segment.

The other thing that isn't ideal is that deserializing a segment actually marshals it into a Rust struct as opposed to keeping the entire thing as raw flatbuffers. We could update this later to have a concept of an open segment (regular rust stuct) and closed segments that are just the flatbuffers.
2021-01-13 18:15:34 -05:00
Dom 5647bfeb6f refactor: error handling & typed errors
Refactors the API method errors.

The user of the API client needs to be able to distinguish between various error
states when an API request fails. The most ergonomic way of exposing this
information is by returning an error enum that is specific to each API method
(or at least the important ones with well defined failure modes) - currently
only the `create_database()` method has significant error states, so this is
the only one with a specific error type in this impl.

This change defines a bunch of API error codes in the API client, adds them to
the IOx API error response body, and maps them in the API client. Due to error
wrapping the error code mapping in the IOx server is less exhaustive than I had
hoped however.
2021-01-13 17:32:12 +00:00