Commit Graph

138 Commits (706115178b080c674e342c8c9d209eafb92ea381)

Author SHA1 Message Date
Paul Dix 706115178b
chore: remove cruft from config/database rules (#935)
The replication, query, and subscription concepts here are going to be signficiantly different. Thought it would be best to just remove this cruft for now to avoid confusion.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-09 13:48:59 +00:00
Andrew Lamb 500c237f62
feat: Add Service + CLI command to write line protocol to IOx (#939)
* feat: Add Service + CLI command to write line protocol to IOx

* fix: clippy

* fix: protobuf lint

* fix: LinePrtocol typo
2021-03-08 17:35:19 +00:00
Raphael Taylor-Davies 5c9dd68bf8
feat: MVP Management CLI (#907)
* feat: MVP CLI implementation

* feat: multiline database description
2021-03-03 17:37:55 +00:00
Raphael Taylor-Davies 51981c92f5
feat: implement gRPC API and migrate influxdb_iox_client to use it (#853)
* feat: implement gRPC management API

* feat: migrate influxdb_iox_client to use gRPC API

* fix: review comments

* refactor: separate influxdb_iox_client error types

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-02 17:51:46 +00:00
Raphael Taylor-Davies 84764002a2
refactor: use structopt for CLI root (#892)
* refactor: use structopt for CLI root

* refactor: PR feedback
2021-03-01 18:45:14 +00:00
Andrew Lamb eca92b69da
feat: implement format parameter on sql query endpoint (#870)
* feat: implement format parameter on sql query endpoint

* feat: include Content-Type header

* fix: return proper JSON

* test: add tests for JsonArrayWriter

* refactor: remove use of tempfile

* fix: clippy / fmt
2021-02-26 11:43:03 +00:00
Raphael Taylor-Davies ffc20fa821
feat: add basic gRPC health service (#862)
* feat: add basic gRPC health service

* feat: update README.md add /health to HTTP API

* feat: add health client to influxdb_iox_client

feat: end-to-end test health check service

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-25 13:24:12 +00:00
Raphael Taylor-Davies 042c61908e
refactor: move gRPC storage into own module (#814)
* refactor: move gRPC storage into own module

refactor: encpasulate grpc server creation within modules

refactor: pass concrete Server<M> to rpc::make_server

* refactor: review comments

* refactor: cargo fmt

Co-authored-by: Edd Robinson <me@edd.io>
2021-02-16 12:26:20 +00:00
Raphael Taylor-Davies 6d3c21b952 feat: Use Bytes instead of Vec<u8> in prost generated code
Also add google.rpc.* Protobuf definitions
2021-02-12 17:10:58 +00:00
Marko Mikulicic c61923f05e fix: Avoid clashing ports when running e2e tests
Closes #745
2021-02-11 17:02:44 +00:00
kodiakhq[bot] a242fc39aa
Merge branch 'main' into jg/flight-client 2021-02-11 14:29:54 +00:00
Paul Dix 4c43c311f0
feat: Add database rules configuration for mutable buffer (#779)
* feat: Add database rules configuration for mutable buffer

* refactor: change all database rules usage to use mutable_buffer_config rather than store_locally
2021-02-10 18:36:29 -05:00
Carol (Nichols || Goulding) afdb0c8274 fix: Return better errors instead of unwrapping 2021-02-10 10:30:58 -05:00
Jake Goulding 699f4a577f feat: Add an optional Flight client to the IOx client library 2021-02-10 10:30:05 -05:00
Carol (Nichols || Goulding) 1f1ebefd7a fix: Switch to database name instead of org/bucket for the Flight API 2021-02-03 10:02:06 -05:00
Jake Goulding 39338b195b feat: Basic scaffolding of a flight server do_get 2021-02-03 09:56:56 -05:00
Carol (Nichols || Goulding) deaeff0173 refactor: Extract smaller functions for grpc tests
This makes it easier to see what's being tested and zero in on what
you're interested in rather than needing to scroll and scan through a
lot of code in the main test.
2021-01-29 08:34:28 -05:00
Carol (Nichols || Goulding) 8ade506a0c refactor: Extract independent test helper functions to separate files 2021-01-29 08:34:28 -05:00
Carol (Nichols || Goulding) 41e0fff039 refactor: Extract gRPC read group and read window agg tests because they manage their own data 2021-01-29 08:34:28 -05:00
Carol (Nichols || Goulding) cfe18e3e61 refactor: Extract a method to create gRPC ReadSource to the Scenario 2021-01-29 08:34:28 -05:00
Carol (Nichols || Goulding) bfe779d7b8 refactor: Incorporate ns_since_epoch into the Scenario
I want this to be an implementation detail of the tests, not something
you have to worry about when wondering what this test is hoping to test.
2021-01-29 08:34:28 -05:00
Carol (Nichols || Goulding) 05e79ed343 refactor: Make getter methods on Scenario for dependent values 2021-01-29 08:34:28 -05:00
Carol (Nichols || Goulding) 6ba27152e3 refactor: Change Scenario to use the builder pattern 2021-01-29 08:34:28 -05:00
Carol (Nichols || Goulding) 352b7ec829 refactor: Extract a Scenario struct to group related setup together 2021-01-29 08:34:28 -05:00
Carol (Nichols || Goulding) daa8b7ef0a refactor: Extract more helper functions 2021-01-29 08:34:28 -05:00
Carol (Nichols || Goulding) c4be42324b test: Remove code testing unused disk-based WAL restoration
Restarting the test server and testing for WAL restoration on startup
(and possible error scenarios) was disabled in 70ff426, which also noted
that significant refactoring will likely be needed when loading info on
startup is re-enabled. With that in mind, remove this code creating an
invalid dir in the WAL dir.
2021-01-29 08:34:28 -05:00
Carol (Nichols || Goulding) 15389d5665 refactor: Improve the names of client and client2 2021-01-29 08:34:28 -05:00
Carol (Nichols || Goulding) 422cbdb3b2 refactor: Move some helper functions after the main test 2021-01-29 08:34:28 -05:00
Carol (Nichols || Goulding) 0d174a2e1f refactor: Use data_types fn to create database name 2021-01-29 08:34:28 -05:00
Paul Dix 45d25fcbcd feat: add loading of database rules on startup
This adds functionality to the server to load database rules on startup. Follow on work will update the rules to store additional data (the catalog) and ensure that updates to the catalog can occur as outlined in #651. This work also updated the configuration to not require a database directory so the server can run entirely in memory. I needed this to get the end-to-end test passing since the file object store API doesn't yet have the functionality needed. I've logged #688 to track adding that in.
2021-01-22 16:06:10 -05:00
Andrew Lamb 4b5204cc8f
fix: Update end-to-end.rs to have moe specific error messages (#674) 2021-01-19 15:54:13 -05:00
Andrew Lamb 71627120b9
refactor: consolidate line protocol schema creation into data_types and port code to use it (#663)
* refactor: consolidate line protocol schema creation into data_types, and port code to use it

refactor: Port mutable buffer to use SchemaBuilder

* fix: doctest

* refactor: remove unecessary clippyisms

* docs: Improve comments via suggestions from code review

Co-authored-by: Edd Robinson <me@edd.io>

* refactor: use more idomatic try_ naming and TryInto trait

* docs: Change from line protocol data model to InfluxDB data model

* refactor: rename LP --> Influx in code

* feat: add support for UInteger type

Co-authored-by: Edd Robinson <me@edd.io>
2021-01-15 17:29:30 -05:00
Dom 30d5857fd4 chore: fix lints 2021-01-13 18:07:19 +00:00
Dom bdc832d040 refactor: replace config system with structopt
Replaces the hand-rolled config system with a StructOpt managed config struct.

I've got most of it ported across, but the interaction between all the logging
config bits is complex! I've left what is there and hooked in the value from
the config struct (which directly replaces the env var in usage, as it also
sources from the env).
2021-01-11 18:43:14 +00:00
Paul Dix d17ef800c5
feat: add create and get database to API (#619)
* feat: add create and get database to API

This commit is start of the IOx specific API. It puts everything under /iox/api/v1 as this is the beginning of the IOx API. Creating a database is done with a PUT and a GET request can retrieve the DatabaseRules details.

* feat: add defaults for DatabaseRules for create_database

* feat: add create and get database to API

This commit is start of the IOx specific API. It puts everything under /iox/api/v1 as this is the beginning of the IOx API. Creating a database is done with a PUT and a GET request can retrieve the DatabaseRules details.
2021-01-07 12:25:37 -05:00
Andrew Lamb 9f0ff678f1
feat: Formalizes the config system for IOx, including tests (#608)
* feat: Create configuration system, port IOx to use it

* docs: Apply suggestions from code review

Co-authored-by: Paul Dix <paul@influxdata.com>

* fix: fix test for setting values

Co-authored-by: Paul Dix <paul@influxdata.com>
2020-12-31 07:02:31 -05:00
Andrew Lamb 163f34c27b
feat: report support for read_group in capabilities request (#586) 2020-12-22 10:06:48 -05:00
Andrew Lamb 263af1eeac
feat: implement read_group in the write_buffer (#583)
* feat: implement read_group in the write_buffer

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: rustfmt

* fix: adjust tests for min/max

* fix: Update write_buffer/src/table.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-12-22 09:03:20 -05:00
Paul Dix 70ff4267ce
chore: Connect server crate to http routes and server command (#569)
* chore: Connect server crate to http routes and server command

This updates the http_routes and main server to use the Server crate. This is the first step in a larger effort to start hooking up the initial IOx API and get things running end to end with in-memory database, WAL buffer, and object storage.

For the time being, this disables the previous disk based WAL. Or rather, it uses the WriteBufferDb without it. That means that this IOx server has no persistence until later. Because of this, the restart in the end-to-end was removed.

Later PRs will add the WAL buffer and restart logic that loads from object store. We can opt to bring the local disk based WAL back later, but it will likely require some refactoring to work with how the WAL Buffer will operate.
2020-12-17 18:48:41 -05:00
Andrew Lamb 699504a022
feat: implement read_group for Sum, Count and Mean aggregates (#557)
* feat: Implement read_group for "normal" aggregates

* test: add tests
2020-12-15 09:35:00 -05:00
Andrew Lamb 1740e26ec3
fix: do not produce gRPC series frames for fields that only contain null values (#558)
* test: add test for field columns with only nulls

* fix: do not produce series for null fields, tests for same

* fix: remove uneeded test printlns
2020-12-15 08:28:23 -05:00
Dom 6f473984d0 style: wrap comments
Runs rustfmt with the new config.
2020-12-11 18:22:26 +00:00
Andrew Lamb 50ba529cb8
test: Adds tests for read_group for None aggregates (#538) 2020-12-10 11:31:15 -05:00
Andrew Lamb 8cb8276819
fix: Update gRPC definitions so tag_key=_field requests work in IOx (#517)
* fix: Update gRPC definitions so tag_key=_field requests work in IOx

* docs: Update src/server/rpc.rs

* fix: fixup test

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: consistent type annotations

* fix: refactor redundant test code into test_helpers

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-12-02 13:58:48 -05:00
Dom c3a0e893ae test: use flate2 2020-12-01 11:01:10 +00:00
Andrew Lamb 20f421e9c6
fix: Do not send GroupFrames in response to read_window_aggregate (#497)
* fix: Do not send GroupFrames in response to read_window_aggregate

* fix: clippy and test
2020-11-30 05:59:05 -05:00
Andrew Lamb 0eaa90e89d
feat: Hook up read_window_aggregate into the write_buffer, end-to-end tests (#483)
* feat: read_window_aggregate_plans

* fix: clippy sacrifice

* fix: clippy

* fix: clippy
2020-11-25 10:20:49 -05:00
Andrew Lamb 85921fe401
feat: Implement gRPC and storage interface for read_group_aggregate (#474)
* feat: Implement gRPC and storage interface for read_group_aggregate

* fix: clippy

* docs: Tweak comments

* fix: moar clippy

* fix: fmt

* docs: Apply doc improvement suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* refactor: improve error creation

* refactor: use match instead of if

* refactor: clearer match

* refactor: clean up storage aggregate matching

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-11-24 13:41:11 -05:00
Andrew Lamb 87626a3635
feat: Update storage protobuf definitions, add stubs for read_window_aggregate (#444)
* feat: Update storage protobuf definitions, add stubs for read_window_aggregate

* refactor: Extract the features field in a clearer way

* docs: Add provenance information to service.proto
2020-11-12 07:07:42 -05:00
Andrew Lamb 659da9264a
chore: Update predicate protobuf definitions (#443) 2020-11-11 18:06:39 -05:00