Commit Graph

478 Commits (196c589ef64f73677eb3e89e60b219f862bde19a)

Author SHA1 Message Date
Marco Neumann 74c251febb
feat: allow IOx catalog to setup itself (no SQLx CLI required) (#3584)
* feat: allow IOx catalog to setup itself (no SQLx CLI required)

* refactor: use SQLx macro instead of hand-rolled build script
2022-01-31 15:07:38 +00:00
Dom 32d7c4cbfe
refactor: remove InfluxColumnType::IOx (#3565)
* refactor: remove InfluxColumnType::IOx

Remove unused column variant - see #3554 for context.

* refactor: reserve SEMANTIC_TYPE_IOX name in proto

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 21:15:36 +00:00
Dom 9201023ea4
feat: schema validation for MutableBuffer instances (#3554)
* refactor: Debug bounds on Catalog trait

* feat: validate MutableBatch schema

Changes the schema validation code to validate MutableBatch instances
(coming from a pre-parsed LP write, and non-LP-based writes) instead of
parsed LP lines.

* refactor: Send bound on boxed errors

* refactor: clippy

Allow assert_eq!(bool, bool) for readability.

* refactor: no PartialEq<MB Column> for ColumnSchema

Remove the PartialEq<mutable_buffer::Column> for ColumnSchema - it's
definitely more readable as a method call.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 20:55:18 +00:00
Dom ce568ab447
build(iox_catalog): remove unused dependencies (#3552)
* build: don't pull in all of tokio

We already specify the tokio features we need so "full" (all features)
is not necessary.

* build: remove chrono dependency

Appears unused.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 15:14:42 +00:00
Paul Dix bb893510a0 feat: Add scaffolding for ingester server
* Adds a new ingester command to start an ingester server
* Moves previous ingester server over to handler
* Skeleton for gRPC and HTTP handlers
2022-01-21 18:02:19 -05:00
Paul Dix bfa54033bd refactor: Clean up the Catalog API
This updates the catalog API to make it easier to work with for consumers. I also found a bug in the MemCatalog implementation while refactoring the tests to work with the new API definition. Consumers will now be able to Arc wrap the catalog and use it across awaits.
2022-01-21 16:01:13 -05:00
Paul Dix bfc085c20d feat: add get kafka_topic by name to catalog 2022-01-19 17:32:23 -05:00
Paul Dix 860e5a30ca refactor: update ingester to get sequencer record and not attempt to create 2022-01-19 17:15:10 -05:00
Paul Dix 172d75c6d7 feat: add sequencer get_by_kafka_topic_id_and_partition to catalog 2022-01-19 16:45:06 -05:00
Paul Dix 28db06297f fix: clear postgres schema in test wasn't deleting parquet_file 2022-01-19 16:30:54 -05:00
Paul Dix 41038721e1 feat: Add parquet file records to iox_catalog
* Adds ParquetFile and scaffolding to IOx catalog
* Changed the file_location in parquet_file to object_store_id which is a uuid
2022-01-19 14:14:54 -05:00
Paul Dix f36d66deb7 feat: Add Tombstone to Catalog
* Adds TombstoneId and Tombstone to the iox_catalog with associated interfaces
* Adds SequenceNumber new type for use with Tombstone
* Adds Timestamp new type for use with Tombstone
* Adds constraint to the Postgres schema to enforce tombstone uniqueness by table_id, sequencer_id, and sequence_number
2022-01-18 18:17:21 -05:00
Paul Dix 8067316c33 fix: typo in partitionid description 2022-01-18 17:33:05 -05:00
Paul Dix b1510675ae refactor: add new type for Kafka Partition in Catalog 2022-01-18 15:13:18 -05:00
Paul Dix e395ef7066 feat: add Partition scaffolding to iox_catalog 2022-01-18 14:50:49 -05:00
Paul Dix 40cac21e21 refactor: change all ids in catalog to their own types 2022-01-18 12:42:10 -05:00
Paul Dix 5e464727d1 refactor: make get_schema_by_name bare function 2022-01-18 10:09:16 -05:00
Paul Dix 4f87600081 chore: make iox_catalog tests generic for any backend implementation 2022-01-17 17:20:11 -05:00
Paul Dix dfe95e1a56 refactor: make postgres and mem catalog implementations public 2022-01-17 14:46:10 -05:00
Paul Dix b3ee1032b3 feat: add memory based catalog
Adds a memory based catalog, useful for testing purposes.
Separates getting the namespace schema from the namespace and moves the schema code out interface out of postgres.
2022-01-17 14:09:10 -05:00
Paul Dix d4d4b4c087 fix: workspace hack after adding iox_catalog crate 2022-01-17 10:04:24 -05:00
Paul Dix b796d5e2d1 fix: query pool type and sequencer create 2022-01-17 10:00:33 -05:00
Paul Dix 1f4469b5a0 feat: Create iox_catalog interface with traits
This creates traits for the catalog API and moves the data objects over to interface.rs.
Updates the postgres module to implement the trait API.
Moves schema vaildation and creation out to the primary lib using the trait API.
Adds setup function to create shared kafka topic, query pool, and sequencer records.
2022-01-17 10:00:33 -05:00
Dom aa6f118487 feat: iox_catalog sequencers (#3465)
* refactor: ensure sequencers are unique

Adds a unique constraint to ensure only one sequencer record exists for
each Kafka (topic, partition).

* test: use DSN from env for integration tests

Removes the hard-coded DSN, instead sourcing it from the DATABASE_URL
environment variable.

* docs: integration testing for iox_catalog

Documents the required steps in order to run the Postgres integration
tests for the iox_catalog crate.

* feat(iox_catalog): create & list sequencers

Adds support for interacting with the "sequencer" table.

* chore: update lockfile

Running cargo in iox_catalog generates a lockfile diff.
2022-01-17 10:00:31 -05:00
Paul Dix 4f908021e4 chore: Update iox_catalog Postgres test to only run on integration 2022-01-17 09:56:20 -05:00
Paul Dix 8d6d9e679f refactor: update iox_catalog
Changed to use the iox_catalog schema in Postgres rather than public.
Updated talbe names to be singular.
Removed the connection_string from query_pool
2022-01-17 09:56:20 -05:00
Paul Dix 4e5bca93fb chore: update iox_catalog to Snafu 0.7 2022-01-17 09:56:20 -05:00
Paul Dix 4764e71c54 feat: Add initial iox_catalog skeleton 2022-01-17 09:56:20 -05:00