History

Dom Dwyer b07f15bec7 refactor: parallel column resolution A quick change to perform the ColumnRepo::create_or_get() calls in parallel (up to a maximum of 3 in-flight at any one time) in order to mitigate the latency of the call and reduce the overall schema validation call duration. The in-flight limit is enforced to avoid starving the DB connection pool of connections.		2022-02-24 21:04:25 +00:00
..
migrations	feat: Add row count to the parquet_file record in the catalog (#3847 )	2022-02-24 15:20:50 +00:00
src	refactor: parallel column resolution	2022-02-24 21:04:25 +00:00
Cargo.toml	feat: catalog instrumentation decorator	2022-02-23 14:34:26 +00:00
README.md	docs: Improve iox_catalog testing docs (#3760 )	2022-02-16 10:23:53 +00:00
build.rs	feat: allow IOx catalog to setup itself (no SQLx CLI required) (#3584 )	2022-01-31 15:07:38 +00:00

README.md

IOx Catalog

This crate contains the code for the IOx Catalog. This includes the definitions of namespaces, their tables, the columns of those tables and their types, what Parquet files are in object storage and delete tombstones. There's also some configuration information that the overal distributed system uses for operation.

To run this crate's tests you'll need Postgres installed and running locally. You'll also need to set the DATABASE_URL environment variable so that sqlx will be able to connect to your local DB. For example with user and password filled in:

DATABASE_URL=postgres://<postgres user>:<postgres password>@localhost/iox_shared

You can omit the host part if your postgres is running on the default unix domain socket (useful on macos because, by default, the config installed by brew install postgres doesn't listen to a TCP port):

DATABASE_URL=postgres:///iox_shared

You'll then need to create the database. You can do this via the sqlx command line.

cargo install sqlx-cli
sqlx database setup

This will set up the database based on the files in ./migrations in this crate. SQLx also creates a table to keep track of which migrations have been run.

Tests

To run the Postgres integration tests, ensure the above setup is complete first.

Set DATABASE_URL=<dsn> env (see above)
Set TEST_INTEGRATION=1
Run cargo test

CAUTION: existing data in the database is dropped when tests are run

Schema namespace

All iox catalog tables are created in a iox_catalog schema. Remember to set the schema search path when accessing the database with psql.

There are several ways to set the default search path, depending if you want to do it for your session, for the database or for the user.

Setting a default search path for the database or user may interfere with tests (e.g. it may make some test pass when they should fail). The safest option is set the search path on a per session basis. As always, there are a few ways to do that:

you can type set search_path to public,iox_catalog; inside psql.
you can add (1) to your ~/.psqlrc
or you can just pass it as a CLI argument with:

psql 'dbname=iox_shared options=-csearch_path=public,iox_catalog'