6.4 KiB

Raw Blame History

Underground Guide to testing IOx locally

This document explains how to run IOx locally (for locally profiling, for example) similarly to how it is deployed in production but from source in your local development environment where you can run low key experiments.

This is an "underground" guide in the sense that it is not meant to define an official setup for profiling or benchmarking and should not be used for such. It is provided in the hope it will be helpful for developers.

Step 1: Build IOx

Build IOx for release with pprof:

cd influxdb_iox
cargo build --release --features=pprof

You can also install the influxdb_iox command locally via

cd influxdb_iox
cargo install --path influxdb_iox

Step 2: Start kafka and postgres

Now, start up kafka and postgres locally in docker containers:

# get rskafka from https://github.com/influxdata/rskafka
cd rskafka
# Run kafka on localhost:9010
docker-compose -f docker-compose-kafka.yml up &
# now run postgres
docker run -p 5432:5432 -e POSTGRES_HOST_AUTH_METHOD=trust postgres &

Of course, you can also use locally running services (if, for example, you have postgres running locally on port 5432).

Step 3: Do one time initialization setup

# initialize the catalog
INFLUXDB_IOX_WRITE_BUFFER_TYPE=kafka \
INFLUXDB_IOX_WRITE_BUFFER_ADDR=localhost:9010 \
INFLUXDB_IOX_WRITE_BUFFER_AUTO_CREATE_TOPICS=10 \
INFLUXDB_IOX_CATALOG_DSN=postgres://postgres@localhost:5432/postgres \
OBJECT_STORE=file \
DATABASE_DIRECTORY=~/data_dir \
LOG_FILTER=debug \
./target/release/influxdb_iox catalog setup

# initialize the kafka topic
INFLUXDB_IOX_WRITE_BUFFER_TYPE=kafka \
INFLUXDB_IOX_WRITE_BUFFER_ADDR=localhost:9010 \
INFLUXDB_IOX_WRITE_BUFFER_AUTO_CREATE_TOPICS=10 \
INFLUXDB_IOX_CATALOG_DSN=postgres://postgres@localhost:5432/postgres \
OBJECT_STORE=file \
DATABASE_DIRECTORY=~/data_dir \
LOG_FILTER=debug \
./target/release/influxdb_iox catalog topic update iox-shared

Inspecting Catalog and Kafka / Redpanda state

Depending on what you are trying to do, you may want to inspect the catalog and/or the contents of Kafka / Redpands.

You can run psql like this to inspect the catalog:

psql -h localhost -p 5432 -U postgres

postgres=# set search_path = iox_catalog;
SET
postgres=# \d
                         List of relations
   Schema    |             Name              |   Type   |  Owner
-------------+-------------------------------+----------+----------
 iox_catalog | _sqlx_migrations              | table    | postgres
 iox_catalog | column_name                   | table    | postgres
 iox_catalog | column_name_id_seq            | sequence | postgres
 iox_catalog | kafka_topic                   | table    | postgres
 iox_catalog | kafka_topic_id_seq            | sequence | postgres
 iox_catalog | namespace                     | table    | postgres
 iox_catalog | namespace_id_seq              | sequence | postgres
 iox_catalog | parquet_file                  | table    | postgres
 iox_catalog | parquet_file_id_seq           | sequence | postgres
 iox_catalog | partition                     | table    | postgres
 iox_catalog | partition_id_seq              | sequence | postgres
 iox_catalog | processed_tombstone           | table    | postgres
 iox_catalog | query_pool                    | table    | postgres
 iox_catalog | query_pool_id_seq             | sequence | postgres
 iox_catalog | sequencer                     | table    | postgres
 iox_catalog | sequencer_id_seq              | sequence | postgres
 iox_catalog | sharding_rule_override        | table    | postgres
 iox_catalog | sharding_rule_override_id_seq | sequence | postgres
 iox_catalog | table_name                    | table    | postgres
 iox_catalog | table_name_id_seq             | sequence | postgres
 iox_catalog | tombstone                     | table    | postgres
 iox_catalog | tombstone_id_seq              | sequence | postgres
(22 rows)

postgres=#

You can mess with redpanda using docker exec redpanda-0 rpk like this:

$ docker exec redpanda-0 rpk topic list
NAME        PARTITIONS  REPLICAS
iox-shared  1           1

Step 4: Run the services

Run Router on port 8080/8081 (http/grpc)

INFLUXDB_IOX_BIND_ADDR=localhost:8080 \
INFLUXDB_IOX_GRPC_BIND_ADDR=localhost:8081 \
INFLUXDB_IOX_WRITE_BUFFER_TYPE=kafka \
INFLUXDB_IOX_WRITE_BUFFER_ADDR=localhost:9010 \
INFLUXDB_IOX_WRITE_BUFFER_AUTO_CREATE_TOPICS=10 \
INFLUXDB_IOX_CATALOG_DSN=postgres://postgres@localhost:5432/postgres \
OBJECT_STORE=file \
DATABASE_DIRECTORY=~/data_dir \
LOG_FILTER=info \
./target/release/influxdb_iox run router

Run Ingester on port 8083/8083 (http/grpc)

INFLUXDB_IOX_BIND_ADDR=localhost:8083 \
INFLUXDB_IOX_GRPC_BIND_ADDR=localhost:8084 \
INFLUXDB_IOX_WRITE_BUFFER_TYPE=kafka \
INFLUXDB_IOX_WRITE_BUFFER_ADDR=localhost:9010 \
xINFLUXDB_IOX_WRITE_BUFFER_AUTO_CREATE_TOPICS=10 \
INFLUXDB_IOX_SHARD_INDEX_RANGE_START=0 \
INFLUXDB_IOX_SHARD_INDEX_RANGE_END=0 \
INFLUXDB_IOX_PAUSE_INGEST_SIZE_BYTES=5000000000 \
INFLUXDB_IOX_PERSIST_MEMORY_THRESHOLD_BYTES=4000000000 \
INFLUXDB_IOX_CATALOG_DSN=postgres://postgres@localhost:5432/postgres \
INFLUXDB_IOX_MAX_HTTP_REQUEST_SIZE=100000000 \
OBJECT_STORE=file \
DATABASE_DIRECTORY=~/data_dir \
LOG_FILTER=info \
./target/release/influxdb_iox run ingester

Step 5: Ingest data

You can load data using the influxdb_iox client:

influxdb_iox  --host=http://localhost:8080 -v write test_db test_fixtures/lineproto/*.lp

Now you can post data to http://localhost:8080 with your favorite load generating tool

My favorite is https://github.com/alamb/low_card

To run:

git clone git@github.com:alamb/low_card.git
cd low_card
cargo run --release

Then tweak the parameters in main.rs code to change the shape of the data. The default settings at the time of this writing would result in posting fairly large requests (necessitating the INFLUXDB_IOX_MAX_HTTP_REQUEST_SIZE setting above)

Step 6: Profile

See profiling.md.

Step 7: Clean up local state

If you find yourself needing to clean up postgres / kafka state use these commands:

docker ps -a -q | xargs docker stop
docker rm rskafka_proxy_1
docker rm rskafka_kafka-0_1
docker rm rskafka_kafka-1_1
docker rm rskafka_kafka-2_1
docker rm rskafka_zookeeper_1
docker volume rm  rskafka_kafka_0_data rskafka_kafka_1_data rskafka_kafka_2_data rskafka_zookeeper_data

6.4 KiB Raw Blame History