201 lines
6.4 KiB
Markdown
201 lines
6.4 KiB
Markdown
# Underground Guide to testing IOx locally
|
|
|
|
This document explains how to run IOx locally (for locally
|
|
profiling, for example) similarly to how it is deployed in production
|
|
but from source in your local development environment where you can
|
|
run low key experiments.
|
|
|
|
This is an "underground" guide in the sense that it is not meant to
|
|
define an official setup for profiling or benchmarking and should not
|
|
be used for such. It is provided in the hope it will be helpful for
|
|
developers.
|
|
|
|
## Step 1: Build IOx
|
|
|
|
Build IOx for release with pprof:
|
|
|
|
```shell
|
|
cd influxdb_iox
|
|
cargo build --release --features=pprof
|
|
```
|
|
|
|
You can also install the `influxdb_iox` command locally via
|
|
|
|
```shell
|
|
cd influxdb_iox
|
|
cargo install --path influxdb_iox
|
|
```
|
|
|
|
## Step 2: Start kafka and postgres
|
|
|
|
Now, start up kafka and postgres locally in docker containers:
|
|
```shell
|
|
# get rskafka from https://github.com/influxdata/rskafka
|
|
cd rskafka
|
|
# Run kafka on localhost:9010
|
|
docker-compose -f docker-compose-kafka.yml up &
|
|
# now run postgres
|
|
docker run -p 5432:5432 -e POSTGRES_HOST_AUTH_METHOD=trust postgres &
|
|
```
|
|
|
|
Of course, you can also use locally running services (if, for example,
|
|
you have postgres running locally on port 5432).
|
|
|
|
|
|
# Step 3: Do one time initialization setup
|
|
|
|
|
|
```shell
|
|
# initialize the catalog
|
|
INFLUXDB_IOX_WRITE_BUFFER_TYPE=kafka \
|
|
INFLUXDB_IOX_WRITE_BUFFER_ADDR=localhost:9010 \
|
|
INFLUXDB_IOX_WRITE_BUFFER_AUTO_CREATE_TOPICS=10 \
|
|
INFLUXDB_IOX_CATALOG_DSN=postgres://postgres@localhost:5432/postgres \
|
|
OBJECT_STORE=file \
|
|
DATABASE_DIRECTORY=~/data_dir \
|
|
LOG_FILTER=debug \
|
|
./target/release/influxdb_iox catalog setup
|
|
|
|
# initialize the kafka topic
|
|
INFLUXDB_IOX_WRITE_BUFFER_TYPE=kafka \
|
|
INFLUXDB_IOX_WRITE_BUFFER_ADDR=localhost:9010 \
|
|
INFLUXDB_IOX_WRITE_BUFFER_AUTO_CREATE_TOPICS=10 \
|
|
INFLUXDB_IOX_CATALOG_DSN=postgres://postgres@localhost:5432/postgres \
|
|
OBJECT_STORE=file \
|
|
DATABASE_DIRECTORY=~/data_dir \
|
|
LOG_FILTER=debug \
|
|
./target/release/influxdb_iox catalog topic update iox-shared
|
|
```
|
|
|
|
## Inspecting Catalog and Kafka / Redpanda state
|
|
|
|
Depending on what you are trying to do, you may want to inspect the
|
|
catalog and/or the contents of Kafka / Redpands.
|
|
|
|
You can run psql like this to inspect the catalog:
|
|
```shell
|
|
psql -h localhost -p 5432 -U postgres
|
|
```
|
|
|
|
```sql
|
|
postgres=# set search_path = iox_catalog;
|
|
SET
|
|
postgres=# \d
|
|
List of relations
|
|
Schema | Name | Type | Owner
|
|
-------------+-------------------------------+----------+----------
|
|
iox_catalog | _sqlx_migrations | table | postgres
|
|
iox_catalog | column_name | table | postgres
|
|
iox_catalog | column_name_id_seq | sequence | postgres
|
|
iox_catalog | kafka_topic | table | postgres
|
|
iox_catalog | kafka_topic_id_seq | sequence | postgres
|
|
iox_catalog | namespace | table | postgres
|
|
iox_catalog | namespace_id_seq | sequence | postgres
|
|
iox_catalog | parquet_file | table | postgres
|
|
iox_catalog | parquet_file_id_seq | sequence | postgres
|
|
iox_catalog | partition | table | postgres
|
|
iox_catalog | partition_id_seq | sequence | postgres
|
|
iox_catalog | processed_tombstone | table | postgres
|
|
iox_catalog | query_pool | table | postgres
|
|
iox_catalog | query_pool_id_seq | sequence | postgres
|
|
iox_catalog | sequencer | table | postgres
|
|
iox_catalog | sequencer_id_seq | sequence | postgres
|
|
iox_catalog | sharding_rule_override | table | postgres
|
|
iox_catalog | sharding_rule_override_id_seq | sequence | postgres
|
|
iox_catalog | table_name | table | postgres
|
|
iox_catalog | table_name_id_seq | sequence | postgres
|
|
iox_catalog | tombstone | table | postgres
|
|
iox_catalog | tombstone_id_seq | sequence | postgres
|
|
(22 rows)
|
|
|
|
postgres=#
|
|
```
|
|
|
|
You can mess with redpanda using `docker exec redpanda-0 rpk` like this:
|
|
|
|
```shell
|
|
$ docker exec redpanda-0 rpk topic list
|
|
NAME PARTITIONS REPLICAS
|
|
iox-shared 1 1
|
|
```
|
|
|
|
|
|
# Step 4: Run the services
|
|
|
|
## Run Router on port 8080/8081 (http/grpc)
|
|
```shell
|
|
INFLUXDB_IOX_BIND_ADDR=localhost:8080 \
|
|
INFLUXDB_IOX_GRPC_BIND_ADDR=localhost:8081 \
|
|
INFLUXDB_IOX_WRITE_BUFFER_TYPE=kafka \
|
|
INFLUXDB_IOX_WRITE_BUFFER_ADDR=localhost:9010 \
|
|
INFLUXDB_IOX_WRITE_BUFFER_AUTO_CREATE_TOPICS=10 \
|
|
INFLUXDB_IOX_CATALOG_DSN=postgres://postgres@localhost:5432/postgres \
|
|
OBJECT_STORE=file \
|
|
DATABASE_DIRECTORY=~/data_dir \
|
|
LOG_FILTER=info \
|
|
./target/release/influxdb_iox run router
|
|
```
|
|
|
|
|
|
## Run Ingester on port 8083/8083 (http/grpc)
|
|
```shell
|
|
INFLUXDB_IOX_BIND_ADDR=localhost:8083 \
|
|
INFLUXDB_IOX_GRPC_BIND_ADDR=localhost:8084 \
|
|
INFLUXDB_IOX_WRITE_BUFFER_TYPE=kafka \
|
|
INFLUXDB_IOX_WRITE_BUFFER_ADDR=localhost:9010 \
|
|
xINFLUXDB_IOX_WRITE_BUFFER_AUTO_CREATE_TOPICS=10 \
|
|
INFLUXDB_IOX_SHARD_INDEX_RANGE_START=0 \
|
|
INFLUXDB_IOX_SHARD_INDEX_RANGE_END=0 \
|
|
INFLUXDB_IOX_PAUSE_INGEST_SIZE_BYTES=5000000000 \
|
|
INFLUXDB_IOX_PERSIST_MEMORY_THRESHOLD_BYTES=4000000000 \
|
|
INFLUXDB_IOX_CATALOG_DSN=postgres://postgres@localhost:5432/postgres \
|
|
INFLUXDB_IOX_MAX_HTTP_REQUEST_SIZE=100000000 \
|
|
OBJECT_STORE=file \
|
|
DATABASE_DIRECTORY=~/data_dir \
|
|
LOG_FILTER=info \
|
|
./target/release/influxdb_iox run ingester
|
|
```
|
|
|
|
|
|
# Step 5: Ingest data
|
|
|
|
You can load data using the influxdb_iox client:
|
|
```shell
|
|
influxdb_iox --host=http://localhost:8080 -v write test_db test_fixtures/lineproto/*.lp
|
|
```
|
|
|
|
Now you can post data to `http://localhost:8080` with your favorite load generating tool
|
|
|
|
My favorite is https://github.com/alamb/low_card
|
|
|
|
To run:
|
|
```shell
|
|
git clone git@github.com:alamb/low_card.git
|
|
cd low_card
|
|
cargo run --release
|
|
```
|
|
|
|
Then tweak the parameters in `main.rs` code to change the shape of the
|
|
data. The default settings at the time of this writing would result in
|
|
posting fairly large requests (necessitating the
|
|
`INFLUXDB_IOX_MAX_HTTP_REQUEST_SIZE` setting above)
|
|
|
|
|
|
# Step 6: Profile
|
|
|
|
See [`profiling.md`](./profiling.md).
|
|
|
|
|
|
# Step 7: Clean up local state
|
|
|
|
If you find yourself needing to clean up postgres / kafka state use these commands:
|
|
```shell
|
|
docker ps -a -q | xargs docker stop
|
|
docker rm rskafka_proxy_1
|
|
docker rm rskafka_kafka-0_1
|
|
docker rm rskafka_kafka-1_1
|
|
docker rm rskafka_kafka-2_1
|
|
docker rm rskafka_zookeeper_1
|
|
docker volume rm rskafka_kafka_0_data rskafka_kafka_1_data rskafka_kafka_2_data rskafka_zookeeper_data
|
|
```
|