# Performance Tests

This tool starts a complete test environment:

- Kafka (docker)
- Minio (docker)
- Jaeger (docker)
- IOx router (local process)
- IOx writer (local process)
- test battery:
  - generate data with iox_data_generator
  - query data and benchmark

Logs live in `perf/logs`.
As long as perf.py is running, this works: `tail -f logs/iox_router.log`
After perf.py exits, log files are closed.
When perf.py is run again, old log files are deleted.

Persistence volumes live in `perf/volumes`.
Similar to log files, these data remain after perf.py exits, and are deleted when perf.py is run again.

## Test Batteries

A test battery is composed of:
- a directory, named for the battery name
  - data generator spec, in file `datagen.toml`
  - query tests, in file `queries.toml`
    - SQL query
    - (optional) name of query test
    - (optional) expected results, as a string or as a file

The data generator spec format is that defined by `iox_data_generator`.
[Read about that here](../iox_data_generator/README.md).

The query tests file looks like this:
```toml
[[queries]]
name = "example query, no expected result"
sql = "select count(*) from cpu"

[[queries]]
name = "example query, expected result in string"
sql = "select count(*) from cpu"
expect = """
COUNT(Uint8(1))
3
"""

[[queries]]
name = "example query, expected result in file"
sql = "select count(*) from cpu"
expect_filename = "foo.csv"
```

## Usage

Help:
```console
perf/perf.py --help
```

Run all test batteries:
```console
perf/perf.py all
```

Run test batteries `battery-0` and `battery-1`:
```console
perf/perf.py battery-0 battery-1
```

Keep all processes running after test batteries have completed:
```console
perf/perf.py battery-0 --hold
```

Do not run any tests, just create an empty playground environment:
```console
perf/perf.py --hold
```

Do not build IOx:
```console
perf/perf.py --skip-build
```

Use debug binaries (`target/debug`) rather than release binaries:
```console
perf/perf.py --debug
```

Use Kafka/Zookeeper instead of Redpanda:
```console
perf/perf.py --kafka-zookeeper
```

Use in-memory object store implementation, instead of S3/Minio:
```console
perf/perf.py --object-store memory
```

Use file object store implementation, instead of S3/Minio:
```console
perf/perf.py --object-store file
```

Just delete docker containers and network, then exit.
In the future, this will also detect orphaned IOx processes generated by perf, and delete those too:
```console
perf/perf.py --cleanup
```

## Install

Install Docker:
https://www.docker.com/products/docker-desktop

Install python3:

```console
brew install python3
```

or:

```console
apt install python3 python3-pip
```

Install the required Python packages:

```console
python3 -m pip install -r perf/requirements.txt
```