135 lines
2.7 KiB
Markdown
135 lines
2.7 KiB
Markdown
|
# Performance Tests
|
||
|
|
|
||
|
|
This tool starts a complete test environment:
|
||
|
|
|
||
|
|
- Kafka (docker)
|
||
|
|
- Minio (docker)
|
||
|
|
- Jaeger (docker)
|
||
|
|
- IOx router (local process)
|
||
|
|
- IOx writer (local process)
|
||
|
|
- test battery:
|
||
|
|
- generate data with iox_data_generator
|
||
|
|
- query data and benchmark
|
||
|
|
|
||
|
|
Logs live in `perf/logs`.
|
||
|
|
As long as perf.py is running, this works: `tail -f logs/iox_router.log`
|
||
|
|
After perf.py exits, log files are closed.
|
||
|
|
When perf.py is run again, old log files are deleted.
|
||
|
|
|
||
|
|
Persistence volumes live in `perf/volumes`.
|
||
|
|
Similar to log files, these data remain after perf.py exits, and are deleted when perf.py is run again.
|
||
|
|
|
||
|
|
## Test Batteries
|
||
|
|
|
||
|
|
A test battery is composed of:
|
||
|
|
- a directory, named for the battery name
|
||
|
|
- data generator spec, in file `datagen.toml`
|
||
|
|
- query tests, in file `queries.toml`
|
||
|
|
- SQL query
|
||
|
|
- (optional) name of query test
|
||
|
|
- (optional) expected results, as a string or as a file
|
||
|
|
|
||
|
|
The data generator spec format is that defined by `iox_data_generator`.
|
||
|
|
[Read about that here](../iox_data_generator/README.md).
|
||
|
|
|
||
|
|
The query tests file looks like this:
|
||
|
|
```toml
|
||
|
|
[[queries]]
|
||
|
|
name = "example query, no expected result"
|
||
|
|
sql = "select count(*) from cpu"
|
||
|
|
|
||
|
|
[[queries]]
|
||
|
|
name = "example query, expected result in string"
|
||
|
|
sql = "select count(*) from cpu"
|
||
|
|
expect = """
|
||
|
|
COUNT(Uint8(1))
|
||
|
|
3
|
||
|
|
"""
|
||
|
|
|
||
|
|
[[queries]]
|
||
|
|
name = "example query, expected result in file"
|
||
|
|
sql = "select count(*) from cpu"
|
||
|
|
expect_filename = "foo.csv"
|
||
|
|
```
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
Help:
|
||
|
|
```console
|
||
|
|
perf/perf.py --help
|
||
|
|
```
|
||
|
|
|
||
|
|
Run all test batteries:
|
||
|
|
```console
|
||
|
|
perf/perf.py all
|
||
|
|
```
|
||
|
|
|
||
|
|
Run test batteries `battery-0` and `battery-1`:
|
||
|
|
```console
|
||
|
|
perf/perf.py battery-0 battery-1
|
||
|
|
```
|
||
|
|
|
||
|
|
Keep all processes running after test batteries have completed:
|
||
|
|
```console
|
||
|
|
perf/perf.py battery-0 --hold
|
||
|
|
```
|
||
|
|
|
||
|
|
Do not run any tests, just create an empty playground environment:
|
||
|
|
```console
|
||
|
|
perf/perf.py --hold
|
||
|
|
```
|
||
|
|
|
||
|
|
Do not build IOx:
|
||
|
|
```console
|
||
|
|
perf/perf.py --skip-build
|
||
|
|
```
|
||
|
|
|
||
|
|
Use debug binaries (`target/debug`) rather than release binaries:
|
||
|
|
```console
|
||
|
|
perf/perf.py --debug
|
||
|
|
```
|
||
|
|
|
||
|
|
Use Kafka/Zookeeper instead of Redpanda:
|
||
|
|
```console
|
||
|
|
perf/perf.py --kafka-zookeeper
|
||
|
|
```
|
||
|
|
|
||
|
|
Use in-memory object store implementation, instead of S3/Minio:
|
||
|
|
```console
|
||
|
|
perf/perf.py --object-store memory
|
||
|
|
```
|
||
|
|
|
||
|
|
Use file object store implementation, instead of S3/Minio:
|
||
|
|
```console
|
||
|
|
perf/perf.py --object-store file
|
||
|
|
```
|
||
|
|
|
||
|
|
Just delete docker containers and network, then exit.
|
||
|
|
In the future, this will also detect orphaned IOx processes generated by perf, and delete those too:
|
||
|
|
```console
|
||
|
|
perf/perf.py --cleanup
|
||
|
|
```
|
||
|
|
|
||
|
|
## Install
|
||
|
|
|
||
|
|
Install Docker:
|
||
|
|
https://www.docker.com/products/docker-desktop
|
||
|
|
|
||
|
|
Install python3:
|
||
|
|
|
||
|
|
```console
|
||
|
|
brew install python3
|
||
|
|
```
|
||
|
|
|
||
|
|
or:
|
||
|
|
|
||
|
|
```console
|
||
|
|
apt install python3 python3-pip
|
||
|
|
```
|
||
|
|
|
||
|
|
Install the required Python packages:
|
||
|
|
|
||
|
|
```console
|
||
|
|
python3 -m pip install -r perf/requirements.txt
|
||
|
|
```
|