Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Merge pull request #6264 from influxdata/jz-stress-readme 

Move stress/README to stress/DESIGN and add a stress/README
pull/6263/head
Jack Zampolin 2016-04-08 09:36:12 -07:00
parent f232c05486 16d1f14d11
commit 94ceb736b6
2 changed files with 152 additions and 37 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

47
stress/DESIGN.md Normal file
View File

@ -0,0 +1,47 @@
## Stress Test
The logic for `StressTest` can be found in `stress/run.go`.
A new `StressTest` type was added and is composed four different parts. The `StressTest` type has one method `Start(wHandle responseHandler, rHandle responseHandler)`. This method starts the stress test.
A `responseHandler` is a function with type signature `func(r <-chan response, t *Timer)`. Response Handlers handle the read and write responses respectively.
### Provisioner
Provisions the InfluxDB instance where the stress test is going to be ran against.
Think things like, creating the database, setting up retention policies, continuous queries, etc.
### Writer
The `Writer` is responsible for Writing data into an InfluxDB instance. It has two components: `PointGenerator` and `InfluxClient`.
##### PointGenerator
The `PointGenerator` is responsible for generating points that will be written into InfluxDB. Additionally, it is reponsible for keeping track of the latest timestamp of the points it is writing (Just incase the its needed by the `Reader`).
Any type that implements the methods `Generate()` and `Time()` is a `PointGenerator`.
##### InfluxClient
The `InfluxClient` is responsible for writing the data that is generated by the `PointGenerator`.
Any type that implements `Batch(ps <-chan Point, r chan<- response)`, and `send(b []byte) response` is an `InfluxClient`.
### Reader
The `Reader` is responsible for querying the database. It has two components: `QueryGenerator` and `QueryClient`.
##### QueryGenerator
The `QueryGenerator` is responsible for generating queries.
##### QueryClient
The `QueryClient` is responsible for executing queries against an InfluxDB instance.
## Basic
`basic.go` implements an each of the components of a stress test.
## Util
`util.go` contains utility methods used throughout the package.
## Config
`config.go` contains the logic for managing the configuration of the stress test.
A sample configuration file can be found in `stress/stress.toml`. This still needs work, but whats there now is good enough IMO.
## Template
`template.go` contains the logic for a basic stress test.

142
stress/README.md
View File

@ -1,47 +1,115 @@
## Stress Test
The logic for `StressTest` can be found in `stress/run.go`.
# `influx_stress` usage and configuration
A new `StressTest` type was added and is composed four different parts. The `StressTest` type has one method `Start(wHandle responseHandler, rHandle responseHandler)`. This method starts the stress test.
The binary for `influx_stress` comes bundled with all influx installations.
To run it against an `influxd` instance located at `localhost:8086` with the default configuration options:
A `responseHandler` is a function with type signature `func(r <-chan response, t *Timer)`. Response Handlers handle the read and write responses respectively.
See more about the [default configuration options](https://github.com/influxdata/influxdb/blob/master/stress/stress.toml)
### Provisioner
Provisions the InfluxDB instance where the stress test is going to be ran against.
```bash
$ influx_stress
```
Think things like, creating the database, setting up retention policies, continuous queries, etc.
To run `influx_stress` with a configuration file:
```bash
$ influx_stress -config my_awesome_test.toml
```
### Writer
The `Writer` is responsible for Writing data into an InfluxDB instance. It has two components: `PointGenerator` and `InfluxClient`.
To daemonize `influx_stress` and save the output to a results file:
```bash
$ influx_stress -config my_awesome_test.toml > my_awesome_test_out.txt 2>&1 &
```
##### PointGenerator
The `PointGenerator` is responsible for generating points that will be written into InfluxDB. Additionally, it is reponsible for keeping track of the latest timestamp of the points it is writing (Just incase the its needed by the `Reader`).
To run multiple instances of `influx_stress` just change the `measurement` each test writes to, details below
```bash
$ influx_stress -config my_awesome_test1.toml > my_awesome_test_out1.txt 2>&1 &
$ influx_stress -config my_awesome_test2.toml > my_awesome_test_out2.txt 2>&1 &
```
Any type that implements the methods `Generate()` and `Time()` is a `PointGenerator`.
Below is a sample configuration file with comments explaining the different options
```toml
# The [provision] section creates a new database on the target instance for the stress test to write points to and perform queries against
# This section can be deleted if the instance is manually configured. In that case make sure that the database referenced in [write] exists
# The provisioner will try to delete the database before trying to recreate it.
##### InfluxClient
The `InfluxClient` is responsible for writing the data that is generated by the `PointGenerator`.
[provision]
[provision.basic]
# If set to false you can delete this section from the config
enabled = true
# address of the node to be provisioned
address = "<node1_ip>:8086"
# name of the database to create
database = "stress"
# This must be set to true
reset_database = true
Any type that implements `Batch(ps <-chan Point, r chan<- response)`, and `send(b []byte) response` is an `InfluxClient`.
# The [write] section defines the shape of the generated data and configures the InfluxDB client
[write]
# The [write.point_generator] defines the shape of the generated data
[write.point_generator]
[write.point_generator.basic]
# This needs to be set to true
enabled = true
# The total number of points a stress_test will write is determined by multiplying the following two numbers:
# point_count * series_count = total_points
# Number of points to write to the database for each series
point_count = 100
# Number of series to write to the database?
series_count = 100000
# This simulates collection interval in the timestamps of generated points
tick = "10s"
# This must be set to true
jitter = true
# The measurement name for the generated points
measurement = "cpu"
# The generated timestamps follow the pattern of { start_date + (n * tick) }
# This sequence is preserved for each series and is always increasing
start_date = "2006-Jan-02"
# Precision for generated points
# This setting MUST be the same as [write.influx_client.basic]precision
precision = "s"
# The '[[]]' in toml format indicates that the element is an array of items.
# [[write.point_generator.basic.tag]] defines a tag on the generated points
# key is the tag key
# value is the tag value
# The first tag defined will have '-0' through '-{series_count}' added to the end of the string
[[write.point_generator.basic.tag]]
key = "host"
value = "server"
[[write.point_generator.basic.tag]]
key = "location"
value = "us-west"
# [[write.point_generator.basic.field]] defines a field on the generated points
# key is the field key
# value is the type of the field
[[write.point_generator.basic.field]]
key = "value"
# Can be either "float64", "int", "bool"
value = "float64"
### Reader
The `Reader` is responsible for querying the database. It has two components: `QueryGenerator` and `QueryClient`.
##### QueryGenerator
The `QueryGenerator` is responsible for generating queries.
##### QueryClient
The `QueryClient` is responsible for executing queries against an InfluxDB instance.
## Basic
`basic.go` implements an each of the components of a stress test.
## Util
`util.go` contains utility methods used throughout the package.
## Config
`config.go` contains the logic for managing the configuration of the stress test.
A sample configuration file can be found in `stress/stress.toml`. This still needs work, but whats there now is good enough IMO.
## Template
`template.go` contains the logic for a basic stress test.
# The [write.influx_client] defines what influx instances the stress_test targets
[write.influx_client]
[write.influx_client.basic]
# This must be set to true
enabled = true
# This is an array of addresses
# addresses = ["<node1_ip>:8086","<node2_ip>:8086","<node3_ip>:8086"] to target a cluster
addresses = ["<node1_ip>:8086"] # to target an individual node
# This database in the in the target influx instance to write to
# This database MUST be created in the target instance or the test will fail
database = "stress"
# Write precision for points
# This setting MUST be the same as [write.point_generator.basic]precision
precision = "s"
# The number of point to write to the database with each POST /write sent
batch_size = 5000
# An optional amount of time for a worker to wait between POST requests
batch_interval = "0s"
# The number of workers to use to write to the database
# More workers == more load with diminishing returns starting at ~5 workers
# 10 workers provides a medium-high level of load to the database
concurrency = 10
# This must be set to false
ssl = false
# This must be set to "line_http"
format = "line_http"
```