finished sql guide, added downloadable sample data

pull/320/head
Scott Anderson 2019-07-10 17:19:27 -06:00
parent 72cc88e814
commit e84a1cf6d8
8 changed files with 1369 additions and 68 deletions

View File

@ -6,7 +6,7 @@ menu:
v2_0:
name: Create histograms
parent: How-to guides
weight: 207
weight: 208
---

View File

@ -0,0 +1,194 @@
---
title: Query SQL data sources
seotitle: Query SQL data sources with InfluxDB
description: >
placeholder
v2.0/tags: [query, flux, sql]
menu:
v2_0:
parent: How-to guides
weight: 207
---
The [Flux](/v2.0/reference/flux) `sql` package provides functions for working with SQL data sources.
[`sql.from()`](/v2.0/reference/flux/functions/sql/from/) lets you query SQL databases
like [PostgreSQL](https://www.postgresql.org/) and [MySQL](https://www.mysql.com/)
and use the results with InfluxDB dashboards, tasks, and other operations.
To query a SQL data source, import the `sql` package in your Flux query and use
the `sql.from()` function:
{{< code-tabs-wrapper >}}
{{% code-tabs %}}
[Postgres](#)
[MySQL](#)
{{% /code-tabs %}}
{{% code-tab-content %}}
```js
import "sql"
sql.from(
driverName: "postgres",
dataSourceName: "postgresql://user:password@localhost",
query: "SELECT * FROM exampleTable"
)
```
{{% /code-tab-content %}}
{{% code-tab-content %}}
```js
import "sql"
sql.from(
driverName: "mysql",
dataSourceName: "user:password@tcp(localhost:3306)/db",
query: "SELECT * FROM exampleTable"
)
```
{{% /code-tab-content %}}
{{< /code-tabs-wrapper >}}
_See the [`sql.from()` documentation](/v2.0/reference/flux/functions/sql/from/) for
information about required function parameters._
## Use cases
### Join SQL results with time series data
One of the primary benefits of querying SQL data sources from InfluxDB
is the ability to enrich query results with data stored outside of InfluxDB.
Using the [air sensor sample data](#sample-data) below, the following query
joins air sensor metrics stored in InfluxDB with sensor information stored in PostgreSQL.
The joined data lets you query and filter results based on sensor information
that isn't stored in InfluxDB.
```js
import "sql"
sensorInfo = sql.from(
driverName: "postgres",
dataSourceName: "postgresql://localhost?sslmode=disable",
query: "SELECT * FROM sensors"
)
sensorMetrics = from(bucket: "example-bucket")
|> range(start: -1h)
|> filter(fn: (r) => r._measurement == "airSensors")
join(tables: {metric: sensorMetrics, info: sensorInfo}, on: ["sensor_id"])
```
### Create dashboard variables using SQL results
With the `sql.from()` function you're able to [create dashboard variables](/v2.0/visualize-data/variables/create-variable/)
from SQL query results.
The following example uses the [air sensor sample data](#sample-data) below to
create a variable that lets you select the location of a sensor.
```js
import "sql"
sql.from(
driverName: "postgres",
dataSourceName: "postgresql://localhost?sslmode=disable",
query: "SELECT * FROM sensors"
)
|> rename(columns: {location: "_value"})
|> keep(columns: ["_value"])
```
You can then use this variable to manipulate queries in your dashboards.
{{< img-hd src="/img/2-0-sql-dashboard-variable.png" alt="Dashboard variable from SQL query results" />}}
---
## Sample data
The [sample data generator](#sample-data-generator) and [sample sensor information](#sample-sensor-information)
simulate a fleet of sensors installed throughout a facility measuring the temperature,
humidity, and carbon monoxide in each room.
Each collected data point is stored in InfluxDB with a `sensor_id` tag that identifies
the specific sensor it came from.
Observed sensor data is stored in the `airSensors` measurement which contains the following fields:
- temperature
- humidity
- co
Information about each sensor is stored in a `sensors` table in a Postgres database:
- sensor_id
- location
- model_number
- last_inspected
#### Sample data generator
`air-sensor-data` is a CLI that generates air sensor data and stores in InfluxDB.
To use it:
1. [Create a bucket](/v2.0/organizations/buckets/create-bucket/) in which to store the generated data.
2. Get your [authorization token](/v2.0/security/tokens/view-tokens/).
3. Download the sample data generator. _This tool requires **Ruby**._
<a class="btn download" href="/downloads/air-sensor-data" download>Download Air Sensor Generator</a>
4. Give `air-sensor-data` executable permissions:
```sh
chmod +x air-sensor-data
```
5. Start the generator by providing it with your organization, bucket, and authorization token:
```sh
air-sensor-data -o your-org -b your-bucket -t YOURAUTHTOKEN
```
The generator will begin writing data to InfluxDB.
_**Note:** Use the `--help` flag to view other configuration options._
## Sample sensor information
1. [Download and install Postgres](https://www.postgresql.org/download/).
2. Download the sample sensor information CSV.
<a class="btn download" href="/downloads/sample-sensor-info.csv" download>Download Sample Data</a>
3. Use a Postgres client (`psql` or a GUI) to import the CSV file using the following:
```sql
DO $$
DECLARE
filepath VARCHAR(200) := '/path/to/sample-sensor-info.csv';
BEGIN
-- Create the sensors table
CREATE TABLE sensors (
sensor_id character varying(50),
location character varying(50),
model_number character varying(50),
last_inspected date
);
-- Import sample CSV from your filesystem
COPY sensors(sensor_id,location,model_number,last_inspected)
FROM filepath DELIMITER ',' CSV HEADER;
END $$
```
{{% note %}}
Update the `filepath` variable to match the path of your to your downloaded sample data CSV.
{{% /note %}}
Query the table to ensure the data was imported correctly:
```sql
SELECT * FROM sensors;
```
#### Sample dashboard
Download and import the Air Sensors dashboard to visualize the generated data:
<a class="btn download" href="/downloads/air_sensors_dashboard.json" download>Download Air Sensors dashboard</a>
_See [Create a dashboard](/v2.0/visualize-data/dashboards/create-dashboard/#create-a-new-dashboard)
for information about importing a dashboard._

View File

@ -1,62 +0,0 @@
---
title: Use SQL data
seotitle: Use SQL data with InfluxDB
description: >
placeholder
v2.0/tags: [query, flux, sql]
menu:
v2_0:
parent: How-to guides
weight: 210
---
- Import Flux's `sql` package and use `sql.from` to query data from a SQL database such as Postgres or MySQL.
### Benefits
- Some data schemas are better suited for SQL databases. With the ability to query a SQL data source,
you can store SQL-suited data there and join it with your time series data.
This allows you to reduce your InfluxDB schema complexity and improve performance.
-
## Use cases
### Reduce cardinality by storing tag data in a SQL data source
High series cardinality can lead to high memory usage, higher hardware costs, and impact the overall performance of InfluxDB.
The primary culprit behind cardinality is unique tag values.
Using a SQL data source, you can offload much of your tag data to the SQL database.
As long as you have at least one common tag on which to join in InfluxDB, that tag data can still be associated data in InfluxDB.
For example:
#### Join relation data with time series data
- Sensor data with relational sensor information
- SensorID
- Name
- Type
- Location
- Model
- Sensor metrics stored in InfluxDB, each with a `sensorID` tag.
Each type of sensor metric is stored in a different measurement.
- air_quality
- temperature
- humidity
- co2
- methane
- light
- uv
```js
import "sql"
sensorInfo = sql.from(
driver: "postgres",
driverName: "",
query: ""
)
sensorMetrics = from(bucket: "sensors" )
|> range(start: -1d)
```

View File

@ -60,11 +60,11 @@ _**Data type:** String_
```js
import "sql"
sql.from(
driverName: "mysql",
dataSourceName: "user:password@tcp(localhost:3306)/db",
query:"SELECT * FROM ExampleTable"
)
sql.from(
driverName: "mysql",
dataSourceName: "user:password@tcp(localhost:3306)/db",
query:"SELECT * FROM ExampleTable"
)
```
### Query a Postgres database

131
static/downloads/air-sensor-data Executable file
View File

@ -0,0 +1,131 @@
#! /usr/bin/ruby
require "optparse"
require "net/http"
require"openssl"
require "uri"
# CLI Options
options = {
protocol: "http",
host: "localhost",
port: "9999",
interval: 5
}
OptionParser.new do |opt|
opt.banner = "Usage: air-sensor-data [OPTIONS]"
opt.on("-o","--org ORG","The organization to write data to. REQUIRED.") do |org|
options[:org] = org
end
opt.on("-b","--bucket BUCKET","The bucket to write data to. REQUIRED.") do |bucket|
options[:bucket] = bucket
end
opt.on("-t","--token TOKEN","Your InfluxDB authentication token. REQUIRED.") do |token|
options[:token] = token
end
opt.on("-h","--host host","Your InfluxDB host. Defaults to 'localhost'") do |host|
options[:host] = host
end
opt.on("-p","--port port","Your InfluxDB port. Defaults to '9999'") do |port|
options[:port] = port
end
opt.on("-i","--interval interval",Integer,"The interval (in seconds) at which to write data. Defaults to '5'.") do |interval|
options[:interval] = interval
end
opt.on("-s","--tls", "Sends data over HTTPS.") do |tls|
options[:protocol] = "https"
end
opt.on("--help","Displays this help information.") do
puts opt
exit
end
end.parse!
unless options[:org] && options[:bucket] && options[:token]
$stderr.puts "\nError: you must specify an organization, bucket, and token.\nUse the '--help' flag for more info.\n\n"
exit 1
end
# Global Variables
$protocol = options[:protocol]
$host = options[:host]
$port = options[:port]
$org = options[:org]
$bucket = options[:bucket]
$token = options[:token]
$interval = options[:interval]
# Seed Data
seeds = [
{id: 100, t: 71.2, h: 35.1, c: 0.5, t_inc: -0.05..0.05, h_inc: -0.05..0.05, c_inc: -0.02..0.02},
{id: 101, t: 71.8, h: 34.9, c: 0.5, t_inc: -0.05..0.05, h_inc: -0.05..0.05, c_inc: -0.02..0.02},
{id: 102, t: 72.0, h: 34.9, c: 0.5, t_inc: -0.05..0.05, h_inc: -0.05..0.05, c_inc: -0.02..0.02},
{id: 103, t: 71.3, h: 35.2, c: 0.4, t_inc: -0.05..0.05, h_inc: -0.05..0.05, c_inc: -0.02..0.02},
{id: 200, t: 73.6, h: 35.8, c: 0.5, t_inc: -0.05..0.05, h_inc: -0.05..0.05, c_inc: -0.02..0.05},
{id: 201, t: 74.0, h: 35.2, c: 0.5, t_inc: -0.05..0.05, h_inc: -0.05..0.05, c_inc: -0.02..0.02},
{id: 202, t: 75.3, h: 35.7, c: 0.5, t_inc: -0.05..0.05, h_inc: -0.05..0.05, c_inc: -0.02..0.02},
{id: 203, t: 74.8, h: 35.9, c: 0.4, t_inc: -0.05..0.05, h_inc: -0.05..0.05, c_inc: -0.02..0.02},
]
def increment_data(data={})
data[:t] += rand(data[:t_inc])
data[:h] += rand(data[:h_inc])
data[:c] += rand(data[:c_inc])
# Avoid negative humidity and co
if data[:h] < 0
data[:h] = 0
end
if data[:c] < 0
data[:c] = 0
end
return data
end
def line_protocol_batch(point_data=[])
batch = []
point_data.each do |v|
batch << "airSensors,sensor_id=TLM0#{v[:id]} temperature=#{v[:t]},humidity=#{v[:h]},co=#{v[:c]}"
end
return batch.join("\n")
end
def send_data(batch)
uri = URI.parse("#{$protocol}://#{$host}:#{$port}/api/v2/write?org=#{URI::encode($org)}&bucket=#{URI::encode($bucket)}")
request = Net::HTTP::Post.new(uri)
request["Authorization"] = "Token #{$token}"
request.body = "#{batch}"
req_options = {
use_ssl: uri.scheme == "https",
ssl_version: :SSLv23
}
response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
http.request(request)
end
end
def send_batches(dataset=[], interval=$interval)
dataset.map! { |seed| increment_data(seed) }
send_data(line_protocol_batch(dataset))
sleep interval
send_batches(dataset,interval)
end
begin
puts "Sending data to #{$protocol}://#{$host}:#{$port}..."
puts " (ctrl-c to kill the data stream)"
send_batches(seeds)
rescue Interrupt
puts "\nStopping data stream..."
end

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,9 @@
sensor_id,location,model_number,last_inspected
TLM0100,Main Lobby,TLM89092A,1/11/2019
TLM0101,Room 101,TLM89092A,1/11/2019
TLM0102,Room 102,TLM89092B,1/11/2019
TLM0103,Mechanical Room,TLM90012Z,1/14/2019
TLM0200,Conference Room,TLM89092B,9/24/2018
TLM0201,Room 201,TLM89092B,9/24/2018
TLM0202,Room 202,TLM89092A,9/24/2018
TLM0203,Room 203,TLM89092A,9/24/2018
1 sensor_id location model_number last_inspected
2 TLM0100 Main Lobby TLM89092A 1/11/2019
3 TLM0101 Room 101 TLM89092A 1/11/2019
4 TLM0102 Room 102 TLM89092B 1/11/2019
5 TLM0103 Mechanical Room TLM90012Z 1/14/2019
6 TLM0200 Conference Room TLM89092B 9/24/2018
7 TLM0201 Room 201 TLM89092B 9/24/2018
8 TLM0202 Room 202 TLM89092A 9/24/2018
9 TLM0203 Room 203 TLM89092A 9/24/2018

Binary file not shown.

After

Width:  |  Height:  |  Size: 50 KiB