finished sql guide, added downloadable sample data
parent
72cc88e814
commit
e84a1cf6d8
|
|
@ -6,7 +6,7 @@ menu:
|
|||
v2_0:
|
||||
name: Create histograms
|
||||
parent: How-to guides
|
||||
weight: 207
|
||||
weight: 208
|
||||
---
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,194 @@
|
|||
---
|
||||
title: Query SQL data sources
|
||||
seotitle: Query SQL data sources with InfluxDB
|
||||
description: >
|
||||
placeholder
|
||||
v2.0/tags: [query, flux, sql]
|
||||
menu:
|
||||
v2_0:
|
||||
parent: How-to guides
|
||||
weight: 207
|
||||
---
|
||||
|
||||
The [Flux](/v2.0/reference/flux) `sql` package provides functions for working with SQL data sources.
|
||||
[`sql.from()`](/v2.0/reference/flux/functions/sql/from/) lets you query SQL databases
|
||||
like [PostgreSQL](https://www.postgresql.org/) and [MySQL](https://www.mysql.com/)
|
||||
and use the results with InfluxDB dashboards, tasks, and other operations.
|
||||
|
||||
To query a SQL data source, import the `sql` package in your Flux query and use
|
||||
the `sql.from()` function:
|
||||
|
||||
{{< code-tabs-wrapper >}}
|
||||
{{% code-tabs %}}
|
||||
[Postgres](#)
|
||||
[MySQL](#)
|
||||
{{% /code-tabs %}}
|
||||
|
||||
{{% code-tab-content %}}
|
||||
```js
|
||||
import "sql"
|
||||
|
||||
sql.from(
|
||||
driverName: "postgres",
|
||||
dataSourceName: "postgresql://user:password@localhost",
|
||||
query: "SELECT * FROM exampleTable"
|
||||
)
|
||||
```
|
||||
{{% /code-tab-content %}}
|
||||
|
||||
{{% code-tab-content %}}
|
||||
```js
|
||||
import "sql"
|
||||
|
||||
sql.from(
|
||||
driverName: "mysql",
|
||||
dataSourceName: "user:password@tcp(localhost:3306)/db",
|
||||
query: "SELECT * FROM exampleTable"
|
||||
)
|
||||
```
|
||||
{{% /code-tab-content %}}
|
||||
{{< /code-tabs-wrapper >}}
|
||||
|
||||
_See the [`sql.from()` documentation](/v2.0/reference/flux/functions/sql/from/) for
|
||||
information about required function parameters._
|
||||
|
||||
## Use cases
|
||||
|
||||
### Join SQL results with time series data
|
||||
One of the primary benefits of querying SQL data sources from InfluxDB
|
||||
is the ability to enrich query results with data stored outside of InfluxDB.
|
||||
|
||||
Using the [air sensor sample data](#sample-data) below, the following query
|
||||
joins air sensor metrics stored in InfluxDB with sensor information stored in PostgreSQL.
|
||||
The joined data lets you query and filter results based on sensor information
|
||||
that isn't stored in InfluxDB.
|
||||
|
||||
```js
|
||||
import "sql"
|
||||
|
||||
sensorInfo = sql.from(
|
||||
driverName: "postgres",
|
||||
dataSourceName: "postgresql://localhost?sslmode=disable",
|
||||
query: "SELECT * FROM sensors"
|
||||
)
|
||||
|
||||
sensorMetrics = from(bucket: "example-bucket")
|
||||
|> range(start: -1h)
|
||||
|> filter(fn: (r) => r._measurement == "airSensors")
|
||||
|
||||
join(tables: {metric: sensorMetrics, info: sensorInfo}, on: ["sensor_id"])
|
||||
```
|
||||
|
||||
### Create dashboard variables using SQL results
|
||||
With the `sql.from()` function you're able to [create dashboard variables](/v2.0/visualize-data/variables/create-variable/)
|
||||
from SQL query results.
|
||||
The following example uses the [air sensor sample data](#sample-data) below to
|
||||
create a variable that lets you select the location of a sensor.
|
||||
|
||||
```js
|
||||
import "sql"
|
||||
|
||||
sql.from(
|
||||
driverName: "postgres",
|
||||
dataSourceName: "postgresql://localhost?sslmode=disable",
|
||||
query: "SELECT * FROM sensors"
|
||||
)
|
||||
|> rename(columns: {location: "_value"})
|
||||
|> keep(columns: ["_value"])
|
||||
```
|
||||
|
||||
You can then use this variable to manipulate queries in your dashboards.
|
||||
|
||||
{{< img-hd src="/img/2-0-sql-dashboard-variable.png" alt="Dashboard variable from SQL query results" />}}
|
||||
|
||||
---
|
||||
|
||||
## Sample data
|
||||
The [sample data generator](#sample-data-generator) and [sample sensor information](#sample-sensor-information)
|
||||
simulate a fleet of sensors installed throughout a facility measuring the temperature,
|
||||
humidity, and carbon monoxide in each room.
|
||||
Each collected data point is stored in InfluxDB with a `sensor_id` tag that identifies
|
||||
the specific sensor it came from.
|
||||
|
||||
Observed sensor data is stored in the `airSensors` measurement which contains the following fields:
|
||||
|
||||
- temperature
|
||||
- humidity
|
||||
- co
|
||||
|
||||
Information about each sensor is stored in a `sensors` table in a Postgres database:
|
||||
|
||||
- sensor_id
|
||||
- location
|
||||
- model_number
|
||||
- last_inspected
|
||||
|
||||
#### Sample data generator
|
||||
`air-sensor-data` is a CLI that generates air sensor data and stores in InfluxDB.
|
||||
To use it:
|
||||
|
||||
1. [Create a bucket](/v2.0/organizations/buckets/create-bucket/) in which to store the generated data.
|
||||
2. Get your [authorization token](/v2.0/security/tokens/view-tokens/).
|
||||
3. Download the sample data generator. _This tool requires **Ruby**._
|
||||
|
||||
<a class="btn download" href="/downloads/air-sensor-data" download>Download Air Sensor Generator</a>
|
||||
|
||||
4. Give `air-sensor-data` executable permissions:
|
||||
|
||||
```sh
|
||||
chmod +x air-sensor-data
|
||||
```
|
||||
|
||||
5. Start the generator by providing it with your organization, bucket, and authorization token:
|
||||
|
||||
```sh
|
||||
air-sensor-data -o your-org -b your-bucket -t YOURAUTHTOKEN
|
||||
```
|
||||
|
||||
The generator will begin writing data to InfluxDB.
|
||||
|
||||
_**Note:** Use the `--help` flag to view other configuration options._
|
||||
|
||||
## Sample sensor information
|
||||
1. [Download and install Postgres](https://www.postgresql.org/download/).
|
||||
2. Download the sample sensor information CSV.
|
||||
|
||||
<a class="btn download" href="/downloads/sample-sensor-info.csv" download>Download Sample Data</a>
|
||||
|
||||
3. Use a Postgres client (`psql` or a GUI) to import the CSV file using the following:
|
||||
|
||||
```sql
|
||||
DO $$
|
||||
DECLARE
|
||||
filepath VARCHAR(200) := '/path/to/sample-sensor-info.csv';
|
||||
BEGIN
|
||||
-- Create the sensors table
|
||||
CREATE TABLE sensors (
|
||||
sensor_id character varying(50),
|
||||
location character varying(50),
|
||||
model_number character varying(50),
|
||||
last_inspected date
|
||||
);
|
||||
-- Import sample CSV from your filesystem
|
||||
COPY sensors(sensor_id,location,model_number,last_inspected)
|
||||
FROM filepath DELIMITER ',' CSV HEADER;
|
||||
END $$
|
||||
```
|
||||
|
||||
{{% note %}}
|
||||
Update the `filepath` variable to match the path of your to your downloaded sample data CSV.
|
||||
{{% /note %}}
|
||||
|
||||
Query the table to ensure the data was imported correctly:
|
||||
|
||||
```sql
|
||||
SELECT * FROM sensors;
|
||||
```
|
||||
|
||||
#### Sample dashboard
|
||||
Download and import the Air Sensors dashboard to visualize the generated data:
|
||||
|
||||
<a class="btn download" href="/downloads/air_sensors_dashboard.json" download>Download Air Sensors dashboard</a>
|
||||
|
||||
_See [Create a dashboard](/v2.0/visualize-data/dashboards/create-dashboard/#create-a-new-dashboard)
|
||||
for information about importing a dashboard._
|
||||
|
|
@ -1,62 +0,0 @@
|
|||
---
|
||||
title: Use SQL data
|
||||
seotitle: Use SQL data with InfluxDB
|
||||
description: >
|
||||
placeholder
|
||||
v2.0/tags: [query, flux, sql]
|
||||
menu:
|
||||
v2_0:
|
||||
parent: How-to guides
|
||||
weight: 210
|
||||
---
|
||||
|
||||
|
||||
- Import Flux's `sql` package and use `sql.from` to query data from a SQL database such as Postgres or MySQL.
|
||||
|
||||
### Benefits
|
||||
- Some data schemas are better suited for SQL databases. With the ability to query a SQL data source,
|
||||
you can store SQL-suited data there and join it with your time series data.
|
||||
This allows you to reduce your InfluxDB schema complexity and improve performance.
|
||||
-
|
||||
|
||||
## Use cases
|
||||
|
||||
### Reduce cardinality by storing tag data in a SQL data source
|
||||
High series cardinality can lead to high memory usage, higher hardware costs, and impact the overall performance of InfluxDB.
|
||||
The primary culprit behind cardinality is unique tag values.
|
||||
Using a SQL data source, you can offload much of your tag data to the SQL database.
|
||||
As long as you have at least one common tag on which to join in InfluxDB, that tag data can still be associated data in InfluxDB.
|
||||
|
||||
For example:
|
||||
|
||||
#### Join relation data with time series data
|
||||
- Sensor data with relational sensor information
|
||||
- SensorID
|
||||
- Name
|
||||
- Type
|
||||
- Location
|
||||
- Model
|
||||
- Sensor metrics stored in InfluxDB, each with a `sensorID` tag.
|
||||
Each type of sensor metric is stored in a different measurement.
|
||||
- air_quality
|
||||
- temperature
|
||||
- humidity
|
||||
- co2
|
||||
- methane
|
||||
- light
|
||||
- uv
|
||||
|
||||
|
||||
```js
|
||||
import "sql"
|
||||
|
||||
sensorInfo = sql.from(
|
||||
driver: "postgres",
|
||||
driverName: "",
|
||||
query: ""
|
||||
)
|
||||
|
||||
sensorMetrics = from(bucket: "sensors" )
|
||||
|> range(start: -1d)
|
||||
|
||||
```
|
||||
|
|
@ -60,11 +60,11 @@ _**Data type:** String_
|
|||
```js
|
||||
import "sql"
|
||||
|
||||
sql.from(
|
||||
driverName: "mysql",
|
||||
dataSourceName: "user:password@tcp(localhost:3306)/db",
|
||||
query:"SELECT * FROM ExampleTable"
|
||||
)
|
||||
sql.from(
|
||||
driverName: "mysql",
|
||||
dataSourceName: "user:password@tcp(localhost:3306)/db",
|
||||
query:"SELECT * FROM ExampleTable"
|
||||
)
|
||||
```
|
||||
|
||||
### Query a Postgres database
|
||||
|
|
|
|||
|
|
@ -0,0 +1,131 @@
|
|||
#! /usr/bin/ruby
|
||||
require "optparse"
|
||||
require "net/http"
|
||||
require"openssl"
|
||||
require "uri"
|
||||
|
||||
# CLI Options
|
||||
options = {
|
||||
protocol: "http",
|
||||
host: "localhost",
|
||||
port: "9999",
|
||||
interval: 5
|
||||
}
|
||||
|
||||
OptionParser.new do |opt|
|
||||
opt.banner = "Usage: air-sensor-data [OPTIONS]"
|
||||
|
||||
opt.on("-o","--org ORG","The organization to write data to. REQUIRED.") do |org|
|
||||
options[:org] = org
|
||||
end
|
||||
|
||||
opt.on("-b","--bucket BUCKET","The bucket to write data to. REQUIRED.") do |bucket|
|
||||
options[:bucket] = bucket
|
||||
end
|
||||
|
||||
opt.on("-t","--token TOKEN","Your InfluxDB authentication token. REQUIRED.") do |token|
|
||||
options[:token] = token
|
||||
end
|
||||
|
||||
opt.on("-h","--host host","Your InfluxDB host. Defaults to 'localhost'") do |host|
|
||||
options[:host] = host
|
||||
end
|
||||
|
||||
opt.on("-p","--port port","Your InfluxDB port. Defaults to '9999'") do |port|
|
||||
options[:port] = port
|
||||
end
|
||||
|
||||
opt.on("-i","--interval interval",Integer,"The interval (in seconds) at which to write data. Defaults to '5'.") do |interval|
|
||||
options[:interval] = interval
|
||||
end
|
||||
|
||||
opt.on("-s","--tls", "Sends data over HTTPS.") do |tls|
|
||||
options[:protocol] = "https"
|
||||
end
|
||||
|
||||
opt.on("--help","Displays this help information.") do
|
||||
puts opt
|
||||
exit
|
||||
end
|
||||
end.parse!
|
||||
|
||||
unless options[:org] && options[:bucket] && options[:token]
|
||||
$stderr.puts "\nError: you must specify an organization, bucket, and token.\nUse the '--help' flag for more info.\n\n"
|
||||
exit 1
|
||||
end
|
||||
|
||||
# Global Variables
|
||||
$protocol = options[:protocol]
|
||||
$host = options[:host]
|
||||
$port = options[:port]
|
||||
$org = options[:org]
|
||||
$bucket = options[:bucket]
|
||||
$token = options[:token]
|
||||
$interval = options[:interval]
|
||||
|
||||
# Seed Data
|
||||
seeds = [
|
||||
{id: 100, t: 71.2, h: 35.1, c: 0.5, t_inc: -0.05..0.05, h_inc: -0.05..0.05, c_inc: -0.02..0.02},
|
||||
{id: 101, t: 71.8, h: 34.9, c: 0.5, t_inc: -0.05..0.05, h_inc: -0.05..0.05, c_inc: -0.02..0.02},
|
||||
{id: 102, t: 72.0, h: 34.9, c: 0.5, t_inc: -0.05..0.05, h_inc: -0.05..0.05, c_inc: -0.02..0.02},
|
||||
{id: 103, t: 71.3, h: 35.2, c: 0.4, t_inc: -0.05..0.05, h_inc: -0.05..0.05, c_inc: -0.02..0.02},
|
||||
{id: 200, t: 73.6, h: 35.8, c: 0.5, t_inc: -0.05..0.05, h_inc: -0.05..0.05, c_inc: -0.02..0.05},
|
||||
{id: 201, t: 74.0, h: 35.2, c: 0.5, t_inc: -0.05..0.05, h_inc: -0.05..0.05, c_inc: -0.02..0.02},
|
||||
{id: 202, t: 75.3, h: 35.7, c: 0.5, t_inc: -0.05..0.05, h_inc: -0.05..0.05, c_inc: -0.02..0.02},
|
||||
{id: 203, t: 74.8, h: 35.9, c: 0.4, t_inc: -0.05..0.05, h_inc: -0.05..0.05, c_inc: -0.02..0.02},
|
||||
]
|
||||
|
||||
def increment_data(data={})
|
||||
data[:t] += rand(data[:t_inc])
|
||||
data[:h] += rand(data[:h_inc])
|
||||
data[:c] += rand(data[:c_inc])
|
||||
|
||||
# Avoid negative humidity and co
|
||||
if data[:h] < 0
|
||||
data[:h] = 0
|
||||
end
|
||||
if data[:c] < 0
|
||||
data[:c] = 0
|
||||
end
|
||||
|
||||
return data
|
||||
end
|
||||
|
||||
def line_protocol_batch(point_data=[])
|
||||
batch = []
|
||||
point_data.each do |v|
|
||||
batch << "airSensors,sensor_id=TLM0#{v[:id]} temperature=#{v[:t]},humidity=#{v[:h]},co=#{v[:c]}"
|
||||
end
|
||||
return batch.join("\n")
|
||||
end
|
||||
|
||||
def send_data(batch)
|
||||
uri = URI.parse("#{$protocol}://#{$host}:#{$port}/api/v2/write?org=#{URI::encode($org)}&bucket=#{URI::encode($bucket)}")
|
||||
request = Net::HTTP::Post.new(uri)
|
||||
request["Authorization"] = "Token #{$token}"
|
||||
request.body = "#{batch}"
|
||||
|
||||
req_options = {
|
||||
use_ssl: uri.scheme == "https",
|
||||
ssl_version: :SSLv23
|
||||
}
|
||||
|
||||
response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
|
||||
http.request(request)
|
||||
end
|
||||
end
|
||||
|
||||
def send_batches(dataset=[], interval=$interval)
|
||||
dataset.map! { |seed| increment_data(seed) }
|
||||
send_data(line_protocol_batch(dataset))
|
||||
sleep interval
|
||||
send_batches(dataset,interval)
|
||||
end
|
||||
|
||||
begin
|
||||
puts "Sending data to #{$protocol}://#{$host}:#{$port}..."
|
||||
puts " (ctrl-c to kill the data stream)"
|
||||
send_batches(seeds)
|
||||
rescue Interrupt
|
||||
puts "\nStopping data stream..."
|
||||
end
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,9 @@
|
|||
sensor_id,location,model_number,last_inspected
|
||||
TLM0100,Main Lobby,TLM89092A,1/11/2019
|
||||
TLM0101,Room 101,TLM89092A,1/11/2019
|
||||
TLM0102,Room 102,TLM89092B,1/11/2019
|
||||
TLM0103,Mechanical Room,TLM90012Z,1/14/2019
|
||||
TLM0200,Conference Room,TLM89092B,9/24/2018
|
||||
TLM0201,Room 201,TLM89092B,9/24/2018
|
||||
TLM0202,Room 202,TLM89092A,9/24/2018
|
||||
TLM0203,Room 203,TLM89092A,9/24/2018
|
||||
|
Binary file not shown.
|
After Width: | Height: | Size: 50 KiB |
Loading…
Reference in New Issue