added aws athena support to sql pkg, resolves #1078

pull/1128/head
Scott Anderson 2020-06-22 12:43:25 -06:00
parent ae81534fc6
commit b21df957bf
4 changed files with 70 additions and 8 deletions

View File

@ -5,7 +5,7 @@ list_title: Query SQL data
description: >
The Flux `sql` package provides functions for working with SQL data sources.
Use `sql.from()` to query SQL databases like PostgreSQL, MySQL, Snowflake,
SQLite, and Microsoft SQL Server.
SQLite, Microsoft SQL Server, Amazon Athena.
v2.0/tags: [query, flux, sql]
menu:
v2_0:
@ -32,8 +32,9 @@ The [Flux](/v2.0/reference/flux) `sql` package provides functions for working wi
[`sql.from()`](/v2.0/reference/flux/stdlib/sql/from/) lets you query SQL data sources
like [PostgreSQL](https://www.postgresql.org/), [MySQL](https://www.mysql.com/),
[Snowflake](https://www.snowflake.com/), [SQLite](https://www.sqlite.org/index.html),
and [Microsoft SQL Server](https://www.microsoft.com/en-us/sql-server/default.aspx),
and use the results with InfluxDB dashboards, tasks, and other operations.
[Microsoft SQL Server](https://www.microsoft.com/en-us/sql-server/default.aspx),
and [Amazon Athena](https://aws.amazon.com/athena/) and use the results with
InfluxDB dashboards, tasks, and other operations.
- [Query a SQL data source](#query-a-sql-data-source)
- [Join SQL data with data in InfluxDB](#join-sql-data-with-data-in-influxdb)

View File

@ -3,7 +3,7 @@ title: Flux SQL package
list_title: SQL package
description: >
The Flux SQL package provides tools for working with data in SQL databases such
as MySQL, PostgreSQL, Snowflake, SQLite, and Microsoft SQL Server.
as MySQL, PostgreSQL, Snowflake, SQLite, Microsoft SQL Server, and Amazon Athena.
Import the `sql` package.
aliases:
- /v2.0/reference/flux/functions/sql/
@ -17,8 +17,15 @@ related:
- /v2.0/query-data/flux/sql/
---
SQL Flux functions provide tools for working with data in SQL databases such as
MySQL, PostgreSQL, Snowflake, SQLite, and Microsoft SQL Server.
SQL Flux functions provide tools for working with data in SQL databases such as:
- Amazon Athena
- Microsoft SQL Server
- MySQL
- PostgreSQL
- Snowflake
- SQLite
Import the `sql` package:
```js

View File

@ -35,6 +35,7 @@ _**Data type:** String_
The following drivers are available:
- awsathena
- mysql
- postgres
- snowflake
@ -49,12 +50,16 @@ _**Data type:** String_
##### Driver dataSourceName examples
```sh
# Postgres Driver DSN
postgres://pqgotest:password@localhost/pqgotest?sslmode=verify-full
# Amazon Athena Driver DSN
s3://myorgqueryresults/?accessID=AKIAJLO3F...&region=us-west-1&secretAccessKey=NnQ7MUMp9PYZsmD47c%2BSsXGOFsd%2F...
s3://myorgqueryresults/?accessID=AKIAJLO3F...&db=dbname&missingAsDefault=false&missingAsEmptyString=false&region=us-west-1&secretAccessKey=NnQ7MUMp9PYZsmD47c%2BSsXGOFsd%2F...&WGRemoteCreation=false
# MySQL Driver DSN
username:password@tcp(localhost:3306)/dbname?param=value
# Postgres Driver DSN
postgres://pqgotest:password@localhost/pqgotest?sslmode=verify-full
# Snowflake Driver DSNs
username[:password]@accountname/dbname/schemaname?param1=value1&paramN=valueN
username[:password]@accountname/dbname?param1=value1&paramN=valueN
@ -77,6 +82,13 @@ _**Data type:** String_
## Examples
- [MySQL](#query-a-mysql-database)
- [Postgres](#query-a-postgres-database)
- [Snowflake](#query-a-snowflake-database)
- [SQLite](#query-an-sqlite-database)
- [Amazon Athena](#query-an-amazon-athena-database)
- [SQL Server](#query-a-sql-server-database)
{{% note %}}
The examples below use [InfluxDB secrets](/v2.0/security/secrets/) to populate
sensitive connection credentials.
@ -147,6 +159,37 @@ sql.from(
)
```
### Query an Amazon Athena database
```js
import "sql"
import "influxdata/influxdb/secrets"
region = us-west-1
accessID = secrets.get(key: "ATHENA_ACCESS_ID")
secretKey = secrets.get(key: "ATHENA_SECRET_KEY")
sql.from(
driverName: "awsathena",
dataSourceName: "s3://myorgqueryresults/?accessID=${accessID}&region=${region}&secretAccessKey=${secretKey}",
query:"SELECT * FROM example_table"
)
```
##### Athena connection string
To query an Amazon Athena database, use the following query parameters in your Athena
S3 connection string (DSN):
<span class="req">\* Required</span>
- **region** - AWS region <span class="req">\*</span>
- **accessID** - AWS IAM access ID <span class="req">\*</span>
- **secretAccessKey** - AWS IAM secret key <span class="req">\*</span>
- **db** - database name
- **WGRemoteCreation** - controls workgroup and tag creation
- **missingAsDefault** - replace missing data with default values
- **missingAsEmptyString** - replace missing data with empty strings
### Query a SQL Server database
```js
import "sql"

View File

@ -86,6 +86,12 @@ If writing to a **SQLite** database, set `batchSize` to `999` or less.
## Examples
- [MySQL](#write-data-to-a-mysql-database)
- [Postgres](#write-data-to-a-postgres-database)
- [Snowflake](#write-data-to-a-snowflake-database)
- [SQLite](#write-data-to-an-sqlite-database)
- [SQL Server](#write-data-to-a-sql-server-database)
{{% note %}}
The examples below use [InfluxDB secrets](/v2.0/security/secrets/) to populate
sensitive connection credentials.
@ -216,3 +222,8 @@ _For information about managed identities, see [Microsoft managed identities](ht
```
azure auth=MSI
```
{{% warn %}}
### sql.to does not support Amazon Athena
The `sql.to` function does not support writing data to [Amazon Athena](https://aws.amazon.com/athena/).
{{% /warn %}}