added bigquery to sql package

pull/1387/head
Scott Anderson 2020-08-11 16:33:50 -06:00
parent 76eecf86be
commit 99f9c388a2
3 changed files with 97 additions and 5 deletions

View File

@ -3,7 +3,8 @@ title: Flux SQL package
list_title: SQL package
description: >
The Flux SQL package provides tools for working with data in SQL databases such
as MySQL, PostgreSQL, Snowflake, SQLite, Microsoft SQL Server, and Amazon Athena.
as MySQL, PostgreSQL, Snowflake, SQLite, Microsoft SQL Server, Amazon Athena,
and Google BigQuery.
Import the `sql` package.
aliases:
- /v2.0/reference/flux/functions/sql/
@ -20,6 +21,7 @@ related:
SQL Flux functions provide tools for working with data in SQL databases such as:
- Amazon Athena
- Google BigQuery
- Microsoft SQL Server
- MySQL
- PostgreSQL

View File

@ -36,6 +36,7 @@ _**Data type:** String_
The following drivers are available:
- awsathena
- bigquery
- mysql
- postgres
- snowflake
@ -73,6 +74,10 @@ sqlserver://username:password@localhost:1234?database=examplebdb
server=localhost;user id=username;database=examplebdb;
server=localhost;user id=username;database=examplebdb;azure auth=ENV
server=localhost;user id=username;database=examplebdbr;azure tenant id=77e7d537;azure client id=58879ce8;azure client secret=0123456789
# Google BigQuery DSNs
bigquery://projectid/?param1=value&param2=value
bigquery://projectid/location?param1=value&param2=value
```
### query
@ -88,6 +93,7 @@ _**Data type:** String_
- [SQLite](#query-an-sqlite-database)
- [Amazon Athena](#query-an-amazon-athena-database)
- [SQL Server](#query-a-sql-server-database)
- [Google BigQuery](#query-a-bigquery-database)
{{% note %}}
The examples below use [InfluxDB secrets](/v2.0/security/secrets/) to populate
@ -250,3 +256,42 @@ _For information about managed identities, see [Microsoft managed identities](ht
```
azure auth=MSI
```
### Query a BigQuery database
```js
import "sql"
import "influxdata/influxdb/secrets"
projectID = secrets.get(key: "BIGQUERY_PROJECT_ID")
apiKey = secrets.get(key: "BIGQUERY_APIKEY")
sql.from(
driverName: "bigquery",
dataSourceName: "bigquery://${projectID}/?apiKey=${apiKey}",
query:"SELECT * FROM exampleTable"
)
```
#### Common BigQuery URL parameters
- **dataset** - BigQuery dataset ID. When set, you can use unqualified tables names in queries.
#### BigQuery authentication parameters
The Flux BigQuery implementation uses the Google Cloud Go SDK.
Provide your authentication credentials using one of the following methods:
- The `GOOGLE_APPLICATION_CREDENTIALS` environment variable that identifies the
location of your credential JSON file.
- Provide your BigQuery API key using the **apiKey** URL parameter in your BigQuery DSN.
###### Example apiKey URL parameter
```
bigquery://projectid/?apiKey=AIzaSyB6XK8IO5AzKZXoioQOVNTFYzbDBjY5hy4
```
- Provide your base-64 encoded service account, refresh token, or JSON credentials
using the **credentials** URL parameter in your BigQuery DSN.
###### Example credentials URL parameter
```
bigquery://projectid/?credentials=eyJ0eXBlIjoiYXV0...
```

View File

@ -34,12 +34,18 @@ _**Data type:** String_
The following drivers are available:
- bigquery
- mysql
- postgres
- snowflake
- sqlite3 _Does not work with InfluxDB OSS or InfluxDB Cloud. More information [below](#write-data-to-an-sqlite-database)._
- sqlserver, mssql
{{% warn %}}
### sql.to does not support Amazon Athena
The `sql.to` function does not support writing data to [Amazon Athena](https://aws.amazon.com/athena/).
{{% /warn %}}
### dataSourceName
The data source name (DSN) or connection string used to connect to the SQL database.
The string's form and structure depend on the [driver](#drivername) used.
@ -67,6 +73,10 @@ sqlserver://username:password@localhost:1234?database=examplebdb
server=localhost;user id=username;database=examplebdb;
server=localhost;user id=username;database=examplebdb;azure auth=ENV
server=localhost;user id=username;database=examplebdbr;azure tenant id=77e7d537;azure client id=58879ce8;azure client secret=0123456789
# Google BigQuery DSNs
bigquery://projectid/?param1=value&param2=value
bigquery://projectid/location?param1=value&param2=value
```
### table
@ -91,6 +101,7 @@ If writing to a **SQLite** database, set `batchSize` to `999` or less.
- [Snowflake](#write-data-to-a-snowflake-database)
- [SQLite](#write-data-to-an-sqlite-database)
- [SQL Server](#write-data-to-a-sql-server-database)
- [Google BigQuery](#write-data-to-a-sql-server-database)
{{% note %}}
The examples below use [InfluxDB secrets](/v2.0/security/secrets/) to populate
@ -223,7 +234,41 @@ _For information about managed identities, see [Microsoft managed identities](ht
azure auth=MSI
```
{{% warn %}}
### sql.to does not support Amazon Athena
The `sql.to` function does not support writing data to [Amazon Athena](https://aws.amazon.com/athena/).
{{% /warn %}}
### Write to a BigQuery database
```js
import "sql"
import "influxdata/influxdb/secrets"
projectID = secrets.get(key: "BIGQUERY_PROJECT_ID")
apiKey = secrets.get(key: "BIGQUERY_APIKEY")
sql.to(
driverName: "bigquery",
dataSourceName: "bigquery://${projectID}/?apiKey=${apiKey}",
table:"exampleTable"
)
```
#### Common BigQuery URL parameters
- **dataset** - BigQuery dataset ID. When set, you can use unqualified tables names in queries.
#### BigQuery authentication parameters
The Flux BigQuery implementation uses the Google Cloud Go SDK.
Provide your authentication credentials using one of the following methods:
- The `GOOGLE_APPLICATION_CREDENTIALS` environment variable that identifies the
location of your credential JSON file.
- Provide your BigQuery API key using the **apiKey** URL parameter in your BigQuery DSN.
###### Example apiKey URL parameter
```
bigquery://projectid/?apiKey=AIzaSyB6XK8IO5AzKZXoioQOVNTFYzbDBjY5hy4
```
- Provide your base-64 encoded service account, refresh token, or JSON credentials
using the **credentials** URL parameter in your BigQuery DSN.
###### Example credentials URL parameter
```
bigquery://projectid/?credentials=eyJ0eXBlIjoiYXV0...
```