added BigQuery support to sql package
parent
db6e89ce04
commit
75b7fdd00a
|
@ -5,7 +5,7 @@ list_title: Query SQL data
|
||||||
description: >
|
description: >
|
||||||
The Flux `sql` package provides functions for working with SQL data sources.
|
The Flux `sql` package provides functions for working with SQL data sources.
|
||||||
Use `sql.from()` to query SQL databases like PostgreSQL, MySQL, Snowflake,
|
Use `sql.from()` to query SQL databases like PostgreSQL, MySQL, Snowflake,
|
||||||
SQLite, Microsoft SQL Server, and Amazon Athena.
|
SQLite, Microsoft SQL Server, Amazon Athena, and Google BigQuery.
|
||||||
influxdb/v2.0/tags: [query, flux, sql]
|
influxdb/v2.0/tags: [query, flux, sql]
|
||||||
menu:
|
menu:
|
||||||
influxdb_2_0:
|
influxdb_2_0:
|
||||||
|
@ -33,8 +33,8 @@ The [Flux](/influxdb/v2.0/reference/flux) `sql` package provides functions for w
|
||||||
like [PostgreSQL](https://www.postgresql.org/), [MySQL](https://www.mysql.com/),
|
like [PostgreSQL](https://www.postgresql.org/), [MySQL](https://www.mysql.com/),
|
||||||
[Snowflake](https://www.snowflake.com/), [SQLite](https://www.sqlite.org/index.html),
|
[Snowflake](https://www.snowflake.com/), [SQLite](https://www.sqlite.org/index.html),
|
||||||
[Microsoft SQL Server](https://www.microsoft.com/en-us/sql-server/default.aspx),
|
[Microsoft SQL Server](https://www.microsoft.com/en-us/sql-server/default.aspx),
|
||||||
and [Amazon Athena](https://aws.amazon.com/athena/) and use the results with
|
[Amazon Athena](https://aws.amazon.com/athena/) and [Google BigQuery](https://cloud.google.com/bigquery)
|
||||||
InfluxDB dashboards, tasks, and other operations.
|
and use the results with InfluxDB dashboards, tasks, and other operations.
|
||||||
|
|
||||||
- [Query a SQL data source](#query-a-sql-data-source)
|
- [Query a SQL data source](#query-a-sql-data-source)
|
||||||
- [Join SQL data with data in InfluxDB](#join-sql-data-with-data-in-influxdb)
|
- [Join SQL data with data in InfluxDB](#join-sql-data-with-data-in-influxdb)
|
||||||
|
@ -61,6 +61,8 @@ To query a SQL data source:
|
||||||
[Snowflake](#)
|
[Snowflake](#)
|
||||||
[SQLite](#)
|
[SQLite](#)
|
||||||
[SQL Server](#)
|
[SQL Server](#)
|
||||||
|
[Athena](#)
|
||||||
|
[BigQuery](#)
|
||||||
{{% /code-tabs %}}
|
{{% /code-tabs %}}
|
||||||
|
|
||||||
{{% code-tab-content %}}
|
{{% code-tab-content %}}
|
||||||
|
@ -129,6 +131,33 @@ sql.from(
|
||||||
_For information about authenticating with SQL Server using ADO-style parameters,
|
_For information about authenticating with SQL Server using ADO-style parameters,
|
||||||
see [SQL Server ADO authentication](/influxdb/v2.0/reference/flux/stdlib/sql/from/#sql-server-ado-authentication)._
|
see [SQL Server ADO authentication](/influxdb/v2.0/reference/flux/stdlib/sql/from/#sql-server-ado-authentication)._
|
||||||
{{% /code-tab-content %}}
|
{{% /code-tab-content %}}
|
||||||
|
|
||||||
|
{{% code-tab-content %}}
|
||||||
|
```js
|
||||||
|
import "sql"
|
||||||
|
sql.from(
|
||||||
|
driverName: "awsathena",
|
||||||
|
dataSourceName: "s3://myorgqueryresults/?accessID=12ab34cd56ef®ion=region-name&secretAccessKey=y0urSup3rs3crEtT0k3n",
|
||||||
|
query: "GO SELECT * FROM Example.Table"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
_For information about parameters to include in the Athena DSN,
|
||||||
|
see [Athena connection string](/influxdb/v2.0/reference/flux/stdlib/sql/from/#athena-connection-string)._
|
||||||
|
{{% /code-tab-content %}}
|
||||||
|
{{% code-tab-content %}}
|
||||||
|
```js
|
||||||
|
import "sql"
|
||||||
|
sql.from(
|
||||||
|
driverName: "bigquery",
|
||||||
|
dataSourceName: "bigquery://projectid/?apiKey=mySuP3r5ecR3tAP1K3y",
|
||||||
|
query: "SELECT * FROM exampleTable"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
_For information about authenticating with BigQuery, see
|
||||||
|
[BigQuery authentication parameters](/influxdb/v2.0/reference/flux/stdlib/sql/from/#bigquery-authentication-parameters)._
|
||||||
|
{{% /code-tab-content %}}
|
||||||
{{< /code-tabs-wrapper >}}
|
{{< /code-tabs-wrapper >}}
|
||||||
|
|
||||||
_See the [`sql.from()` documentation](/influxdb/v2.0/reference/flux/stdlib/sql/from/) for
|
_See the [`sql.from()` documentation](/influxdb/v2.0/reference/flux/stdlib/sql/from/) for
|
||||||
|
|
|
@ -3,7 +3,8 @@ title: Flux SQL package
|
||||||
list_title: SQL package
|
list_title: SQL package
|
||||||
description: >
|
description: >
|
||||||
The Flux SQL package provides tools for working with data in SQL databases such
|
The Flux SQL package provides tools for working with data in SQL databases such
|
||||||
as MySQL, PostgreSQL, Snowflake, SQLite, Microsoft SQL Server, and Amazon Athena.
|
as MySQL, PostgreSQL, Snowflake, SQLite, Microsoft SQL Server, Amazon Athena,
|
||||||
|
and Google BigQuery.
|
||||||
Import the `sql` package.
|
Import the `sql` package.
|
||||||
aliases:
|
aliases:
|
||||||
- /influxdb/v2.0/reference/flux/functions/sql/
|
- /influxdb/v2.0/reference/flux/functions/sql/
|
||||||
|
@ -20,6 +21,7 @@ related:
|
||||||
SQL Flux functions provide tools for working with data in SQL databases such as:
|
SQL Flux functions provide tools for working with data in SQL databases such as:
|
||||||
|
|
||||||
- Amazon Athena
|
- Amazon Athena
|
||||||
|
- Google BigQuery
|
||||||
- Microsoft SQL Server
|
- Microsoft SQL Server
|
||||||
- MySQL
|
- MySQL
|
||||||
- PostgreSQL
|
- PostgreSQL
|
||||||
|
|
|
@ -36,6 +36,7 @@ _**Data type:** String_
|
||||||
The following drivers are available:
|
The following drivers are available:
|
||||||
|
|
||||||
- awsathena
|
- awsathena
|
||||||
|
- bigquery
|
||||||
- mysql
|
- mysql
|
||||||
- postgres
|
- postgres
|
||||||
- snowflake
|
- snowflake
|
||||||
|
@ -73,6 +74,10 @@ sqlserver://username:password@localhost:1234?database=examplebdb
|
||||||
server=localhost;user id=username;database=examplebdb;
|
server=localhost;user id=username;database=examplebdb;
|
||||||
server=localhost;user id=username;database=examplebdb;azure auth=ENV
|
server=localhost;user id=username;database=examplebdb;azure auth=ENV
|
||||||
server=localhost;user id=username;database=examplebdbr;azure tenant id=77e7d537;azure client id=58879ce8;azure client secret=0123456789
|
server=localhost;user id=username;database=examplebdbr;azure tenant id=77e7d537;azure client id=58879ce8;azure client secret=0123456789
|
||||||
|
|
||||||
|
# Google BigQuery DSNs
|
||||||
|
bigquery://projectid/?param1=value¶m2=value
|
||||||
|
bigquery://projectid/location?param1=value¶m2=value
|
||||||
```
|
```
|
||||||
|
|
||||||
### query
|
### query
|
||||||
|
@ -88,6 +93,7 @@ _**Data type:** String_
|
||||||
- [SQLite](#query-an-sqlite-database)
|
- [SQLite](#query-an-sqlite-database)
|
||||||
- [Amazon Athena](#query-an-amazon-athena-database)
|
- [Amazon Athena](#query-an-amazon-athena-database)
|
||||||
- [SQL Server](#query-a-sql-server-database)
|
- [SQL Server](#query-a-sql-server-database)
|
||||||
|
- [Google BigQuery](#query-a-bigquery-database)
|
||||||
|
|
||||||
{{% note %}}
|
{{% note %}}
|
||||||
The examples below use [InfluxDB secrets](/influxdb/v2.0/security/secrets/) to populate
|
The examples below use [InfluxDB secrets](/influxdb/v2.0/security/secrets/) to populate
|
||||||
|
@ -250,3 +256,40 @@ _For information about managed identities, see [Microsoft managed identities](ht
|
||||||
```
|
```
|
||||||
azure auth=MSI
|
azure auth=MSI
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Query a BigQuery database
|
||||||
|
```js
|
||||||
|
import "sql"
|
||||||
|
import "influxdata/influxdb/secrets"
|
||||||
|
projectID = secrets.get(key: "BIGQUERY_PROJECT_ID")
|
||||||
|
apiKey = secrets.get(key: "BIGQUERY_APIKEY")
|
||||||
|
sql.from(
|
||||||
|
driverName: "bigquery",
|
||||||
|
dataSourceName: "bigquery://${projectID}/?apiKey=${apiKey}",
|
||||||
|
query:"SELECT * FROM exampleTable"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Common BigQuery URL parameters
|
||||||
|
- **dataset** - BigQuery dataset ID. When set, you can use unqualified table names in queries.
|
||||||
|
|
||||||
|
#### BigQuery authentication parameters
|
||||||
|
The Flux BigQuery implementation uses the Google Cloud Go SDK.
|
||||||
|
Provide your authentication credentials using one of the following methods:
|
||||||
|
|
||||||
|
- The `GOOGLE_APPLICATION_CREDENTIALS` environment variable that identifies the
|
||||||
|
location of your credential JSON file.
|
||||||
|
- Provide your BigQuery API key using the **apiKey** URL parameter in your BigQuery DSN.
|
||||||
|
|
||||||
|
###### Example apiKey URL parameter
|
||||||
|
```
|
||||||
|
bigquery://projectid/?apiKey=AIzaSyB6XK8IO5AzKZXoioQOVNTFYzbDBjY5hy4
|
||||||
|
```
|
||||||
|
|
||||||
|
- Provide your base-64 encoded service account, refresh token, or JSON credentials
|
||||||
|
using the **credentials** URL parameter in your BigQuery DSN.
|
||||||
|
|
||||||
|
###### Example credentials URL parameter
|
||||||
|
```
|
||||||
|
bigquery://projectid/?credentials=eyJ0eXBlIjoiYXV0...
|
||||||
|
```
|
||||||
|
|
|
@ -34,12 +34,18 @@ _**Data type:** String_
|
||||||
|
|
||||||
The following drivers are available:
|
The following drivers are available:
|
||||||
|
|
||||||
|
- bigquery
|
||||||
- mysql
|
- mysql
|
||||||
- postgres
|
- postgres
|
||||||
- snowflake
|
- snowflake
|
||||||
- sqlite3 – _Does not work with InfluxDB OSS or InfluxDB Cloud. More information [below](#write-data-to-an-sqlite-database)._
|
- sqlite3 – _Does not work with InfluxDB OSS or InfluxDB Cloud. More information [below](#write-data-to-an-sqlite-database)._
|
||||||
- sqlserver, mssql
|
- sqlserver, mssql
|
||||||
|
|
||||||
|
{{% warn %}}
|
||||||
|
#### sql.to does not support Amazon Athena
|
||||||
|
The `sql.to` function does not support writing data to [Amazon Athena](https://aws.amazon.com/athena/).
|
||||||
|
{{% /warn %}}
|
||||||
|
|
||||||
### dataSourceName
|
### dataSourceName
|
||||||
The data source name (DSN) or connection string used to connect to the SQL database.
|
The data source name (DSN) or connection string used to connect to the SQL database.
|
||||||
The string's form and structure depend on the [driver](#drivername) used.
|
The string's form and structure depend on the [driver](#drivername) used.
|
||||||
|
@ -67,6 +73,10 @@ sqlserver://username:password@localhost:1234?database=examplebdb
|
||||||
server=localhost;user id=username;database=examplebdb;
|
server=localhost;user id=username;database=examplebdb;
|
||||||
server=localhost;user id=username;database=examplebdb;azure auth=ENV
|
server=localhost;user id=username;database=examplebdb;azure auth=ENV
|
||||||
server=localhost;user id=username;database=examplebdbr;azure tenant id=77e7d537;azure client id=58879ce8;azure client secret=0123456789
|
server=localhost;user id=username;database=examplebdbr;azure tenant id=77e7d537;azure client id=58879ce8;azure client secret=0123456789
|
||||||
|
|
||||||
|
# Google BigQuery DSNs
|
||||||
|
bigquery://projectid/?param1=value¶m2=value
|
||||||
|
bigquery://projectid/location?param1=value¶m2=value
|
||||||
```
|
```
|
||||||
|
|
||||||
### table
|
### table
|
||||||
|
@ -91,6 +101,7 @@ If writing to a **SQLite** database, set `batchSize` to `999` or less.
|
||||||
- [Snowflake](#write-data-to-a-snowflake-database)
|
- [Snowflake](#write-data-to-a-snowflake-database)
|
||||||
- [SQLite](#write-data-to-an-sqlite-database)
|
- [SQLite](#write-data-to-an-sqlite-database)
|
||||||
- [SQL Server](#write-data-to-a-sql-server-database)
|
- [SQL Server](#write-data-to-a-sql-server-database)
|
||||||
|
- [Google BigQuery](#write-data-to-a-sql-server-database)
|
||||||
|
|
||||||
{{% note %}}
|
{{% note %}}
|
||||||
The examples below use [InfluxDB secrets](/influxdb/v2.0/security/secrets/) to populate
|
The examples below use [InfluxDB secrets](/influxdb/v2.0/security/secrets/) to populate
|
||||||
|
@ -223,7 +234,39 @@ _For information about managed identities, see [Microsoft managed identities](ht
|
||||||
azure auth=MSI
|
azure auth=MSI
|
||||||
```
|
```
|
||||||
|
|
||||||
{{% warn %}}
|
### Write to a BigQuery database
|
||||||
### sql.to does not support Amazon Athena
|
```js
|
||||||
The `sql.to` function does not support writing data to [Amazon Athena](https://aws.amazon.com/athena/).
|
import "sql"
|
||||||
{{% /warn %}}
|
import "influxdata/influxdb/secrets"
|
||||||
|
projectID = secrets.get(key: "BIGQUERY_PROJECT_ID")
|
||||||
|
apiKey = secrets.get(key: "BIGQUERY_APIKEY")
|
||||||
|
sql.to(
|
||||||
|
driverName: "bigquery",
|
||||||
|
dataSourceName: "bigquery://${projectID}/?apiKey=${apiKey}",
|
||||||
|
table:"exampleTable"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Common BigQuery URL parameters
|
||||||
|
- **dataset** - BigQuery dataset ID. When set, you can use unqualified table names in queries.
|
||||||
|
|
||||||
|
#### BigQuery authentication parameters
|
||||||
|
The Flux BigQuery implementation uses the Google Cloud Go SDK.
|
||||||
|
Provide your authentication credentials using one of the following methods:
|
||||||
|
|
||||||
|
- The `GOOGLE_APPLICATION_CREDENTIALS` environment variable that identifies the
|
||||||
|
location of your credential JSON file.
|
||||||
|
- Provide your BigQuery API key using the **apiKey** URL parameter in your BigQuery DSN.
|
||||||
|
|
||||||
|
###### Example apiKey URL parameter
|
||||||
|
```
|
||||||
|
bigquery://projectid/?apiKey=AIzaSyB6XK8IO5AzKZXoioQOVNTFYzbDBjY5hy4
|
||||||
|
```
|
||||||
|
|
||||||
|
- Provide your base-64 encoded service account, refresh token, or JSON credentials
|
||||||
|
using the **credentials** URL parameter in your BigQuery DSN.
|
||||||
|
|
||||||
|
###### Example credentials URL parameter
|
||||||
|
```
|
||||||
|
bigquery://projectid/?credentials=eyJ0eXBlIjoiYXV0...
|
||||||
|
```
|
||||||
|
|
|
@ -25,7 +25,8 @@ InfluxDB until the next InfluxDB v2.0 release._
|
||||||
|
|
||||||
### Features
|
### Features
|
||||||
- Improve window errors.
|
- Improve window errors.
|
||||||
- Add BigQuery support to [`sql` package](#).
|
- Add [BigQuery](https://cloud.google.com/bigquery) support to
|
||||||
|
[`sql` package](/influxdb/v2.0/reference/flux/stdlib/sql/).
|
||||||
- Add `TypeExpression` to `BuiltinStmt` and fix tests.
|
- Add `TypeExpression` to `BuiltinStmt` and fix tests.
|
||||||
- Time-weighted average ([`timeWeightedAvg()` function](/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/timeweightedavg/)).
|
- Time-weighted average ([`timeWeightedAvg()` function](/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/timeweightedavg/)).
|
||||||
- Update [`integral()`](/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/integral/)
|
- Update [`integral()`](/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/integral/)
|
||||||
|
|
Loading…
Reference in New Issue