From 75b7fdd00a8bacd134dc1ec1cef6cbb53264635c Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Thu, 3 Sep 2020 15:26:13 -0600 Subject: [PATCH] added BigQuery support to sql package --- content/influxdb/v2.0/query-data/flux/sql.md | 35 +++++++++++-- .../v2.0/reference/flux/stdlib/sql/_index.md | 4 +- .../v2.0/reference/flux/stdlib/sql/from.md | 43 ++++++++++++++++ .../v2.0/reference/flux/stdlib/sql/to.md | 51 +++++++++++++++++-- .../v2.0/reference/release-notes/flux.md | 3 +- 5 files changed, 127 insertions(+), 9 deletions(-) diff --git a/content/influxdb/v2.0/query-data/flux/sql.md b/content/influxdb/v2.0/query-data/flux/sql.md index 21a45b055..1da75b3f3 100644 --- a/content/influxdb/v2.0/query-data/flux/sql.md +++ b/content/influxdb/v2.0/query-data/flux/sql.md @@ -5,7 +5,7 @@ list_title: Query SQL data description: > The Flux `sql` package provides functions for working with SQL data sources. Use `sql.from()` to query SQL databases like PostgreSQL, MySQL, Snowflake, - SQLite, Microsoft SQL Server, and Amazon Athena. + SQLite, Microsoft SQL Server, Amazon Athena, and Google BigQuery. influxdb/v2.0/tags: [query, flux, sql] menu: influxdb_2_0: @@ -33,8 +33,8 @@ The [Flux](/influxdb/v2.0/reference/flux) `sql` package provides functions for w like [PostgreSQL](https://www.postgresql.org/), [MySQL](https://www.mysql.com/), [Snowflake](https://www.snowflake.com/), [SQLite](https://www.sqlite.org/index.html), [Microsoft SQL Server](https://www.microsoft.com/en-us/sql-server/default.aspx), -and [Amazon Athena](https://aws.amazon.com/athena/) and use the results with -InfluxDB dashboards, tasks, and other operations. +[Amazon Athena](https://aws.amazon.com/athena/) and [Google BigQuery](https://cloud.google.com/bigquery) +and use the results with InfluxDB dashboards, tasks, and other operations. - [Query a SQL data source](#query-a-sql-data-source) - [Join SQL data with data in InfluxDB](#join-sql-data-with-data-in-influxdb) @@ -61,6 +61,8 @@ To query a SQL data source: [Snowflake](#) [SQLite](#) [SQL Server](#) +[Athena](#) +[BigQuery](#) {{% /code-tabs %}} {{% code-tab-content %}} @@ -129,6 +131,33 @@ sql.from( _For information about authenticating with SQL Server using ADO-style parameters, see [SQL Server ADO authentication](/influxdb/v2.0/reference/flux/stdlib/sql/from/#sql-server-ado-authentication)._ {{% /code-tab-content %}} + +{{% code-tab-content %}} +```js +import "sql" +sql.from( + driverName: "awsathena", + dataSourceName: "s3://myorgqueryresults/?accessID=12ab34cd56ef®ion=region-name&secretAccessKey=y0urSup3rs3crEtT0k3n", + query: "GO SELECT * FROM Example.Table" +) +``` + +_For information about parameters to include in the Athena DSN, +see [Athena connection string](/influxdb/v2.0/reference/flux/stdlib/sql/from/#athena-connection-string)._ +{{% /code-tab-content %}} +{{% code-tab-content %}} +```js +import "sql" +sql.from( + driverName: "bigquery", + dataSourceName: "bigquery://projectid/?apiKey=mySuP3r5ecR3tAP1K3y", + query: "SELECT * FROM exampleTable" +) +``` + +_For information about authenticating with BigQuery, see +[BigQuery authentication parameters](/influxdb/v2.0/reference/flux/stdlib/sql/from/#bigquery-authentication-parameters)._ +{{% /code-tab-content %}} {{< /code-tabs-wrapper >}} _See the [`sql.from()` documentation](/influxdb/v2.0/reference/flux/stdlib/sql/from/) for diff --git a/content/influxdb/v2.0/reference/flux/stdlib/sql/_index.md b/content/influxdb/v2.0/reference/flux/stdlib/sql/_index.md index a1a2f6875..b274e9290 100644 --- a/content/influxdb/v2.0/reference/flux/stdlib/sql/_index.md +++ b/content/influxdb/v2.0/reference/flux/stdlib/sql/_index.md @@ -3,7 +3,8 @@ title: Flux SQL package list_title: SQL package description: > The Flux SQL package provides tools for working with data in SQL databases such - as MySQL, PostgreSQL, Snowflake, SQLite, Microsoft SQL Server, and Amazon Athena. + as MySQL, PostgreSQL, Snowflake, SQLite, Microsoft SQL Server, Amazon Athena, + and Google BigQuery. Import the `sql` package. aliases: - /influxdb/v2.0/reference/flux/functions/sql/ @@ -20,6 +21,7 @@ related: SQL Flux functions provide tools for working with data in SQL databases such as: - Amazon Athena +- Google BigQuery - Microsoft SQL Server - MySQL - PostgreSQL diff --git a/content/influxdb/v2.0/reference/flux/stdlib/sql/from.md b/content/influxdb/v2.0/reference/flux/stdlib/sql/from.md index 54f59b3d8..5babdafc1 100644 --- a/content/influxdb/v2.0/reference/flux/stdlib/sql/from.md +++ b/content/influxdb/v2.0/reference/flux/stdlib/sql/from.md @@ -36,6 +36,7 @@ _**Data type:** String_ The following drivers are available: - awsathena +- bigquery - mysql - postgres - snowflake @@ -73,6 +74,10 @@ sqlserver://username:password@localhost:1234?database=examplebdb server=localhost;user id=username;database=examplebdb; server=localhost;user id=username;database=examplebdb;azure auth=ENV server=localhost;user id=username;database=examplebdbr;azure tenant id=77e7d537;azure client id=58879ce8;azure client secret=0123456789 + +# Google BigQuery DSNs +bigquery://projectid/?param1=value¶m2=value +bigquery://projectid/location?param1=value¶m2=value ``` ### query @@ -88,6 +93,7 @@ _**Data type:** String_ - [SQLite](#query-an-sqlite-database) - [Amazon Athena](#query-an-amazon-athena-database) - [SQL Server](#query-a-sql-server-database) +- [Google BigQuery](#query-a-bigquery-database) {{% note %}} The examples below use [InfluxDB secrets](/influxdb/v2.0/security/secrets/) to populate @@ -250,3 +256,40 @@ _For information about managed identities, see [Microsoft managed identities](ht ``` azure auth=MSI ``` + +### Query a BigQuery database +```js +import "sql" +import "influxdata/influxdb/secrets" +projectID = secrets.get(key: "BIGQUERY_PROJECT_ID") +apiKey = secrets.get(key: "BIGQUERY_APIKEY") +sql.from( + driverName: "bigquery", + dataSourceName: "bigquery://${projectID}/?apiKey=${apiKey}", + query:"SELECT * FROM exampleTable" +) +``` + +#### Common BigQuery URL parameters +- **dataset** - BigQuery dataset ID. When set, you can use unqualified table names in queries. + +#### BigQuery authentication parameters +The Flux BigQuery implementation uses the Google Cloud Go SDK. +Provide your authentication credentials using one of the following methods: + +- The `GOOGLE_APPLICATION_CREDENTIALS` environment variable that identifies the + location of your credential JSON file. +- Provide your BigQuery API key using the **apiKey** URL parameter in your BigQuery DSN. + + ###### Example apiKey URL parameter + ``` + bigquery://projectid/?apiKey=AIzaSyB6XK8IO5AzKZXoioQOVNTFYzbDBjY5hy4 + ``` + +- Provide your base-64 encoded service account, refresh token, or JSON credentials + using the **credentials** URL parameter in your BigQuery DSN. + + ###### Example credentials URL parameter + ``` + bigquery://projectid/?credentials=eyJ0eXBlIjoiYXV0... + ``` diff --git a/content/influxdb/v2.0/reference/flux/stdlib/sql/to.md b/content/influxdb/v2.0/reference/flux/stdlib/sql/to.md index 8f66a6afc..c06fec1ff 100644 --- a/content/influxdb/v2.0/reference/flux/stdlib/sql/to.md +++ b/content/influxdb/v2.0/reference/flux/stdlib/sql/to.md @@ -34,12 +34,18 @@ _**Data type:** String_ The following drivers are available: +- bigquery - mysql - postgres - snowflake - sqlite3 – _Does not work with InfluxDB OSS or InfluxDB Cloud. More information [below](#write-data-to-an-sqlite-database)._ - sqlserver, mssql +{{% warn %}} +#### sql.to does not support Amazon Athena +The `sql.to` function does not support writing data to [Amazon Athena](https://aws.amazon.com/athena/). +{{% /warn %}} + ### dataSourceName The data source name (DSN) or connection string used to connect to the SQL database. The string's form and structure depend on the [driver](#drivername) used. @@ -67,6 +73,10 @@ sqlserver://username:password@localhost:1234?database=examplebdb server=localhost;user id=username;database=examplebdb; server=localhost;user id=username;database=examplebdb;azure auth=ENV server=localhost;user id=username;database=examplebdbr;azure tenant id=77e7d537;azure client id=58879ce8;azure client secret=0123456789 + +# Google BigQuery DSNs +bigquery://projectid/?param1=value¶m2=value +bigquery://projectid/location?param1=value¶m2=value ``` ### table @@ -91,6 +101,7 @@ If writing to a **SQLite** database, set `batchSize` to `999` or less. - [Snowflake](#write-data-to-a-snowflake-database) - [SQLite](#write-data-to-an-sqlite-database) - [SQL Server](#write-data-to-a-sql-server-database) +- [Google BigQuery](#write-data-to-a-sql-server-database) {{% note %}} The examples below use [InfluxDB secrets](/influxdb/v2.0/security/secrets/) to populate @@ -223,7 +234,39 @@ _For information about managed identities, see [Microsoft managed identities](ht azure auth=MSI ``` -{{% warn %}} -### sql.to does not support Amazon Athena -The `sql.to` function does not support writing data to [Amazon Athena](https://aws.amazon.com/athena/). -{{% /warn %}} +### Write to a BigQuery database +```js +import "sql" +import "influxdata/influxdb/secrets" +projectID = secrets.get(key: "BIGQUERY_PROJECT_ID") +apiKey = secrets.get(key: "BIGQUERY_APIKEY") +sql.to( + driverName: "bigquery", + dataSourceName: "bigquery://${projectID}/?apiKey=${apiKey}", + table:"exampleTable" +) +``` + +#### Common BigQuery URL parameters +- **dataset** - BigQuery dataset ID. When set, you can use unqualified table names in queries. + +#### BigQuery authentication parameters +The Flux BigQuery implementation uses the Google Cloud Go SDK. +Provide your authentication credentials using one of the following methods: + +- The `GOOGLE_APPLICATION_CREDENTIALS` environment variable that identifies the + location of your credential JSON file. +- Provide your BigQuery API key using the **apiKey** URL parameter in your BigQuery DSN. + + ###### Example apiKey URL parameter + ``` + bigquery://projectid/?apiKey=AIzaSyB6XK8IO5AzKZXoioQOVNTFYzbDBjY5hy4 + ``` + +- Provide your base-64 encoded service account, refresh token, or JSON credentials + using the **credentials** URL parameter in your BigQuery DSN. + + ###### Example credentials URL parameter + ``` + bigquery://projectid/?credentials=eyJ0eXBlIjoiYXV0... + ``` diff --git a/content/influxdb/v2.0/reference/release-notes/flux.md b/content/influxdb/v2.0/reference/release-notes/flux.md index 6f983cb59..e918fd559 100644 --- a/content/influxdb/v2.0/reference/release-notes/flux.md +++ b/content/influxdb/v2.0/reference/release-notes/flux.md @@ -25,7 +25,8 @@ InfluxDB until the next InfluxDB v2.0 release._ ### Features - Improve window errors. -- Add BigQuery support to [`sql` package](#). +- Add [BigQuery](https://cloud.google.com/bigquery) support to + [`sql` package](/influxdb/v2.0/reference/flux/stdlib/sql/). - Add `TypeExpression` to `BuiltinStmt` and fix tests. - Time-weighted average ([`timeWeightedAvg()` function](/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/timeweightedavg/)). - Update [`integral()`](/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/integral/)