From fee62e27e79730b8cb03e4cbfd7fecc163755f0e Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Thu, 3 Sep 2020 10:41:19 -0600 Subject: [PATCH 01/10] added release notes for flux 0.82-0.83 --- .../v2.0/reference/release-notes/flux.md | 50 ++++++++++++++++++- 1 file changed, 49 insertions(+), 1 deletion(-) diff --git a/content/influxdb/v2.0/reference/release-notes/flux.md b/content/influxdb/v2.0/reference/release-notes/flux.md index d99cfcac3..72a96c364 100644 --- a/content/influxdb/v2.0/reference/release-notes/flux.md +++ b/content/influxdb/v2.0/reference/release-notes/flux.md @@ -9,11 +9,59 @@ menu: --- {{% note %}} -The latest release of InfluxDB v2.0 beta includes **Flux v0.77.1**. +_The latest release of InfluxDB v2.0 beta includes **Flux v0.77.1**. Though newer versions of Flux may be available, they will not be included with InfluxDB until the next InfluxDB v2.0 release._ {{% /note %}} +## v0.83.1 [2020-09-02] + +### Bug fixes +- Single value integral interpolation. + +--- + +## v0.83.0 [2020-09-01] + +### Features +- Improve window errors. +- Add BigQuery support to [`sql` package](#). +- Add `TypeExpression` to `BuiltinStmt` and fix tests. +- Time-weighted average ([`timeWeightedAvg()` function](#)). +- Integration with linear interpolation. +- Make experimental tracing an attribute of the context. + +### Bug fixes +- Update builtin statement for `integral`. +- Add Rust JSON tests. +- CSV no longer deadlocks when next transformation does not consume table. + +--- + +## v0.82.2 [2020-08-25] + +### Features +- Add `tasks.lastSuccess` function to retrieve the last successful run of an InfluxDB task. + +--- + +## v0.82.1 [2020-08-25] +- _Internal code cleanup._ + +--- + +## v0.82.0 [2020-08-24] + +### Features +- Add the [`profiler` package](#). +- Add a documentation URL field to Flux errors. +- Check InfluxDB schema compatibility. + +### Bug fixes +- Panic when a map object property contains an invalid type. + +--- + ## v0.81.0 [2020-08-17] ### Features From 729845c9afde68d635312feab1d3e272114f0b2d Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Thu, 3 Sep 2020 10:53:27 -0600 Subject: [PATCH 02/10] added the flux profiler package --- .../reference/flux/stdlib/profiler/_index.md | 38 +++++++++++++++++++ .../v2.0/reference/release-notes/flux.md | 2 +- 2 files changed, 39 insertions(+), 1 deletion(-) create mode 100644 content/influxdb/v2.0/reference/flux/stdlib/profiler/_index.md diff --git a/content/influxdb/v2.0/reference/flux/stdlib/profiler/_index.md b/content/influxdb/v2.0/reference/flux/stdlib/profiler/_index.md new file mode 100644 index 000000000..5325c4379 --- /dev/null +++ b/content/influxdb/v2.0/reference/flux/stdlib/profiler/_index.md @@ -0,0 +1,38 @@ +--- +title: Flux Profiler package +list_title: Profiler package +description: > + The Flux Profiler package provides performance profiling tools for Flux queries and operations. + Import the `profiler` package. +menu: + influxdb_2_0_ref: + name: Profiler + parent: Flux standard library +weight: 202 +influxdb/v2.0/tags: [functions, optimize, package] +--- + +The Flux Profiler package provides performance profiling tools for Flux queries and operations. +Import the `profiler` package: + +```js +import "profiler" +``` + +## Options +The Profiler package includes the following options: + +#### enabledProfilers +Use the `enabledProfilers` option to enable Flux profilers. + +{{% note %}} +_Available profilers will be added as they are released._ +{{% /note %}} + +_**Data type:** Array of strings_ + +```js +import "profiler" + +option profiler.enabledProfilers = [""] +``` \ No newline at end of file diff --git a/content/influxdb/v2.0/reference/release-notes/flux.md b/content/influxdb/v2.0/reference/release-notes/flux.md index 72a96c364..649d1148d 100644 --- a/content/influxdb/v2.0/reference/release-notes/flux.md +++ b/content/influxdb/v2.0/reference/release-notes/flux.md @@ -41,7 +41,7 @@ InfluxDB until the next InfluxDB v2.0 release._ ## v0.82.2 [2020-08-25] ### Features -- Add `tasks.lastSuccess` function to retrieve the last successful run of an InfluxDB task. +- Add [`tasks.lastSuccess` function](#) to retrieve the last successful run of an InfluxDB task. --- From a47b2e0111e20819a1c376c158640d8ae2fae0b8 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Thu, 3 Sep 2020 10:55:33 -0600 Subject: [PATCH 03/10] added link to profiler package in flux release notes --- content/influxdb/v2.0/reference/release-notes/flux.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/influxdb/v2.0/reference/release-notes/flux.md b/content/influxdb/v2.0/reference/release-notes/flux.md index 649d1148d..6daeb6ed3 100644 --- a/content/influxdb/v2.0/reference/release-notes/flux.md +++ b/content/influxdb/v2.0/reference/release-notes/flux.md @@ -53,7 +53,7 @@ InfluxDB until the next InfluxDB v2.0 release._ ## v0.82.0 [2020-08-24] ### Features -- Add the [`profiler` package](#). +- Add the [`profiler` package](/influxdb/v2.0/reference/flux/stdlib/profiler/). - Add a documentation URL field to Flux errors. - Check InfluxDB schema compatibility. From 97429b4b0f0e99a4cdc307093618de21535b5371 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Thu, 3 Sep 2020 11:25:58 -0600 Subject: [PATCH 04/10] added flux influxdb tasks package --- .../flux/stdlib/influxdb-tasks/_index.md | 44 +++++++++++++++++++ .../flux/stdlib/influxdb-tasks/lastsuccess.md | 41 +++++++++++++++++ .../reference/flux/stdlib/profiler/_index.md | 4 +- .../v2.0/reference/release-notes/flux.md | 3 +- 4 files changed, 90 insertions(+), 2 deletions(-) create mode 100644 content/influxdb/v2.0/reference/flux/stdlib/influxdb-tasks/_index.md create mode 100644 content/influxdb/v2.0/reference/flux/stdlib/influxdb-tasks/lastsuccess.md diff --git a/content/influxdb/v2.0/reference/flux/stdlib/influxdb-tasks/_index.md b/content/influxdb/v2.0/reference/flux/stdlib/influxdb-tasks/_index.md new file mode 100644 index 000000000..be8476054 --- /dev/null +++ b/content/influxdb/v2.0/reference/flux/stdlib/influxdb-tasks/_index.md @@ -0,0 +1,44 @@ +--- +title: Flux InfluxDB Tasks package +list_title: InfluxDB Tasks package +description: > + The Flux InfluxDB Tasks package provides options and functions for working with + [InfluxDB tasks](/influxdb/v2.0/process-data/get-started/). + Import the `influxdata/influxdb/tasks` package. +aliases: + - /influxdb/v2.0/reference/flux/functions/influxdb-v1/ +menu: + influxdb_2_0_ref: + name: InfluxDB Tasks + parent: Flux standard library +weight: 202 +influxdb/v2.0/tags: [functions, tasks, package] +related: + - /influxdb/v2.0/process-data/get-started/ +--- + +The Flux InfluxDB Tasks package provides options and functions for working with +[InfluxDB tasks](/influxdb/v2.0/process-data/get-started/). +Import the `influxdata/influxdb/tasks` package: + +```js +import "influxdata/influxdb/tasks" +``` + +## Options +The InfluxDB Tasks package provides the following options: + +#### lastSuccessTime +Define the time of the last successful task run. +_Only use this option to override the time of the last successful run provided by +the InfluxDB task engine._ + +```js +import "influxdata/influxdb/tasks" + +option tasks.lastSuccessTime = 0000-01-01T00:00:00Z +``` + +## Functions + +{{< children type="functions" show="pages" >}} diff --git a/content/influxdb/v2.0/reference/flux/stdlib/influxdb-tasks/lastsuccess.md b/content/influxdb/v2.0/reference/flux/stdlib/influxdb-tasks/lastsuccess.md new file mode 100644 index 000000000..8a2b0d9bb --- /dev/null +++ b/content/influxdb/v2.0/reference/flux/stdlib/influxdb-tasks/lastsuccess.md @@ -0,0 +1,41 @@ +--- +title: tasks.lastSuccess() function +description: The `tasks.lastSuccess()` function returns ... +menu: + influxdb_2_0_ref: + name: tasks.lastSuccess + parent: InfluxDB Tasks +weight: 301 +--- + +The `tasks.lastSuccess()` function returns the time of last successful run of the +InfluxDB task or the value of the `orTime` parameter if the task has never successfully run. + +```js +import "influxdata/influxdb/tasks" + +tasks.lastSuccess(orTime: 2020-01-01T00:00:00Z) +``` + +## Parameters + +### orTime +The default time value returned if the task has never successfully run. + +_**Data type:** Time_ + +## Examples + +##### Query data since that last successful task run +```js +import "influxdata/influxdb/tasks" + +options task = { + name: "Example task", + every: 30m +} + +from(bucket: "example-bucket") + |> range(start: tasks.lastSuccess(orTime: 2020-01-01T00:00:00Z)) + // ... +``` \ No newline at end of file diff --git a/content/influxdb/v2.0/reference/flux/stdlib/profiler/_index.md b/content/influxdb/v2.0/reference/flux/stdlib/profiler/_index.md index 5325c4379..eda8a643a 100644 --- a/content/influxdb/v2.0/reference/flux/stdlib/profiler/_index.md +++ b/content/influxdb/v2.0/reference/flux/stdlib/profiler/_index.md @@ -10,6 +10,8 @@ menu: parent: Flux standard library weight: 202 influxdb/v2.0/tags: [functions, optimize, package] +related: + - /influxdb/v2.0/query-data/optimize-queries/ --- The Flux Profiler package provides performance profiling tools for Flux queries and operations. @@ -23,7 +25,7 @@ import "profiler" The Profiler package includes the following options: #### enabledProfilers -Use the `enabledProfilers` option to enable Flux profilers. +Enable Flux profilers. {{% note %}} _Available profilers will be added as they are released._ diff --git a/content/influxdb/v2.0/reference/release-notes/flux.md b/content/influxdb/v2.0/reference/release-notes/flux.md index 6daeb6ed3..2b78efba1 100644 --- a/content/influxdb/v2.0/reference/release-notes/flux.md +++ b/content/influxdb/v2.0/reference/release-notes/flux.md @@ -41,7 +41,8 @@ InfluxDB until the next InfluxDB v2.0 release._ ## v0.82.2 [2020-08-25] ### Features -- Add [`tasks.lastSuccess` function](#) to retrieve the last successful run of an InfluxDB task. +- Add [`tasks.lastSuccess` function](/influxdb/v2.0/reference/flux/stdlib/influxdb-tasks/lastsuccess/) + to retrieve the time of the last successful run of an InfluxDB task. --- From 269581842ad8fb851104767cea8ef932869ae7ff Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Thu, 3 Sep 2020 14:57:57 -0600 Subject: [PATCH 05/10] updated flux integral with timeColumn and interpolate params --- .../transformations/aggregates/integral.md | 40 +++++++++++++++++-- .../reference/flux/stdlib/profiler/_index.md | 38 +++++++++++++++--- .../v2.0/reference/release-notes/flux.md | 4 +- 3 files changed, 71 insertions(+), 11 deletions(-) diff --git a/content/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/integral.md b/content/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/integral.md index 090579c14..04ab2b672 100644 --- a/content/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/integral.md +++ b/content/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/integral.md @@ -20,23 +20,46 @@ _**Function type:** Aggregate_ _**Output data type:** Float_ ```js -integral(unit: 10s, column: "_value") +integral( + unit: 10s, + column: "_value", + timeColumn: "_time", + interpolation: "" +) ``` ## Parameters ### unit -The time duration used when computing the integral. +Time duration used when computing the integral. _**Data type:** Duration_ ### column -The column on which to operate. +Column on which to operate. Defaults to `"_value"`. _**Data type:** String_ +### timeColumn +Column that contains time values to use in the operation. +Defaults to `"_time"`. + +_**Data type:** String_ + +### interpolate +Type of value interpolation to use. +Defaults to `""`. + +Use one of the following interpolation options: + +- linear + +_**Data type:** String_ + ## Examples + +##### Calculate the integral without interpolation ```js from(bucket: "example-bucket") |> range(start: -5m) @@ -46,3 +69,14 @@ from(bucket: "example-bucket") ) |> integral(unit:10s) ``` + +##### Calculate the integral with linear interpolation +```js +from(bucket: "example-bucket") + |> range(start: -5m) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_system" + ) + |> integral(unit:10s, interpolate: "linear") +``` diff --git a/content/influxdb/v2.0/reference/flux/stdlib/profiler/_index.md b/content/influxdb/v2.0/reference/flux/stdlib/profiler/_index.md index eda8a643a..a210d078c 100644 --- a/content/influxdb/v2.0/reference/flux/stdlib/profiler/_index.md +++ b/content/influxdb/v2.0/reference/flux/stdlib/profiler/_index.md @@ -24,17 +24,43 @@ import "profiler" ## Options The Profiler package includes the following options: -#### enabledProfilers +### enabledProfilers Enable Flux profilers. -{{% note %}} -_Available profilers will be added as they are released._ -{{% /note %}} - _**Data type:** Array of strings_ ```js import "profiler" option profiler.enabledProfilers = [""] -``` \ No newline at end of file +``` + +#### Available profilers + +##### query +The `query` profiler provides statistics about the execution of an entire Flux script. +When enabled, results returned by [`yield()`](/influxdb/v2.0/reference/flux/stdlib/built-in/outputs/yield/) +include a table with the following columns: + +- **TotalDuration**: total query duration in nanoseconds. +- **CompileDuration**: number of nanoseconds spent compiling the query. +- **QueueDuration**: number of nanoseconds spent queueing. +- **RequeueDuration**: number fo nanoseconds spent requeueing. +- **PlanDuration**: number of nanoseconds spent planning the query. +- **ExecuteDuration**: number of nanoseconds spent executing the query. +- **Concurrency**: number of goroutines allocated to process the query. +- **MaxAllocated**: maximum number of bytes the query allocated. +- **TotalAllocated**: total number of bytes the query allocated (includes memory that was freed and then used again). +- **RuntimeErrors**: error messages returned during query execution. +- **flux/query-plan**: Flux query plan. +- **influxdb/scanned-values**: value scanned by InfluxDB. +- **influxdb/scanned-bytes**: number of bytes scanned by InfluxDB. + +#### Use the query profiler to output statistics about query execution +```js +import "profilers" + +option profiler.enabledProfiles["query"] + +// ... Query to profile +``` diff --git a/content/influxdb/v2.0/reference/release-notes/flux.md b/content/influxdb/v2.0/reference/release-notes/flux.md index 2b78efba1..7b8cdf586 100644 --- a/content/influxdb/v2.0/reference/release-notes/flux.md +++ b/content/influxdb/v2.0/reference/release-notes/flux.md @@ -28,11 +28,11 @@ InfluxDB until the next InfluxDB v2.0 release._ - Add BigQuery support to [`sql` package](#). - Add `TypeExpression` to `BuiltinStmt` and fix tests. - Time-weighted average ([`timeWeightedAvg()` function](#)). -- Integration with linear interpolation. +- Integral with linear interpolation. - Make experimental tracing an attribute of the context. ### Bug fixes -- Update builtin statement for `integral`. +- Update builtin statement for `integral()`. - Add Rust JSON tests. - CSV no longer deadlocks when next transformation does not consume table. From 40021cb275c56d7d806e923533805e41e831dfa8 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Thu, 3 Sep 2020 14:58:53 -0600 Subject: [PATCH 06/10] minor updates to flux integral --- .../stdlib/built-in/transformations/aggregates/integral.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/integral.md b/content/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/integral.md index 04ab2b672..c36cbe630 100644 --- a/content/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/integral.md +++ b/content/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/integral.md @@ -48,7 +48,7 @@ Defaults to `"_time"`. _**Data type:** String_ ### interpolate -Type of value interpolation to use. +Type of interpolation to use. Defaults to `""`. Use one of the following interpolation options: @@ -59,7 +59,7 @@ _**Data type:** String_ ## Examples -##### Calculate the integral without interpolation +##### Calculate the integral ```js from(bucket: "example-bucket") |> range(start: -5m) From db6e89ce041136d542bb29eecf025cda9897e978 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Thu, 3 Sep 2020 15:15:40 -0600 Subject: [PATCH 07/10] added timeWeightedAvg function --- .../aggregates/timeweightedavg.md | 53 +++++++++++++++++++ .../v2.0/reference/release-notes/flux.md | 5 +- 2 files changed, 56 insertions(+), 2 deletions(-) create mode 100644 content/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/timeweightedavg.md diff --git a/content/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/timeweightedavg.md b/content/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/timeweightedavg.md new file mode 100644 index 000000000..0b8b4cc1a --- /dev/null +++ b/content/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/timeweightedavg.md @@ -0,0 +1,53 @@ +--- +title: timeWeightedAvg() function +description: The `timeWeightedAvg()` function outputs the timeWeightedAvg of non-null records as a float. +menu: + influxdb_2_0_ref: + name: timeWeightedAvg + parent: built-in-aggregates +weight: 501 +related: + - /influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/integral/ +--- + +The `timeWeightedAvg()` function outputs the time-weighted average of non-null records +in a table as a float. +Time is weighted using the linearly interpolated integral of values in the table. + +_**Function type:** Aggregate_ +_**Output data type:** Float_ + +```js +timeWeightedAvg(unit: "_value") +``` + +## Parameters + +### unit +Time duration used when computing the time-weighted average. + +_**Data type:** Duration_ + +## Examples +```js +from(bucket: "example-bucket") + |> range(start: -5m) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_system" + ) + |> timeWeightedAvg(unit: 1m) +``` + +## Function definition +```js +timeWeightedAvg = (tables=<-, unit) => tables + |> integral( + unit: unit, + interpolate: "linear" + ) + |> map(fn: (r) => ({ + r with + _value: (r._value * float(v: uint(v: unit))) / float(v: int(v: r._stop) - int(v: r._start)) + })) +``` \ No newline at end of file diff --git a/content/influxdb/v2.0/reference/release-notes/flux.md b/content/influxdb/v2.0/reference/release-notes/flux.md index 7b8cdf586..6f983cb59 100644 --- a/content/influxdb/v2.0/reference/release-notes/flux.md +++ b/content/influxdb/v2.0/reference/release-notes/flux.md @@ -27,8 +27,9 @@ InfluxDB until the next InfluxDB v2.0 release._ - Improve window errors. - Add BigQuery support to [`sql` package](#). - Add `TypeExpression` to `BuiltinStmt` and fix tests. -- Time-weighted average ([`timeWeightedAvg()` function](#)). -- Integral with linear interpolation. +- Time-weighted average ([`timeWeightedAvg()` function](/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/timeweightedavg/)). +- Update [`integral()`](/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/integral/) + with linear interpolation. - Make experimental tracing an attribute of the context. ### Bug fixes From 75b7fdd00a8bacd134dc1ec1cef6cbb53264635c Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Thu, 3 Sep 2020 15:26:13 -0600 Subject: [PATCH 08/10] added BigQuery support to sql package --- content/influxdb/v2.0/query-data/flux/sql.md | 35 +++++++++++-- .../v2.0/reference/flux/stdlib/sql/_index.md | 4 +- .../v2.0/reference/flux/stdlib/sql/from.md | 43 ++++++++++++++++ .../v2.0/reference/flux/stdlib/sql/to.md | 51 +++++++++++++++++-- .../v2.0/reference/release-notes/flux.md | 3 +- 5 files changed, 127 insertions(+), 9 deletions(-) diff --git a/content/influxdb/v2.0/query-data/flux/sql.md b/content/influxdb/v2.0/query-data/flux/sql.md index 21a45b055..1da75b3f3 100644 --- a/content/influxdb/v2.0/query-data/flux/sql.md +++ b/content/influxdb/v2.0/query-data/flux/sql.md @@ -5,7 +5,7 @@ list_title: Query SQL data description: > The Flux `sql` package provides functions for working with SQL data sources. Use `sql.from()` to query SQL databases like PostgreSQL, MySQL, Snowflake, - SQLite, Microsoft SQL Server, and Amazon Athena. + SQLite, Microsoft SQL Server, Amazon Athena, and Google BigQuery. influxdb/v2.0/tags: [query, flux, sql] menu: influxdb_2_0: @@ -33,8 +33,8 @@ The [Flux](/influxdb/v2.0/reference/flux) `sql` package provides functions for w like [PostgreSQL](https://www.postgresql.org/), [MySQL](https://www.mysql.com/), [Snowflake](https://www.snowflake.com/), [SQLite](https://www.sqlite.org/index.html), [Microsoft SQL Server](https://www.microsoft.com/en-us/sql-server/default.aspx), -and [Amazon Athena](https://aws.amazon.com/athena/) and use the results with -InfluxDB dashboards, tasks, and other operations. +[Amazon Athena](https://aws.amazon.com/athena/) and [Google BigQuery](https://cloud.google.com/bigquery) +and use the results with InfluxDB dashboards, tasks, and other operations. - [Query a SQL data source](#query-a-sql-data-source) - [Join SQL data with data in InfluxDB](#join-sql-data-with-data-in-influxdb) @@ -61,6 +61,8 @@ To query a SQL data source: [Snowflake](#) [SQLite](#) [SQL Server](#) +[Athena](#) +[BigQuery](#) {{% /code-tabs %}} {{% code-tab-content %}} @@ -129,6 +131,33 @@ sql.from( _For information about authenticating with SQL Server using ADO-style parameters, see [SQL Server ADO authentication](/influxdb/v2.0/reference/flux/stdlib/sql/from/#sql-server-ado-authentication)._ {{% /code-tab-content %}} + +{{% code-tab-content %}} +```js +import "sql" +sql.from( + driverName: "awsathena", + dataSourceName: "s3://myorgqueryresults/?accessID=12ab34cd56ef®ion=region-name&secretAccessKey=y0urSup3rs3crEtT0k3n", + query: "GO SELECT * FROM Example.Table" +) +``` + +_For information about parameters to include in the Athena DSN, +see [Athena connection string](/influxdb/v2.0/reference/flux/stdlib/sql/from/#athena-connection-string)._ +{{% /code-tab-content %}} +{{% code-tab-content %}} +```js +import "sql" +sql.from( + driverName: "bigquery", + dataSourceName: "bigquery://projectid/?apiKey=mySuP3r5ecR3tAP1K3y", + query: "SELECT * FROM exampleTable" +) +``` + +_For information about authenticating with BigQuery, see +[BigQuery authentication parameters](/influxdb/v2.0/reference/flux/stdlib/sql/from/#bigquery-authentication-parameters)._ +{{% /code-tab-content %}} {{< /code-tabs-wrapper >}} _See the [`sql.from()` documentation](/influxdb/v2.0/reference/flux/stdlib/sql/from/) for diff --git a/content/influxdb/v2.0/reference/flux/stdlib/sql/_index.md b/content/influxdb/v2.0/reference/flux/stdlib/sql/_index.md index a1a2f6875..b274e9290 100644 --- a/content/influxdb/v2.0/reference/flux/stdlib/sql/_index.md +++ b/content/influxdb/v2.0/reference/flux/stdlib/sql/_index.md @@ -3,7 +3,8 @@ title: Flux SQL package list_title: SQL package description: > The Flux SQL package provides tools for working with data in SQL databases such - as MySQL, PostgreSQL, Snowflake, SQLite, Microsoft SQL Server, and Amazon Athena. + as MySQL, PostgreSQL, Snowflake, SQLite, Microsoft SQL Server, Amazon Athena, + and Google BigQuery. Import the `sql` package. aliases: - /influxdb/v2.0/reference/flux/functions/sql/ @@ -20,6 +21,7 @@ related: SQL Flux functions provide tools for working with data in SQL databases such as: - Amazon Athena +- Google BigQuery - Microsoft SQL Server - MySQL - PostgreSQL diff --git a/content/influxdb/v2.0/reference/flux/stdlib/sql/from.md b/content/influxdb/v2.0/reference/flux/stdlib/sql/from.md index 54f59b3d8..5babdafc1 100644 --- a/content/influxdb/v2.0/reference/flux/stdlib/sql/from.md +++ b/content/influxdb/v2.0/reference/flux/stdlib/sql/from.md @@ -36,6 +36,7 @@ _**Data type:** String_ The following drivers are available: - awsathena +- bigquery - mysql - postgres - snowflake @@ -73,6 +74,10 @@ sqlserver://username:password@localhost:1234?database=examplebdb server=localhost;user id=username;database=examplebdb; server=localhost;user id=username;database=examplebdb;azure auth=ENV server=localhost;user id=username;database=examplebdbr;azure tenant id=77e7d537;azure client id=58879ce8;azure client secret=0123456789 + +# Google BigQuery DSNs +bigquery://projectid/?param1=value¶m2=value +bigquery://projectid/location?param1=value¶m2=value ``` ### query @@ -88,6 +93,7 @@ _**Data type:** String_ - [SQLite](#query-an-sqlite-database) - [Amazon Athena](#query-an-amazon-athena-database) - [SQL Server](#query-a-sql-server-database) +- [Google BigQuery](#query-a-bigquery-database) {{% note %}} The examples below use [InfluxDB secrets](/influxdb/v2.0/security/secrets/) to populate @@ -250,3 +256,40 @@ _For information about managed identities, see [Microsoft managed identities](ht ``` azure auth=MSI ``` + +### Query a BigQuery database +```js +import "sql" +import "influxdata/influxdb/secrets" +projectID = secrets.get(key: "BIGQUERY_PROJECT_ID") +apiKey = secrets.get(key: "BIGQUERY_APIKEY") +sql.from( + driverName: "bigquery", + dataSourceName: "bigquery://${projectID}/?apiKey=${apiKey}", + query:"SELECT * FROM exampleTable" +) +``` + +#### Common BigQuery URL parameters +- **dataset** - BigQuery dataset ID. When set, you can use unqualified table names in queries. + +#### BigQuery authentication parameters +The Flux BigQuery implementation uses the Google Cloud Go SDK. +Provide your authentication credentials using one of the following methods: + +- The `GOOGLE_APPLICATION_CREDENTIALS` environment variable that identifies the + location of your credential JSON file. +- Provide your BigQuery API key using the **apiKey** URL parameter in your BigQuery DSN. + + ###### Example apiKey URL parameter + ``` + bigquery://projectid/?apiKey=AIzaSyB6XK8IO5AzKZXoioQOVNTFYzbDBjY5hy4 + ``` + +- Provide your base-64 encoded service account, refresh token, or JSON credentials + using the **credentials** URL parameter in your BigQuery DSN. + + ###### Example credentials URL parameter + ``` + bigquery://projectid/?credentials=eyJ0eXBlIjoiYXV0... + ``` diff --git a/content/influxdb/v2.0/reference/flux/stdlib/sql/to.md b/content/influxdb/v2.0/reference/flux/stdlib/sql/to.md index 8f66a6afc..c06fec1ff 100644 --- a/content/influxdb/v2.0/reference/flux/stdlib/sql/to.md +++ b/content/influxdb/v2.0/reference/flux/stdlib/sql/to.md @@ -34,12 +34,18 @@ _**Data type:** String_ The following drivers are available: +- bigquery - mysql - postgres - snowflake - sqlite3 – _Does not work with InfluxDB OSS or InfluxDB Cloud. More information [below](#write-data-to-an-sqlite-database)._ - sqlserver, mssql +{{% warn %}} +#### sql.to does not support Amazon Athena +The `sql.to` function does not support writing data to [Amazon Athena](https://aws.amazon.com/athena/). +{{% /warn %}} + ### dataSourceName The data source name (DSN) or connection string used to connect to the SQL database. The string's form and structure depend on the [driver](#drivername) used. @@ -67,6 +73,10 @@ sqlserver://username:password@localhost:1234?database=examplebdb server=localhost;user id=username;database=examplebdb; server=localhost;user id=username;database=examplebdb;azure auth=ENV server=localhost;user id=username;database=examplebdbr;azure tenant id=77e7d537;azure client id=58879ce8;azure client secret=0123456789 + +# Google BigQuery DSNs +bigquery://projectid/?param1=value¶m2=value +bigquery://projectid/location?param1=value¶m2=value ``` ### table @@ -91,6 +101,7 @@ If writing to a **SQLite** database, set `batchSize` to `999` or less. - [Snowflake](#write-data-to-a-snowflake-database) - [SQLite](#write-data-to-an-sqlite-database) - [SQL Server](#write-data-to-a-sql-server-database) +- [Google BigQuery](#write-data-to-a-sql-server-database) {{% note %}} The examples below use [InfluxDB secrets](/influxdb/v2.0/security/secrets/) to populate @@ -223,7 +234,39 @@ _For information about managed identities, see [Microsoft managed identities](ht azure auth=MSI ``` -{{% warn %}} -### sql.to does not support Amazon Athena -The `sql.to` function does not support writing data to [Amazon Athena](https://aws.amazon.com/athena/). -{{% /warn %}} +### Write to a BigQuery database +```js +import "sql" +import "influxdata/influxdb/secrets" +projectID = secrets.get(key: "BIGQUERY_PROJECT_ID") +apiKey = secrets.get(key: "BIGQUERY_APIKEY") +sql.to( + driverName: "bigquery", + dataSourceName: "bigquery://${projectID}/?apiKey=${apiKey}", + table:"exampleTable" +) +``` + +#### Common BigQuery URL parameters +- **dataset** - BigQuery dataset ID. When set, you can use unqualified table names in queries. + +#### BigQuery authentication parameters +The Flux BigQuery implementation uses the Google Cloud Go SDK. +Provide your authentication credentials using one of the following methods: + +- The `GOOGLE_APPLICATION_CREDENTIALS` environment variable that identifies the + location of your credential JSON file. +- Provide your BigQuery API key using the **apiKey** URL parameter in your BigQuery DSN. + + ###### Example apiKey URL parameter + ``` + bigquery://projectid/?apiKey=AIzaSyB6XK8IO5AzKZXoioQOVNTFYzbDBjY5hy4 + ``` + +- Provide your base-64 encoded service account, refresh token, or JSON credentials + using the **credentials** URL parameter in your BigQuery DSN. + + ###### Example credentials URL parameter + ``` + bigquery://projectid/?credentials=eyJ0eXBlIjoiYXV0... + ``` diff --git a/content/influxdb/v2.0/reference/release-notes/flux.md b/content/influxdb/v2.0/reference/release-notes/flux.md index 6f983cb59..e918fd559 100644 --- a/content/influxdb/v2.0/reference/release-notes/flux.md +++ b/content/influxdb/v2.0/reference/release-notes/flux.md @@ -25,7 +25,8 @@ InfluxDB until the next InfluxDB v2.0 release._ ### Features - Improve window errors. -- Add BigQuery support to [`sql` package](#). +- Add [BigQuery](https://cloud.google.com/bigquery) support to + [`sql` package](/influxdb/v2.0/reference/flux/stdlib/sql/). - Add `TypeExpression` to `BuiltinStmt` and fix tests. - Time-weighted average ([`timeWeightedAvg()` function](/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/timeweightedavg/)). - Update [`integral()`](/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/integral/) From fa9a79bef00083d65c25a9e7df9862cb9934a8b6 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Fri, 4 Sep 2020 09:42:12 -0600 Subject: [PATCH 09/10] updated flux docs to address PR review --- .../flux/stdlib/built-in/transformations/aggregates/integral.md | 1 + .../v2.0/reference/flux/stdlib/influxdb-tasks/lastsuccess.md | 2 +- content/influxdb/v2.0/reference/release-notes/flux.md | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/content/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/integral.md b/content/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/integral.md index c36cbe630..7cb52c5bf 100644 --- a/content/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/integral.md +++ b/content/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/integral.md @@ -53,6 +53,7 @@ Defaults to `""`. Use one of the following interpolation options: +- _empty sting for no interpolation_ - linear _**Data type:** String_ diff --git a/content/influxdb/v2.0/reference/flux/stdlib/influxdb-tasks/lastsuccess.md b/content/influxdb/v2.0/reference/flux/stdlib/influxdb-tasks/lastsuccess.md index 8a2b0d9bb..cab6e0ade 100644 --- a/content/influxdb/v2.0/reference/flux/stdlib/influxdb-tasks/lastsuccess.md +++ b/content/influxdb/v2.0/reference/flux/stdlib/influxdb-tasks/lastsuccess.md @@ -26,7 +26,7 @@ _**Data type:** Time_ ## Examples -##### Query data since that last successful task run +##### Query data since the last successful task run ```js import "influxdata/influxdb/tasks" diff --git a/content/influxdb/v2.0/reference/release-notes/flux.md b/content/influxdb/v2.0/reference/release-notes/flux.md index e918fd559..bc4e16efc 100644 --- a/content/influxdb/v2.0/reference/release-notes/flux.md +++ b/content/influxdb/v2.0/reference/release-notes/flux.md @@ -28,7 +28,7 @@ InfluxDB until the next InfluxDB v2.0 release._ - Add [BigQuery](https://cloud.google.com/bigquery) support to [`sql` package](/influxdb/v2.0/reference/flux/stdlib/sql/). - Add `TypeExpression` to `BuiltinStmt` and fix tests. -- Time-weighted average ([`timeWeightedAvg()` function](/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/timeweightedavg/)). +- Add time-weighted average ([`timeWeightedAvg()` function](/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/timeweightedavg/)). - Update [`integral()`](/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/integral/) with linear interpolation. - Make experimental tracing an attribute of the context. From c3562ee0decab4d4c2d70b268985775531b0085f Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Fri, 4 Sep 2020 09:52:54 -0600 Subject: [PATCH 10/10] updated flux operator precendence --- .../reference/flux/language/expressions.md | 17 +++++----- .../v2.0/reference/flux/language/operators.md | 34 ++++++++++--------- 2 files changed, 27 insertions(+), 24 deletions(-) diff --git a/content/influxdb/v2.0/reference/flux/language/expressions.md b/content/influxdb/v2.0/reference/flux/language/expressions.md index a014c1ff4..b589e9d12 100644 --- a/content/influxdb/v2.0/reference/flux/language/expressions.md +++ b/content/influxdb/v2.0/reference/flux/language/expressions.md @@ -223,18 +223,19 @@ Operators with a lower number have higher precedence. | 1 | `a()` | Function call | | | `a[]` | Member or index access | | | `.` | Member access | -| 2 | `^` | Exponentiation | -| 3 | `*` `/` `%` | Multiplication, division, and modulo | -| 4 | `+` `-` | Addition and subtraction | -| 5 |`==` `!=` | Comparison operators | +| 2 | \|> | Pipe forward | +| 3 | `^` | Exponentiation | +| 4 | `*` `/` `%` | Multiplication, division, and modulo | +| 5 | `+` `-` | Addition and subtraction | +| 6 |`==` `!=` | Comparison operators | | | `<` `<=` | | | | `>` `>=` | | | |`=~` `!~` | | -| 6 | `not` | Unary logical operator | +| 7 | `not` | Unary logical operator | | | `exists` | Null check operator | -| 7 | `and` | Logical AND | -| 8 | `or` | Logical OR | -| 9 | `if` `then` `else` | Conditional | +| 8 | `and` | Logical AND | +| 9 | `or` | Logical OR | +| 10 | `if` `then` `else` | Conditional | The operator precedence is encoded directly into the grammar as the following. diff --git a/content/influxdb/v2.0/reference/flux/language/operators.md b/content/influxdb/v2.0/reference/flux/language/operators.md index d3cbe2d1b..64fef0b33 100644 --- a/content/influxdb/v2.0/reference/flux/language/operators.md +++ b/content/influxdb/v2.0/reference/flux/language/operators.md @@ -131,19 +131,21 @@ Literal constructors define fixed values. The table below outlines operator precedence. Operators with a lower number have higher precedence. -| Precedence | Operator | Description | -|:----------:|:--------: |:--------------------------| -| 1 | `a()` | Function call | -| | `a[]` | Member or index access | -| | `.` | Member access | -| 2 | `*` `/` |Multiplication and division| -| 3 | `+` `-` | Addition and subtraction | -| 4 |`==` `!=` | Comparison operators | -| | `<` `<=` | | -| | `>` `>=` | | -| |`=~` `!~` | | -| 5 | `not` | Unary logical operator | -| | `exists` | Null check operator | -| 6 | `and` | Logical AND | -| 7 | `or` | Logical OR | -| 8 | `if` `then` `else` | Conditional | +| Precedence | Operator | Description | +|:----------:|:--------: |:-------------------------- | +| 1 | `a()` | Function call | +| | `a[]` | Member or index access | +| | `.` | Member access | +| 2 | \|> | Pipe forward | +| 3 | `^` | Exponentiation | +| 4 | `*` `/` `%` | Multiplication, division, and modulo | +| 5 | `+` `-` | Addition and subtraction | +| 6 |`==` `!=` | Comparison operators | +| | `<` `<=` | | +| | `>` `>=` | | +| |`=~` `!~` | | +| 7 | `not` | Unary logical operator | +| | `exists` | Null check operator | +| 8 | `and` | Logical AND | +| 9 | `or` | Logical OR | +| 10 | `if` `then` `else` | Conditional |