From ea219c7f375fb5dffff783f0d6c26bb59b4f1923 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Fri, 20 Mar 2020 22:32:40 -0600 Subject: [PATCH 01/15] added the geo.shapeData function --- assets/styles/layouts/article/_flex.scss | 1 + .../flux/stdlib/experimental/geo/_index.md | 13 ++ .../flux/stdlib/experimental/geo/shapedata.md | 123 ++++++++++++++++++ 3 files changed, 137 insertions(+) create mode 100644 content/v2.0/reference/flux/stdlib/experimental/geo/shapedata.md diff --git a/assets/styles/layouts/article/_flex.scss b/assets/styles/layouts/article/_flex.scss index 93ba285d0..7f5fca1d9 100644 --- a/assets/styles/layouts/article/_flex.scss +++ b/assets/styles/layouts/article/_flex.scss @@ -3,6 +3,7 @@ .flex-wrapper { display: flex; flex-wrap: wrap; + margin: 1.5rem 0; } .flex-container { diff --git a/content/v2.0/reference/flux/stdlib/experimental/geo/_index.md b/content/v2.0/reference/flux/stdlib/experimental/geo/_index.md index 514761f14..2568350ba 100644 --- a/content/v2.0/reference/flux/stdlib/experimental/geo/_index.md +++ b/content/v2.0/reference/flux/stdlib/experimental/geo/_index.md @@ -60,6 +60,19 @@ generating S2 Cell ID tokens. For example: - **Python:** [`s2sphere.CellId.to_token()`](https://s2sphere.readthedocs.io/en/latest/api.html#s2sphere.CellId) - **Javascript:** [`s2.cellid.toToken()`](https://github.com/mapbox/node-s2/blob/master/API.md#cellidtotoken---string) +### Add S2 Cell IDs to existing geo-temporal data +Use the [`geo.shapeData()` function](/v2.0/reference/flux/stdlib/experimental/geo/shapedata/) +to add `s2_cell_id` tags to data that includes fields with latitude and longitude values. + +```js +//... + |> shapeData( + latField: "latitude", + lonField: "longitude", + level: 10 + ) +``` + ## Region definitions Many functions in the Geo package filter data based on geographic region. Define geographic regions using the following shapes: diff --git a/content/v2.0/reference/flux/stdlib/experimental/geo/shapedata.md b/content/v2.0/reference/flux/stdlib/experimental/geo/shapedata.md new file mode 100644 index 000000000..85cb2da14 --- /dev/null +++ b/content/v2.0/reference/flux/stdlib/experimental/geo/shapedata.md @@ -0,0 +1,123 @@ +--- +title: geo.shapeData() function +description: > + The `geo.shapeData()` function restructures data with existing latitude and longitude + fields to meet the requirements of the Geo package by renaming the existing latitude + and longitude fields to **lat** and **lon** and adding an **s2_cell_id** tag. +menu: + v2_0_ref: + name: geo.shapeData + parent: Geo +weight: 401 +v2.0/tags: [functions, geo] +--- + +The `geo.shapeData()` function restructures data with existing latitude and longitude +fields to meet the requirements of the Geo package by renaming the existing latitude +and longitude fields to **lat** and **lon** and adding an **s2_cell_id** tag. + +**`geo.shapeData()` does the following:** + +- renames existing latitude and longitude fields to `lat` and `lon` +- pivots data into row-wise sets based on the [`correlatinKey`](#correlationkey) +- uses the `lat` and `lon` values to generate the `s2_cell_id` tag based on a + specified [S2 cell level](https://s2geometry.io/resources/s2cell_statistics.html) +- adds the `s2_cell_id` column to the group key + +_**Function type:** Transformation_ + +```js +import "experimental/geo" + +geo.shapeData( + latField: "latitude", + lonField: "longitude", + level: 10, + correlationKey: ["_time"] +) +``` + +## Parameters + +### latField +Name of the existing field that contains the latitude value in **decimal degrees** (WGS 84). +Field is renamed to `lat`. + +_**Data type:** String_ + +### lonField +Name of the existing field that contains the longitude value in **decimal degrees** (WGS 84). +Field is renamed to `lon`. + +_**Data type:** String_ + +{{% note %}} +`point` and `token` are mutually exclusive. +{{% /note %}} + +### level +[S2 cell level](https://s2geometry.io/resources/s2cell_statistics.html) to use +when generating the S2 cell ID token. + +_**Data type:** Integer_ + +### correlationKey +List of columns used to uniquely identify a row for output. +Default is `["_time"]`. + +_**Data type:** Array of strings_ + +## Examples + +##### Shape data to meet the requirements of the Geo package +```js +import "experimental/geo" + +from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "example-measurement") + |> geo.shapeData( + latField: "latitude", + lonField: "longitude", + level: 10 + ) +``` + +### geo.shapeData input and ouput + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _field | _value | +|:----- |:------: | ------:| +| 0001 | latitude | 30.0 | +| 0002 | latitude | 30.5 | +| 0003 | latitude | 30.7 | +| 0004 | latitude | 31.1 | +| • • • | • • • | • • • | +| 0001 | longitude | 20.0 | +| 0002 | longitude | 19.8 | +| 0003 | longitude | 19.2 | +| 0004 | longitude | 19.5 | +{{% /flex-content %}} +{{% flex-content %}} + +**The following function would output:** + +```js +|> geo.shapeData( + latField: "latitude", + lonField: "longitude", + level: 5 +) +``` + +| _time | lat | lon | s2_cell_id | +|:----- |:--------:|:---------:| ----------:| +| 0001 | 30.0 | 20.0 | 138c | +| 0002 | 30.5 | 19.8 | 1384 | +| 0003 | 30.7 | 19.2 | 139c | +| 0004 | 31.1 | 19.5 | 139c | +{{% /flex-content %}} +{{< /flex >}} From a7baeb8fb46f26ba7f3e1b81ce88df2fec1b7d97 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Fri, 20 Mar 2020 22:34:28 -0600 Subject: [PATCH 02/15] added placeholder for flux 0.65.0 release notes --- content/v2.0/reference/release-notes/flux.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/v2.0/reference/release-notes/flux.md b/content/v2.0/reference/release-notes/flux.md index 182d55e77..74ae09a95 100644 --- a/content/v2.0/reference/release-notes/flux.md +++ b/content/v2.0/reference/release-notes/flux.md @@ -16,6 +16,8 @@ Though newer versions of Flux may be available, they will not be included with InfluxDB until the next InfluxDB v2.0 release._ {{% /note %}} +## v0.65.0 [unreleased] + ## v0.64.0 [2020-03-11] ### Features From 26c5fb2e586432cdd9bd720d932d8b686f52b12a Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Fri, 20 Mar 2020 22:45:40 -0600 Subject: [PATCH 03/15] added note about adding s2_cell_id to the group key, resolves #839 --- .../flux/stdlib/experimental/geo/filterrows.md | 15 +++++++++++++++ .../flux/stdlib/experimental/geo/gridfilter.md | 15 +++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/content/v2.0/reference/flux/stdlib/experimental/geo/filterrows.md b/content/v2.0/reference/flux/stdlib/experimental/geo/filterrows.md index 597af7bf5..e70a9766c 100644 --- a/content/v2.0/reference/flux/stdlib/experimental/geo/filterrows.md +++ b/content/v2.0/reference/flux/stdlib/experimental/geo/filterrows.md @@ -35,6 +35,21 @@ geo.filterRows( ) ``` +{{% note %}} +#### s2_cell_id must be part of the group key +In order for `geo.filterRows()` to filter geo-temporal data, the `s2_cell_id` column +must be part of the [group key](/v2.0/reference/glossary/#group-key). +If it is not, use the [`experimental.group` function](/v2.0/reference/flux/stdlib/experimental/group) +to add it to the group key: + +```js +import "experimental" + +// ... + |> experimental.group(columns: ["s2_cell_id"], mode: "extend") +``` +{{% /note %}} + ### Strict and non-strict filtering In most cases, the specified geographic region does not perfectly align with S2 grid cells. diff --git a/content/v2.0/reference/flux/stdlib/experimental/geo/gridfilter.md b/content/v2.0/reference/flux/stdlib/experimental/geo/gridfilter.md index a48587f1e..8da664989 100644 --- a/content/v2.0/reference/flux/stdlib/experimental/geo/gridfilter.md +++ b/content/v2.0/reference/flux/stdlib/experimental/geo/gridfilter.md @@ -39,6 +39,21 @@ geo.gridFilter( ) ``` +{{% note %}} +#### s2_cell_id must be part of the group key +In order for `geo.gridFilter()` to filter geo-temporal data, the `s2_cell_id` column +must be part of the [group key](/v2.0/reference/glossary/#group-key). +If it is not, use the [`experimental.group` function](/v2.0/reference/flux/stdlib/experimental/group) +to add it to the group key: + +```js +import "experimental" + +// ... + |> experimental.group(columns: ["s2_cell_id"], mode: "extend") +``` +{{% /note %}} + ### Non-strict and strict filtering In most cases, the specified geographic region does not perfectly align with S2 grid cells. From 38710ce4ac0a483f0e6ada86d00179498e51b5c6 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Mon, 23 Mar 2020 15:26:37 -0600 Subject: [PATCH 04/15] updated geo.shapeData doc to address PR feedback --- .../stdlib/experimental/geo/filterrows.md | 7 ++--- .../stdlib/experimental/geo/gridfilter.md | 7 ++--- .../flux/stdlib/experimental/geo/shapedata.md | 30 ++++++++----------- 3 files changed, 18 insertions(+), 26 deletions(-) diff --git a/content/v2.0/reference/flux/stdlib/experimental/geo/filterrows.md b/content/v2.0/reference/flux/stdlib/experimental/geo/filterrows.md index e70a9766c..a9b849a3a 100644 --- a/content/v2.0/reference/flux/stdlib/experimental/geo/filterrows.md +++ b/content/v2.0/reference/flux/stdlib/experimental/geo/filterrows.md @@ -37,10 +37,9 @@ geo.filterRows( {{% note %}} #### s2_cell_id must be part of the group key -In order for `geo.filterRows()` to filter geo-temporal data, the `s2_cell_id` column -must be part of the [group key](/v2.0/reference/glossary/#group-key). -If it is not, use the [`experimental.group` function](/v2.0/reference/flux/stdlib/experimental/group) -to add it to the group key: +To filter geo-temporal data with `geo.filterRows()`, `s2_cell_id` must be part +of the [group key](/v2.0/reference/glossary/#group-key). +To add `s2_cell_id` to the group key, use [`experimental.group`](/v2.0/reference/flux/stdlib/experimental/group): ```js import "experimental" diff --git a/content/v2.0/reference/flux/stdlib/experimental/geo/gridfilter.md b/content/v2.0/reference/flux/stdlib/experimental/geo/gridfilter.md index 8da664989..821933550 100644 --- a/content/v2.0/reference/flux/stdlib/experimental/geo/gridfilter.md +++ b/content/v2.0/reference/flux/stdlib/experimental/geo/gridfilter.md @@ -41,10 +41,9 @@ geo.gridFilter( {{% note %}} #### s2_cell_id must be part of the group key -In order for `geo.gridFilter()` to filter geo-temporal data, the `s2_cell_id` column -must be part of the [group key](/v2.0/reference/glossary/#group-key). -If it is not, use the [`experimental.group` function](/v2.0/reference/flux/stdlib/experimental/group) -to add it to the group key: +To filter geo-temporal data with `geo.gridFilter()`, `s2_cell_id` must be part +of the [group key](/v2.0/reference/glossary/#group-key). +To add `s2_cell_id` to the group key, use [`experimental.group`](/v2.0/reference/flux/stdlib/experimental/group): ```js import "experimental" diff --git a/content/v2.0/reference/flux/stdlib/experimental/geo/shapedata.md b/content/v2.0/reference/flux/stdlib/experimental/geo/shapedata.md index 85cb2da14..ab4d18c88 100644 --- a/content/v2.0/reference/flux/stdlib/experimental/geo/shapedata.md +++ b/content/v2.0/reference/flux/stdlib/experimental/geo/shapedata.md @@ -1,9 +1,9 @@ --- title: geo.shapeData() function description: > - The `geo.shapeData()` function restructures data with existing latitude and longitude - fields to meet the requirements of the Geo package by renaming the existing latitude - and longitude fields to **lat** and **lon** and adding an **s2_cell_id** tag. + The `geo.shapeData()` function renames existing latitude and longitude fields to + **lat** and **lon** and adds an **s2_cell_id** tag. + Use `geo.shapeData()` to ensure geo-temporal data meets the requirements of the Geo package. menu: v2_0_ref: name: geo.shapeData @@ -12,17 +12,15 @@ weight: 401 v2.0/tags: [functions, geo] --- -The `geo.shapeData()` function restructures data with existing latitude and longitude -fields to meet the requirements of the Geo package by renaming the existing latitude -and longitude fields to **lat** and **lon** and adding an **s2_cell_id** tag. +The `geo.shapeData()` function renames existing latitude and longitude fields to +**lat** and **lon** and adds an **s2_cell_id** tag. +Use `geo.shapeData()` to ensure geo-temporal data meets the +[requirements of the Geo package](/v2.0/reference/flux/stdlib/experimental/geo/#geo-schema-requirements): -**`geo.shapeData()` does the following:** - -- renames existing latitude and longitude fields to `lat` and `lon` -- pivots data into row-wise sets based on the [`correlatinKey`](#correlationkey) -- uses the `lat` and `lon` values to generate the `s2_cell_id` tag based on a - specified [S2 cell level](https://s2geometry.io/resources/s2cell_statistics.html) -- adds the `s2_cell_id` column to the group key +1. Rename existing latitude and longitude fields to `lat` and `lon`. +2. Pivot data into row-wise sets based on the [`correlatinKey`](#correlationkey). +3. Generate `s2_cell_id` tags using `lat` and `lon` values and a specified + [S2 cell level](https://s2geometry.io/resources/s2cell_statistics.html). _**Function type:** Transformation_ @@ -51,10 +49,6 @@ Field is renamed to `lon`. _**Data type:** String_ -{{% note %}} -`point` and `token` are mutually exclusive. -{{% /note %}} - ### level [S2 cell level](https://s2geometry.io/resources/s2cell_statistics.html) to use when generating the S2 cell ID token. @@ -83,7 +77,7 @@ from(bucket: "example-bucket") ) ``` -### geo.shapeData input and ouput +### geo.shapeData input and output {{< flex >}} {{% flex-content %}} From 3cb94b8de03717151eca3dc972674b1d5ef0ed1d Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Fri, 27 Mar 2020 12:08:30 -0600 Subject: [PATCH 05/15] added the experimental.join function, resolves #875 --- .../flux/stdlib/experimental/group.md | 4 +- .../flux/stdlib/experimental/join.md | 159 ++++++++++++++++++ 2 files changed, 162 insertions(+), 1 deletion(-) create mode 100644 content/v2.0/reference/flux/stdlib/experimental/join.md diff --git a/content/v2.0/reference/flux/stdlib/experimental/group.md b/content/v2.0/reference/flux/stdlib/experimental/group.md index 60c632c0f..34e32a9ac 100644 --- a/content/v2.0/reference/flux/stdlib/experimental/group.md +++ b/content/v2.0/reference/flux/stdlib/experimental/group.md @@ -52,7 +52,9 @@ Appends columns defined in the [`columns` parameter](#columns) to all existing ###### Include the value column in each groups' group key ```js +import "experimental" + from(bucket: "example-bucket") |> range(start: -1m) - |> group(columns: ["_value"], mode: "extend") + |> experimental.group(columns: ["_value"], mode: "extend") ``` diff --git a/content/v2.0/reference/flux/stdlib/experimental/join.md b/content/v2.0/reference/flux/stdlib/experimental/join.md new file mode 100644 index 000000000..faa0bf7af --- /dev/null +++ b/content/v2.0/reference/flux/stdlib/experimental/join.md @@ -0,0 +1,159 @@ +--- +title: experimental.join() function +description: > + The `experimental.join()` function ... + `group()` function. +menu: + v2_0_ref: + name: experimental.join + parent: Experimental +weight: 302 +--- + +The `experimental.join()` function joins two streams of tables on the +[group key](/v2.0/reference/glossary/#group-key) with the addition of the `_time` column. +Columns that are not part of the group key or explicitly mapped in the join operation +are dropped from output tables. + +{{% note %}} +To join streams of tables with different fields, use [`group()`](/v2.0/reference/flux/stdlib/built-in/transformations/group/) +or [`drop()`](/v2.0/reference/flux/stdlib/built-in/transformations/drop/) to remove +`_field` from the group key before joining. +_See an example [below](#join-two-streams-of-tables-with-different-fields)._ +{{% /note %}} + +_**Function type:** Transformation_ + +```js +import "experimental" + +// ... + +experimental.join( + left: left, + right: right, + fn: (left, right) => ({left with lv: left._value, rv: right._value }) +) +``` + +{{% note %}} +This function will likely replace the [`join` function](/v2.0/reference/flux/stdlib/built-in/transformations/join/) +when sufficiently vetted. +{{% /note %}} + +## Parameters + +### left +First of two streams of tables to join. + +_**Data type:** Stream of tables_ + +### right +Second of two streams of tables to join. + +_**Data type:** Stream of tables_ + +### fn +A function with `left` and `right` arguments that maps a new output object +using values from the `left` and `right` input objects. +The return value must be an object. + +_**Data type:** Function_ + +## Examples + +**Given the following input tables:** +{{< flex >}} +{{% flex-content %}} +##### left +| _time | _field | _value | +|:----- |:------:| ------:| +| 0001 | temp | 80.1 | +| 0002 | temp | 80.2 | +| 0003 | temp | 79.9 | +| 0004 | temp | 80.0 | +{{% /flex-content %}} +{{% flex-content %}} +##### right +| _time | _field | _value | +|:----- |:------:| ------:| +| 0001 | temp | 72.1 | +| 0002 | temp | 72.2 | +| 0003 | temp | 71.9 | +| 0004 | temp | 72.0 | +{{% /flex-content %}} +{{< /flex >}} + +**The following `experimental.join()` function would output:** + +```js +import "experimental" + +experimental.join( + left: left, + right: right, + fn: (left, right) => ({ + left with + lv: left._value, + rv: right._value, + diff: left._value - right._value + }) +) +``` + +| _time | _field | lv | rv | diff | +|:----- |:------:|:--: |:--: | ----:| +| 0001 | temp | 80.1 | 72.1 | 8.0 | +| 0002 | temp | 80.2 | 72.2 | 8.0 | +| 0003 | temp | 79.9 | 71.9 | 8.0 | +| 0004 | temp | 80.0 | 72.0 | 8.0 | + +--- + +###### Join two streams of tables +```js +import "experimental" + +s1 = from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "foo") + +s2 = from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "bar") + +experimental.join( + left: s1, + right: s2, + fn: (left, right) => ({ + left with + s1_value: left._value, + s2_value: right._value + }) +) +``` + +###### Join two streams of tables with different fields +```js +import "experimental" + +s1 = from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "foo" and r._field == "bar") + |> group(columns: ["_time", "_field", "_value"], mode: "except") + +s2 = from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "baz" and r._field == "quz") + |> group(columns: ["_time", "_field", "_value"], mode: "except") + +experimental.join( + left: s1, + right: s2, + fn: (left, right) => ({ + left with + bar_value: left._value, + quz_value: right._value + }) +) +``` From a83b4bd3a2943a6a4db2a8fcec7a5d7e5273e26a Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Fri, 27 Mar 2020 12:11:56 -0600 Subject: [PATCH 06/15] added page description to experimental.join func --- content/v2.0/reference/flux/stdlib/experimental/join.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/v2.0/reference/flux/stdlib/experimental/join.md b/content/v2.0/reference/flux/stdlib/experimental/join.md index faa0bf7af..6605ef486 100644 --- a/content/v2.0/reference/flux/stdlib/experimental/join.md +++ b/content/v2.0/reference/flux/stdlib/experimental/join.md @@ -1,8 +1,8 @@ --- title: experimental.join() function description: > - The `experimental.join()` function ... - `group()` function. + The `experimental.join()` function function joins two streams of tables on the + group key with the addition of the `_time` column. menu: v2_0_ref: name: experimental.join From 9819abb58502a72c4b361958a522c062a1ae8a46 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Fri, 27 Mar 2020 12:12:27 -0600 Subject: [PATCH 07/15] fixed typo in experimental.join page description --- content/v2.0/reference/flux/stdlib/experimental/join.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/v2.0/reference/flux/stdlib/experimental/join.md b/content/v2.0/reference/flux/stdlib/experimental/join.md index 6605ef486..46bf3e95e 100644 --- a/content/v2.0/reference/flux/stdlib/experimental/join.md +++ b/content/v2.0/reference/flux/stdlib/experimental/join.md @@ -1,7 +1,7 @@ --- title: experimental.join() function description: > - The `experimental.join()` function function joins two streams of tables on the + The `experimental.join()` function joins two streams of tables on the group key with the addition of the `_time` column. menu: v2_0_ref: From d74730d81943af7ecbe4fd16272143153d68ec73 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Fri, 27 Mar 2020 15:10:01 -0600 Subject: [PATCH 08/15] updated experimental.join to address PR feedback --- content/v2.0/reference/flux/stdlib/experimental/join.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/content/v2.0/reference/flux/stdlib/experimental/join.md b/content/v2.0/reference/flux/stdlib/experimental/join.md index 46bf3e95e..d1bdd8462 100644 --- a/content/v2.0/reference/flux/stdlib/experimental/join.md +++ b/content/v2.0/reference/flux/stdlib/experimental/join.md @@ -11,7 +11,7 @@ weight: 302 --- The `experimental.join()` function joins two streams of tables on the -[group key](/v2.0/reference/glossary/#group-key) with the addition of the `_time` column. +[group key](/v2.0/reference/glossary/#group-key) and `_time` column. Columns that are not part of the group key or explicitly mapped in the join operation are dropped from output tables. @@ -62,6 +62,8 @@ _**Data type:** Function_ ## Examples +### Input and output tables + **Given the following input tables:** {{< flex >}} {{% flex-content %}} From e5848e041dcb2521fa57f37d03c2db0b11cfede0 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Fri, 27 Mar 2020 15:53:51 -0600 Subject: [PATCH 09/15] included measurements in experimental.join note --- .../v2.0/reference/flux/stdlib/experimental/join.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/content/v2.0/reference/flux/stdlib/experimental/join.md b/content/v2.0/reference/flux/stdlib/experimental/join.md index d1bdd8462..719cdcc57 100644 --- a/content/v2.0/reference/flux/stdlib/experimental/join.md +++ b/content/v2.0/reference/flux/stdlib/experimental/join.md @@ -16,9 +16,9 @@ Columns that are not part of the group key or explicitly mapped in the join oper are dropped from output tables. {{% note %}} -To join streams of tables with different fields, use [`group()`](/v2.0/reference/flux/stdlib/built-in/transformations/group/) +To join streams of tables with different fields or measuremtns, use [`group()`](/v2.0/reference/flux/stdlib/built-in/transformations/group/) or [`drop()`](/v2.0/reference/flux/stdlib/built-in/transformations/drop/) to remove -`_field` from the group key before joining. +`_field` and `_measurement` from the group key before joining. _See an example [below](#join-two-streams-of-tables-with-different-fields)._ {{% /note %}} @@ -135,19 +135,19 @@ experimental.join( ) ``` -###### Join two streams of tables with different fields +###### Join two streams of tables with different fields and measurements ```js import "experimental" s1 = from(bucket: "example-bucket") |> range(start: -1h) |> filter(fn: (r) => r._measurement == "foo" and r._field == "bar") - |> group(columns: ["_time", "_field", "_value"], mode: "except") + |> group(columns: ["_time", "_measurement", "_field", "_value"], mode: "except") s2 = from(bucket: "example-bucket") |> range(start: -1h) |> filter(fn: (r) => r._measurement == "baz" and r._field == "quz") - |> group(columns: ["_time", "_field", "_value"], mode: "except") + |> group(columns: ["_time", "_measurement", "_field", "_value"], mode: "except") experimental.join( left: s1, From 827d41847e037a9c2dd874cda571c3ab94917951 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Mon, 30 Mar 2020 09:50:33 -0600 Subject: [PATCH 10/15] added flux 0.65 to changelog --- content/v2.0/reference/release-notes/flux.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/content/v2.0/reference/release-notes/flux.md b/content/v2.0/reference/release-notes/flux.md index 74ae09a95..0897076b4 100644 --- a/content/v2.0/reference/release-notes/flux.md +++ b/content/v2.0/reference/release-notes/flux.md @@ -16,7 +16,21 @@ Though newer versions of Flux may be available, they will not be included with InfluxDB until the next InfluxDB v2.0 release._ {{% /note %}} -## v0.65.0 [unreleased] +## v0.65.0 [2020-03-27] + +### Features +- Add `experimental.join()` function. +- Store comments in the AST and preserve on format. +- Add `shapeData()` function to Geo package. +- Expose format to Wasm users. + +### Bug fixes +- Reimplement `stateChanges()` function. +- Remove the `set -x` in the xcc script. +- Publishes Flux as a public npm package. +- Pivot message passing. + +--- ## v0.64.0 [2020-03-11] From f50d110da6b8c64753276fab8d647482843d6b04 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Mon, 30 Mar 2020 14:35:17 -0600 Subject: [PATCH 11/15] updated experimental join doc to address PR feedback --- content/v2.0/reference/flux/stdlib/experimental/join.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/content/v2.0/reference/flux/stdlib/experimental/join.md b/content/v2.0/reference/flux/stdlib/experimental/join.md index 719cdcc57..26c0d8765 100644 --- a/content/v2.0/reference/flux/stdlib/experimental/join.md +++ b/content/v2.0/reference/flux/stdlib/experimental/join.md @@ -12,11 +12,10 @@ weight: 302 The `experimental.join()` function joins two streams of tables on the [group key](/v2.0/reference/glossary/#group-key) and `_time` column. -Columns that are not part of the group key or explicitly mapped in the join operation -are dropped from output tables. +Use the [`fn` parameter](#fn) to map new output tables using values from input tables. {{% note %}} -To join streams of tables with different fields or measuremtns, use [`group()`](/v2.0/reference/flux/stdlib/built-in/transformations/group/) +To join streams of tables with different fields or measurements, use [`group()`](/v2.0/reference/flux/stdlib/built-in/transformations/group/) or [`drop()`](/v2.0/reference/flux/stdlib/built-in/transformations/drop/) to remove `_field` and `_measurement` from the group key before joining. _See an example [below](#join-two-streams-of-tables-with-different-fields)._ From 5a48cddc3c167d80566ec1f0f87bf4adcc2f01b2 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Mon, 30 Mar 2020 16:16:38 -0600 Subject: [PATCH 12/15] updated stateChanges function, added stateChangesOnly, resolves #873 --- .../flux/stdlib/monitor/statechanges.md | 22 ++++++++----- .../flux/stdlib/monitor/statechangesonly.md | 33 +++++++++++++++++++ 2 files changed, 47 insertions(+), 8 deletions(-) create mode 100644 content/v2.0/reference/flux/stdlib/monitor/statechangesonly.md diff --git a/content/v2.0/reference/flux/stdlib/monitor/statechanges.md b/content/v2.0/reference/flux/stdlib/monitor/statechanges.md index 4f0b8eda7..33448b524 100644 --- a/content/v2.0/reference/flux/stdlib/monitor/statechanges.md +++ b/content/v2.0/reference/flux/stdlib/monitor/statechanges.md @@ -12,12 +12,8 @@ menu: weight: 202 --- -The `monitor.stateChanges()` function detects state changes in a stream of data and -outputs records that change from `fromLevel` to `toLevel`. - -{{% note %}} -`monitor.stateChanges` operates on data in the `statuses` measurement and requires a `_level` column . -{{% /note %}} +The `monitor.stateChanges()` function detects state changes in a stream of data with +a `_level` column and outputs records that change from `fromLevel` to `toLevel`. _**Function type:** Transformation_ @@ -26,7 +22,7 @@ import "influxdata/influxdb/monitor" monitor.stateChanges( fromLevel: "any", - toLevel: "crit" + toLevel: "any" ) ``` @@ -41,15 +37,25 @@ _**Data type:** String_ ### toLevel The level to detect a change to. The function output records that change to this level. +Defaults to `"any"`. _**Data type:** String_ ## Examples -### Detect when the state changes to critical +##### Detect when the state changes to critical ```js import "influxdata/influxdb/monitor" monitor.from(start: -1h) |> monitor.stateChanges(toLevel: "crit") ``` + +## Function definition +```js +stateChanges = (fromLevel="any", toLevel="any", tables=<-) => { + return + if fromLevel == "any" and toLevel == "any" then tables |> stateChangesOnly() + else tables |> _stateChanges(fromLevel: fromLevel, toLevel: toLevel) +} +``` diff --git a/content/v2.0/reference/flux/stdlib/monitor/statechangesonly.md b/content/v2.0/reference/flux/stdlib/monitor/statechangesonly.md new file mode 100644 index 000000000..2eae4e582 --- /dev/null +++ b/content/v2.0/reference/flux/stdlib/monitor/statechangesonly.md @@ -0,0 +1,33 @@ +--- +title: monitor.stateChangesOnly() function +description: > + The `monitor.stateChangesOnly()` function takes a stream of tables that contains a` _level` + column and returns a stream of tables where each record represents a state change. +menu: + v2_0_ref: + name: monitor.stateChangesOnly + parent: InfluxDB Monitor +weight: 202 +--- + +The `monitor.stateChangesOnly()` function takes a stream of tables that contains a` _level` +column and returns a stream of tables where each record represents a state change. + +_**Function type:** Transformation_ + +```js +import "influxdata/influxdb/monitor" + +monitor.stateChangesOnly() +``` + +## Examples + +##### Return records representing state changes +```js +import "influxdata/influxdb/monitor" + + +monitor.from(start: -1h) + |> monitor.stateChangesOnly() +``` From 662a486eea5f7a6fe12179dd257fc76cb8e8b101 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Mon, 30 Mar 2020 17:05:43 -0600 Subject: [PATCH 13/15] added sample input and output to state change functions, updated for PR feedback --- .../flux/stdlib/monitor/statechanges.md | 28 ++++++++++++++++++- .../flux/stdlib/monitor/statechangesonly.md | 24 +++++++++++++++- 2 files changed, 50 insertions(+), 2 deletions(-) diff --git a/content/v2.0/reference/flux/stdlib/monitor/statechanges.md b/content/v2.0/reference/flux/stdlib/monitor/statechanges.md index 33448b524..9e5ea5b34 100644 --- a/content/v2.0/reference/flux/stdlib/monitor/statechanges.md +++ b/content/v2.0/reference/flux/stdlib/monitor/statechanges.md @@ -1,7 +1,7 @@ --- title: monitor.stateChanges() function description: > - The `monitor.stateChanges()` function detects state changes in a stream of data and + The `monitor.stateChanges()` function detects state changes in a stream of tables and outputs records that change from `fromLevel` to `toLevel`. aliases: - /v2.0/reference/flux/functions/monitor/statechanges/ @@ -51,6 +51,32 @@ monitor.from(start: -1h) |> monitor.stateChanges(toLevel: "crit") ``` +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _level | +|:----- |:------:| +| 0001 | ok | +| 0002 | ok | +| 0003 | warn | +| 0004 | crit | +{{% /flex-content %}} +{{% flex-content %}} +**The following function outputs:** + +```js +monitor.stateChanges( + toLevel: "crit" +) +``` + +| _time | _level | +|:----- |:------:| +| 0004 | crit | +{{% /flex-content %}} +{{< /flex >}} + ## Function definition ```js stateChanges = (fromLevel="any", toLevel="any", tables=<-) => { diff --git a/content/v2.0/reference/flux/stdlib/monitor/statechangesonly.md b/content/v2.0/reference/flux/stdlib/monitor/statechangesonly.md index 2eae4e582..9cac17636 100644 --- a/content/v2.0/reference/flux/stdlib/monitor/statechangesonly.md +++ b/content/v2.0/reference/flux/stdlib/monitor/statechangesonly.md @@ -1,7 +1,7 @@ --- title: monitor.stateChangesOnly() function description: > - The `monitor.stateChangesOnly()` function takes a stream of tables that contains a` _level` + The `monitor.stateChangesOnly()` function takes a stream of tables that contains a` _level` column and returns a stream of tables where each record represents a state change. menu: v2_0_ref: @@ -31,3 +31,25 @@ import "influxdata/influxdb/monitor" monitor.from(start: -1h) |> monitor.stateChangesOnly() ``` + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _level | +|:----- |:------:| +| 0001 | ok | +| 0002 | ok | +| 0003 | warn | +| 0004 | crit | +{{% /flex-content %}} +{{% flex-content %}} +**`monitor.stateChangesOnly()` outputs:** + +| _time | _level | +|:----- |:------:| +| 0002 | ok | +| 0003 | warn | +| 0004 | crit | +{{% /flex-content %}} +{{< /flex >}} From 597e1f3e909bf2d9d8f6b6c5a853b1e88ea285ce Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Tue, 31 Mar 2020 09:49:29 -0600 Subject: [PATCH 14/15] removed extra space from stateChangesOnly description --- .../v2.0/reference/flux/stdlib/monitor/statechangesonly.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/v2.0/reference/flux/stdlib/monitor/statechangesonly.md b/content/v2.0/reference/flux/stdlib/monitor/statechangesonly.md index 9cac17636..492c172a5 100644 --- a/content/v2.0/reference/flux/stdlib/monitor/statechangesonly.md +++ b/content/v2.0/reference/flux/stdlib/monitor/statechangesonly.md @@ -1,7 +1,7 @@ --- title: monitor.stateChangesOnly() function description: > - The `monitor.stateChangesOnly()` function takes a stream of tables that contains a` _level` + The `monitor.stateChangesOnly()` function takes a stream of tables that contains a `_level` column and returns a stream of tables where each record represents a state change. menu: v2_0_ref: @@ -10,7 +10,7 @@ menu: weight: 202 --- -The `monitor.stateChangesOnly()` function takes a stream of tables that contains a` _level` +The `monitor.stateChangesOnly()` function takes a stream of tables that contains a `_level` column and returns a stream of tables where each record represents a state change. _**Function type:** Transformation_ From ec9e79f0316fe31bf22a6f0764f4a3bd26e08bcd Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Tue, 31 Mar 2020 11:26:01 -0600 Subject: [PATCH 15/15] added links to flux 0.65 changelog, fixed typo in shapeData doc --- .../v2.0/reference/flux/stdlib/experimental/geo/shapedata.md | 2 +- content/v2.0/reference/release-notes/flux.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/content/v2.0/reference/flux/stdlib/experimental/geo/shapedata.md b/content/v2.0/reference/flux/stdlib/experimental/geo/shapedata.md index ab4d18c88..2fa2f11e2 100644 --- a/content/v2.0/reference/flux/stdlib/experimental/geo/shapedata.md +++ b/content/v2.0/reference/flux/stdlib/experimental/geo/shapedata.md @@ -18,7 +18,7 @@ Use `geo.shapeData()` to ensure geo-temporal data meets the [requirements of the Geo package](/v2.0/reference/flux/stdlib/experimental/geo/#geo-schema-requirements): 1. Rename existing latitude and longitude fields to `lat` and `lon`. -2. Pivot data into row-wise sets based on the [`correlatinKey`](#correlationkey). +2. Pivot data into row-wise sets based on the [`correlationKey`](#correlationkey). 3. Generate `s2_cell_id` tags using `lat` and `lon` values and a specified [S2 cell level](https://s2geometry.io/resources/s2cell_statistics.html). diff --git a/content/v2.0/reference/release-notes/flux.md b/content/v2.0/reference/release-notes/flux.md index 0897076b4..6e531a9a3 100644 --- a/content/v2.0/reference/release-notes/flux.md +++ b/content/v2.0/reference/release-notes/flux.md @@ -19,9 +19,9 @@ InfluxDB until the next InfluxDB v2.0 release._ ## v0.65.0 [2020-03-27] ### Features -- Add `experimental.join()` function. +- Add [`experimental.join()`](/v2.0/reference/flux/stdlib/experimental/join/) function. - Store comments in the AST and preserve on format. -- Add `shapeData()` function to Geo package. +- Add [`shapeData()`](/v2.0/reference/flux/stdlib/experimental/geo/shapedata/) function to Geo package. - Expose format to Wasm users. ### Bug fixes