diff --git a/assets/styles/layouts/article/_flex.scss b/assets/styles/layouts/article/_flex.scss index 93ba285d0..7f5fca1d9 100644 --- a/assets/styles/layouts/article/_flex.scss +++ b/assets/styles/layouts/article/_flex.scss @@ -3,6 +3,7 @@ .flex-wrapper { display: flex; flex-wrap: wrap; + margin: 1.5rem 0; } .flex-container { diff --git a/content/v2.0/reference/flux/stdlib/experimental/geo/_index.md b/content/v2.0/reference/flux/stdlib/experimental/geo/_index.md index 46d77d612..a95256013 100644 --- a/content/v2.0/reference/flux/stdlib/experimental/geo/_index.md +++ b/content/v2.0/reference/flux/stdlib/experimental/geo/_index.md @@ -59,6 +59,19 @@ generating S2 Cell ID tokens. For example: - **Python:** [`s2sphere.CellId.to_token()`](https://s2sphere.readthedocs.io/en/latest/api.html#s2sphere.CellId) - **Javascript:** [`s2.cellid.toToken()`](https://github.com/mapbox/node-s2/blob/master/API.md#cellidtotoken---string) +### Add S2 Cell IDs to existing geo-temporal data +Use the [`geo.shapeData()` function](/v2.0/reference/flux/stdlib/experimental/geo/shapedata/) +to add `s2_cell_id` tags to data that includes fields with latitude and longitude values. + +```js +//... + |> shapeData( + latField: "latitude", + lonField: "longitude", + level: 10 + ) +``` + ## Region definitions Many functions in the Geo package filter data based on geographic region. Define geographic regions using the following shapes: diff --git a/content/v2.0/reference/flux/stdlib/experimental/geo/filterrows.md b/content/v2.0/reference/flux/stdlib/experimental/geo/filterrows.md index 597af7bf5..a9b849a3a 100644 --- a/content/v2.0/reference/flux/stdlib/experimental/geo/filterrows.md +++ b/content/v2.0/reference/flux/stdlib/experimental/geo/filterrows.md @@ -35,6 +35,20 @@ geo.filterRows( ) ``` +{{% note %}} +#### s2_cell_id must be part of the group key +To filter geo-temporal data with `geo.filterRows()`, `s2_cell_id` must be part +of the [group key](/v2.0/reference/glossary/#group-key). +To add `s2_cell_id` to the group key, use [`experimental.group`](/v2.0/reference/flux/stdlib/experimental/group): + +```js +import "experimental" + +// ... + |> experimental.group(columns: ["s2_cell_id"], mode: "extend") +``` +{{% /note %}} + ### Strict and non-strict filtering In most cases, the specified geographic region does not perfectly align with S2 grid cells. diff --git a/content/v2.0/reference/flux/stdlib/experimental/geo/gridfilter.md b/content/v2.0/reference/flux/stdlib/experimental/geo/gridfilter.md index 82ebdf166..c046f0237 100644 --- a/content/v2.0/reference/flux/stdlib/experimental/geo/gridfilter.md +++ b/content/v2.0/reference/flux/stdlib/experimental/geo/gridfilter.md @@ -39,6 +39,20 @@ geo.gridFilter( ) ``` +{{% note %}} +#### s2_cell_id must be part of the group key +To filter geo-temporal data with `geo.gridFilter()`, `s2_cell_id` must be part +of the [group key](/v2.0/reference/glossary/#group-key). +To add `s2_cell_id` to the group key, use [`experimental.group`](/v2.0/reference/flux/stdlib/experimental/group): + +```js +import "experimental" + +// ... + |> experimental.group(columns: ["s2_cell_id"], mode: "extend") +``` +{{% /note %}} + ### Non-strict and strict filtering In most cases, the specified geographic region does not perfectly align with S2 grid cells. diff --git a/content/v2.0/reference/flux/stdlib/experimental/geo/shapedata.md b/content/v2.0/reference/flux/stdlib/experimental/geo/shapedata.md new file mode 100644 index 000000000..2fa2f11e2 --- /dev/null +++ b/content/v2.0/reference/flux/stdlib/experimental/geo/shapedata.md @@ -0,0 +1,117 @@ +--- +title: geo.shapeData() function +description: > + The `geo.shapeData()` function renames existing latitude and longitude fields to + **lat** and **lon** and adds an **s2_cell_id** tag. + Use `geo.shapeData()` to ensure geo-temporal data meets the requirements of the Geo package. +menu: + v2_0_ref: + name: geo.shapeData + parent: Geo +weight: 401 +v2.0/tags: [functions, geo] +--- + +The `geo.shapeData()` function renames existing latitude and longitude fields to +**lat** and **lon** and adds an **s2_cell_id** tag. +Use `geo.shapeData()` to ensure geo-temporal data meets the +[requirements of the Geo package](/v2.0/reference/flux/stdlib/experimental/geo/#geo-schema-requirements): + +1. Rename existing latitude and longitude fields to `lat` and `lon`. +2. Pivot data into row-wise sets based on the [`correlationKey`](#correlationkey). +3. Generate `s2_cell_id` tags using `lat` and `lon` values and a specified + [S2 cell level](https://s2geometry.io/resources/s2cell_statistics.html). + +_**Function type:** Transformation_ + +```js +import "experimental/geo" + +geo.shapeData( + latField: "latitude", + lonField: "longitude", + level: 10, + correlationKey: ["_time"] +) +``` + +## Parameters + +### latField +Name of the existing field that contains the latitude value in **decimal degrees** (WGS 84). +Field is renamed to `lat`. + +_**Data type:** String_ + +### lonField +Name of the existing field that contains the longitude value in **decimal degrees** (WGS 84). +Field is renamed to `lon`. + +_**Data type:** String_ + +### level +[S2 cell level](https://s2geometry.io/resources/s2cell_statistics.html) to use +when generating the S2 cell ID token. + +_**Data type:** Integer_ + +### correlationKey +List of columns used to uniquely identify a row for output. +Default is `["_time"]`. + +_**Data type:** Array of strings_ + +## Examples + +##### Shape data to meet the requirements of the Geo package +```js +import "experimental/geo" + +from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "example-measurement") + |> geo.shapeData( + latField: "latitude", + lonField: "longitude", + level: 10 + ) +``` + +### geo.shapeData input and output + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _field | _value | +|:----- |:------: | ------:| +| 0001 | latitude | 30.0 | +| 0002 | latitude | 30.5 | +| 0003 | latitude | 30.7 | +| 0004 | latitude | 31.1 | +| • • • | • • • | • • • | +| 0001 | longitude | 20.0 | +| 0002 | longitude | 19.8 | +| 0003 | longitude | 19.2 | +| 0004 | longitude | 19.5 | +{{% /flex-content %}} +{{% flex-content %}} + +**The following function would output:** + +```js +|> geo.shapeData( + latField: "latitude", + lonField: "longitude", + level: 5 +) +``` + +| _time | lat | lon | s2_cell_id | +|:----- |:--------:|:---------:| ----------:| +| 0001 | 30.0 | 20.0 | 138c | +| 0002 | 30.5 | 19.8 | 1384 | +| 0003 | 30.7 | 19.2 | 139c | +| 0004 | 31.1 | 19.5 | 139c | +{{% /flex-content %}} +{{< /flex >}} diff --git a/content/v2.0/reference/flux/stdlib/experimental/group.md b/content/v2.0/reference/flux/stdlib/experimental/group.md index 60c632c0f..34e32a9ac 100644 --- a/content/v2.0/reference/flux/stdlib/experimental/group.md +++ b/content/v2.0/reference/flux/stdlib/experimental/group.md @@ -52,7 +52,9 @@ Appends columns defined in the [`columns` parameter](#columns) to all existing ###### Include the value column in each groups' group key ```js +import "experimental" + from(bucket: "example-bucket") |> range(start: -1m) - |> group(columns: ["_value"], mode: "extend") + |> experimental.group(columns: ["_value"], mode: "extend") ``` diff --git a/content/v2.0/reference/flux/stdlib/experimental/join.md b/content/v2.0/reference/flux/stdlib/experimental/join.md new file mode 100644 index 000000000..26c0d8765 --- /dev/null +++ b/content/v2.0/reference/flux/stdlib/experimental/join.md @@ -0,0 +1,160 @@ +--- +title: experimental.join() function +description: > + The `experimental.join()` function joins two streams of tables on the + group key with the addition of the `_time` column. +menu: + v2_0_ref: + name: experimental.join + parent: Experimental +weight: 302 +--- + +The `experimental.join()` function joins two streams of tables on the +[group key](/v2.0/reference/glossary/#group-key) and `_time` column. +Use the [`fn` parameter](#fn) to map new output tables using values from input tables. + +{{% note %}} +To join streams of tables with different fields or measurements, use [`group()`](/v2.0/reference/flux/stdlib/built-in/transformations/group/) +or [`drop()`](/v2.0/reference/flux/stdlib/built-in/transformations/drop/) to remove +`_field` and `_measurement` from the group key before joining. +_See an example [below](#join-two-streams-of-tables-with-different-fields)._ +{{% /note %}} + +_**Function type:** Transformation_ + +```js +import "experimental" + +// ... + +experimental.join( + left: left, + right: right, + fn: (left, right) => ({left with lv: left._value, rv: right._value }) +) +``` + +{{% note %}} +This function will likely replace the [`join` function](/v2.0/reference/flux/stdlib/built-in/transformations/join/) +when sufficiently vetted. +{{% /note %}} + +## Parameters + +### left +First of two streams of tables to join. + +_**Data type:** Stream of tables_ + +### right +Second of two streams of tables to join. + +_**Data type:** Stream of tables_ + +### fn +A function with `left` and `right` arguments that maps a new output object +using values from the `left` and `right` input objects. +The return value must be an object. + +_**Data type:** Function_ + +## Examples + +### Input and output tables + +**Given the following input tables:** +{{< flex >}} +{{% flex-content %}} +##### left +| _time | _field | _value | +|:----- |:------:| ------:| +| 0001 | temp | 80.1 | +| 0002 | temp | 80.2 | +| 0003 | temp | 79.9 | +| 0004 | temp | 80.0 | +{{% /flex-content %}} +{{% flex-content %}} +##### right +| _time | _field | _value | +|:----- |:------:| ------:| +| 0001 | temp | 72.1 | +| 0002 | temp | 72.2 | +| 0003 | temp | 71.9 | +| 0004 | temp | 72.0 | +{{% /flex-content %}} +{{< /flex >}} + +**The following `experimental.join()` function would output:** + +```js +import "experimental" + +experimental.join( + left: left, + right: right, + fn: (left, right) => ({ + left with + lv: left._value, + rv: right._value, + diff: left._value - right._value + }) +) +``` + +| _time | _field | lv | rv | diff | +|:----- |:------:|:--: |:--: | ----:| +| 0001 | temp | 80.1 | 72.1 | 8.0 | +| 0002 | temp | 80.2 | 72.2 | 8.0 | +| 0003 | temp | 79.9 | 71.9 | 8.0 | +| 0004 | temp | 80.0 | 72.0 | 8.0 | + +--- + +###### Join two streams of tables +```js +import "experimental" + +s1 = from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "foo") + +s2 = from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "bar") + +experimental.join( + left: s1, + right: s2, + fn: (left, right) => ({ + left with + s1_value: left._value, + s2_value: right._value + }) +) +``` + +###### Join two streams of tables with different fields and measurements +```js +import "experimental" + +s1 = from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "foo" and r._field == "bar") + |> group(columns: ["_time", "_measurement", "_field", "_value"], mode: "except") + +s2 = from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "baz" and r._field == "quz") + |> group(columns: ["_time", "_measurement", "_field", "_value"], mode: "except") + +experimental.join( + left: s1, + right: s2, + fn: (left, right) => ({ + left with + bar_value: left._value, + quz_value: right._value + }) +) +``` diff --git a/content/v2.0/reference/flux/stdlib/monitor/statechanges.md b/content/v2.0/reference/flux/stdlib/monitor/statechanges.md index 4f0b8eda7..9e5ea5b34 100644 --- a/content/v2.0/reference/flux/stdlib/monitor/statechanges.md +++ b/content/v2.0/reference/flux/stdlib/monitor/statechanges.md @@ -1,7 +1,7 @@ --- title: monitor.stateChanges() function description: > - The `monitor.stateChanges()` function detects state changes in a stream of data and + The `monitor.stateChanges()` function detects state changes in a stream of tables and outputs records that change from `fromLevel` to `toLevel`. aliases: - /v2.0/reference/flux/functions/monitor/statechanges/ @@ -12,12 +12,8 @@ menu: weight: 202 --- -The `monitor.stateChanges()` function detects state changes in a stream of data and -outputs records that change from `fromLevel` to `toLevel`. - -{{% note %}} -`monitor.stateChanges` operates on data in the `statuses` measurement and requires a `_level` column . -{{% /note %}} +The `monitor.stateChanges()` function detects state changes in a stream of data with +a `_level` column and outputs records that change from `fromLevel` to `toLevel`. _**Function type:** Transformation_ @@ -26,7 +22,7 @@ import "influxdata/influxdb/monitor" monitor.stateChanges( fromLevel: "any", - toLevel: "crit" + toLevel: "any" ) ``` @@ -41,15 +37,51 @@ _**Data type:** String_ ### toLevel The level to detect a change to. The function output records that change to this level. +Defaults to `"any"`. _**Data type:** String_ ## Examples -### Detect when the state changes to critical +##### Detect when the state changes to critical ```js import "influxdata/influxdb/monitor" monitor.from(start: -1h) |> monitor.stateChanges(toLevel: "crit") ``` + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _level | +|:----- |:------:| +| 0001 | ok | +| 0002 | ok | +| 0003 | warn | +| 0004 | crit | +{{% /flex-content %}} +{{% flex-content %}} +**The following function outputs:** + +```js +monitor.stateChanges( + toLevel: "crit" +) +``` + +| _time | _level | +|:----- |:------:| +| 0004 | crit | +{{% /flex-content %}} +{{< /flex >}} + +## Function definition +```js +stateChanges = (fromLevel="any", toLevel="any", tables=<-) => { + return + if fromLevel == "any" and toLevel == "any" then tables |> stateChangesOnly() + else tables |> _stateChanges(fromLevel: fromLevel, toLevel: toLevel) +} +``` diff --git a/content/v2.0/reference/flux/stdlib/monitor/statechangesonly.md b/content/v2.0/reference/flux/stdlib/monitor/statechangesonly.md new file mode 100644 index 000000000..492c172a5 --- /dev/null +++ b/content/v2.0/reference/flux/stdlib/monitor/statechangesonly.md @@ -0,0 +1,55 @@ +--- +title: monitor.stateChangesOnly() function +description: > + The `monitor.stateChangesOnly()` function takes a stream of tables that contains a `_level` + column and returns a stream of tables where each record represents a state change. +menu: + v2_0_ref: + name: monitor.stateChangesOnly + parent: InfluxDB Monitor +weight: 202 +--- + +The `monitor.stateChangesOnly()` function takes a stream of tables that contains a `_level` +column and returns a stream of tables where each record represents a state change. + +_**Function type:** Transformation_ + +```js +import "influxdata/influxdb/monitor" + +monitor.stateChangesOnly() +``` + +## Examples + +##### Return records representing state changes +```js +import "influxdata/influxdb/monitor" + + +monitor.from(start: -1h) + |> monitor.stateChangesOnly() +``` + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _level | +|:----- |:------:| +| 0001 | ok | +| 0002 | ok | +| 0003 | warn | +| 0004 | crit | +{{% /flex-content %}} +{{% flex-content %}} +**`monitor.stateChangesOnly()` outputs:** + +| _time | _level | +|:----- |:------:| +| 0002 | ok | +| 0003 | warn | +| 0004 | crit | +{{% /flex-content %}} +{{< /flex >}} diff --git a/content/v2.0/reference/release-notes/flux.md b/content/v2.0/reference/release-notes/flux.md index 182d55e77..6e531a9a3 100644 --- a/content/v2.0/reference/release-notes/flux.md +++ b/content/v2.0/reference/release-notes/flux.md @@ -16,6 +16,22 @@ Though newer versions of Flux may be available, they will not be included with InfluxDB until the next InfluxDB v2.0 release._ {{% /note %}} +## v0.65.0 [2020-03-27] + +### Features +- Add [`experimental.join()`](/v2.0/reference/flux/stdlib/experimental/join/) function. +- Store comments in the AST and preserve on format. +- Add [`shapeData()`](/v2.0/reference/flux/stdlib/experimental/geo/shapedata/) function to Geo package. +- Expose format to Wasm users. + +### Bug fixes +- Reimplement `stateChanges()` function. +- Remove the `set -x` in the xcc script. +- Publishes Flux as a public npm package. +- Pivot message passing. + +--- + ## v0.64.0 [2020-03-11] ### Features