From 2c05f032b99e496d6cd75f2d390d998d93d14780 Mon Sep 17 00:00:00 2001 From: Anaisdg <30506042+Anaisdg@users.noreply.github.com> Date: Wed, 22 Jun 2022 10:22:11 -0500 Subject: [PATCH] =?UTF-8?q?added=20how=20to=20guides=20on:=201.=20assignin?= =?UTF-8?q?g=20more=20than=20four=20states=20to=20data=202.=E2=80=A6=20(#4?= =?UTF-8?q?087)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * added how to guides on: 1. assigning more than four states to data 2. selecting specific hours from data 3. monitoring state changes across task executions * Update content/resources/how-to-guides/_index.md Co-authored-by: Scott Anderson * Update content/resources/how-to-guides/_index.md Co-authored-by: Scott Anderson * Update content/resources/how-to-guides/assigning-more-than-four-states.md Co-authored-by: Scott Anderson * Update content/resources/how-to-guides/state-changes-across-task-executions.md Co-authored-by: Scott Anderson * Update content/resources/how-to-guides/state-changes-across-task-executions.md Co-authored-by: Scott Anderson * Update content/resources/how-to-guides/assigning-more-than-four-states.md Co-authored-by: Scott Anderson * Update content/resources/how-to-guides/state-changes-across-task-executions.md Co-authored-by: Scott Anderson * Update content/resources/how-to-guides/state-changes-across-task-executions.md Co-authored-by: Scott Anderson * Update content/resources/how-to-guides/assigning-more-than-four-states.md Co-authored-by: Scott Anderson * Update content/resources/how-to-guides/assigning-more-than-four-states.md Co-authored-by: Scott Anderson * Update content/resources/how-to-guides/assigning-more-than-four-states.md Co-authored-by: Scott Anderson * Update content/resources/how-to-guides/assigning-more-than-four-states.md Co-authored-by: Scott Anderson * Update content/resources/how-to-guides/assigning-more-than-four-states.md Co-authored-by: Scott Anderson * Update content/resources/how-to-guides/assigning-more-than-four-states.md Co-authored-by: Scott Anderson * Apply suggestions from code review Co-authored-by: Scott Anderson * fix suggestions: header corrections, details on packages imported * Apply suggestions from code review Co-authored-by: Scott Anderson Co-authored-by: Anais Dotis-Georgiou Co-authored-by: Scott Anderson --- content/resources/how-to-guides/_index.md | 14 ++ .../assigning-more-than-four-states.md | 159 ++++++++++++++++ .../how-to-guides/select-hours-from-data.md | 40 +++++ .../state-changes-across-task-executions.md | 170 ++++++++++++++++++ 4 files changed, 383 insertions(+) create mode 100644 content/resources/how-to-guides/_index.md create mode 100644 content/resources/how-to-guides/assigning-more-than-four-states.md create mode 100644 content/resources/how-to-guides/select-hours-from-data.md create mode 100644 content/resources/how-to-guides/state-changes-across-task-executions.md diff --git a/content/resources/how-to-guides/_index.md b/content/resources/how-to-guides/_index.md new file mode 100644 index 000000000..2ecb8776d --- /dev/null +++ b/content/resources/how-to-guides/_index.md @@ -0,0 +1,14 @@ +--- +title: InfluxData how-to guides +seotitle: InfluxDB and InfluxData how-to guides +description: > + How-to guides related to InfluxDB and other InfluxData products. +menu: + resources: + name: How-to guides +weight: 1 +--- + +Use the following how-to guides to learn more about InfluxDB and other InfluxData products. + +{{< children >}} \ No newline at end of file diff --git a/content/resources/how-to-guides/assigning-more-than-four-states.md b/content/resources/how-to-guides/assigning-more-than-four-states.md new file mode 100644 index 000000000..82ccd87e1 --- /dev/null +++ b/content/resources/how-to-guides/assigning-more-than-four-states.md @@ -0,0 +1,159 @@ +--- +title: Assign custom states to data +description: > + Learn how overcome a limitation of the `monitor.stateChanges()` function and assign custom states to your data. +menu: + resources: + parent: How-to guides +weight: 101 +--- +## Problem +You may want to use the [`monitor` package](/flux/v0.x/stdlib/influxdata/influxdb/monitor/) and take advantage of functions like [monitor.stateChangesOnly()](flux/v0.x/stdlib/influxdata/influxdb/monitor/statechangesonly/). However, `monitor.stateChangesOnly()` only allows you to monitor four states: "crit", "warn", "ok", and "info". What if you want to be able to assign and monitor state changes across custom states or more than four states? + +## Solution +Define your own custom `stateChangesOnly()` function. Use the function from the source code here and alter it to accommodate more than four levels. Here we account for six different levels instead of just four. + +```js +import "dict" +import "experimental" + +stateChangesOnly = (tables=<-) => { + levelInts = + [ + "customLevel1": 1, + "customLevel2": 2, + "customLevel3": 3, + "customLevel4": 4, + "customLevel5": 5, + "customLevel6": 6, + ] + + return + tables + |> map(fn: (r) => ({r with level_value: dict.get(dict: levelInts, key: r._level, default: 0)})) + |> duplicate(column: "_level", as: "____temp_level____") + |> drop(columns: ["_level"]) + |> rename(columns: {"____temp_level____": "_level"}) + |> sort(columns: ["_source_timestamp", "_time"], desc: false) + |> difference(columns: ["level_value"]) + |> filter(fn: (r) => r.level_value != 0) + |> drop(columns: ["level_value"]) + |> experimental.group(mode: "extend", columns: ["_level"]) +} +``` + +Construct some example data with [`array.from()`](/flux/v0.x/stdlib/array/from/) and map custom levels to it: + +```js +array.from( + rows: [ + {_value: 0.0}, + {_value: 3.0}, + {_value: 5.0}, + {_value: 7.0}, + {_value: 7.5}, + {_value: 9.0}, + {_value: 11.0}, + ], +) + |> map( + fn: (r) => + ({r with _level: + if r._value <= 2.0 then + "customLevel2" + else if r._value <= 4.0 and r._value > 2.0 then + "customLevel3" + else if r._value <= 6.0 and r._value > 4.0 then + "customLevel4" + else if r._value <= 8.0 and r._value > 6.0 then + "customLevel5" + else + "customLevel6", + }), + ) +``` + +Where the example data looks like: + +| _value | _level | +| ------ | ------------ | +| 0.0 | customLevel2 | +| 3.0 | customLevel3 | +| 5.0 | customLevel4 | +| 7.0 | customLevel5 | +| 7.5 | customLevel5 | +| 9.0 | customLevel6 | +| 11.0 | customLevel6 | + +Now apply our custom `stateChangesOnly()` function: + +```js +import "array" +import "dict" +import "experimental" + +stateChangesOnly = (tables=<-) => { + levelInts = + [ + "customLevel1": 1, + "customLevel2": 2, + "customLevel3": 3, + "customLevel4": 4, + "customLevel5": 5, + "customLevel6": 6, + ] + + return + tables + |> map(fn: (r) => ({r with level_value: dict.get(dict: levelInts, key: r._level, default: 0)})) + |> duplicate(column: "_level", as: "____temp_level____") + |> drop(columns: ["_level"]) + |> rename(columns: {"____temp_level____": "_level"}) + |> sort(columns: ["_source_timestamp", "_time"], desc: false) + |> difference(columns: ["level_value"]) + |> filter(fn: (r) => r.level_value != 0) + |> drop(columns: ["level_value"]) + |> experimental.group(mode: "extend", columns: ["_level"]) +} + +data = + array.from( + rows: [ + {_value: 0.0}, + {_value: 3.0}, + {_value: 5.0}, + {_value: 7.0}, + {_value: 7.5}, + {_value: 9.0}, + {_value: 11.0}, + ], + ) + |> map( + fn: (r) => + ({r with _level: + if r._value <= 2.0 then + "customLevel2" + else if r._value <= 4.0 and r._value > 2.0 then + "customLevel3" + else if r._value <= 6.0 and r._value > 4.0 then + "customLevel4" + else if r._value <= 8.0 and r._value > 6.0 then + "customLevel5" + else + "customLevel6", + }), + ) + +data + |> stateChangesOnly() +``` + +This returns: + +| _value | _level | +| ------ | ------------ | +| 3.0 | customLevel3 | +| 5.0 | customLevel4 | +| 7.0 | customLevel5 | +| 9.0 | customLevel6 | + diff --git a/content/resources/how-to-guides/select-hours-from-data.md b/content/resources/how-to-guides/select-hours-from-data.md new file mode 100644 index 000000000..7f35f3f95 --- /dev/null +++ b/content/resources/how-to-guides/select-hours-from-data.md @@ -0,0 +1,40 @@ +--- +title: Select data from specific hours +description: > + Learn how to select data from specific hours of the day. +menu: + resources: + parent: How-to guides +weight: 102 +--- + +## Problem +You may want to select data from specific hours of the day. For example, you may only want data within normal business hours (9am - 5pm). + +## Solution 1 +Use [hourSelection()](/flux/v0.x/stdlib/universe/hourselection/) to filter data by a specific hour range in each day. + +```js +import "date" + +from(bucket: "example-bucket") + |> range(start: -7d) + |> filter(fn: (r) => r["_measurement"] == "example-measurement") + |> filter(fn: (r) => r["_field"] == "example-field") + |> hourSelection(start: 9, stop: 17) +``` + + +## Solution 2 +Use [date.hour()](/flux/v0.x/stdlib/date/hour/) to evaluate hours in a `filter()` predicate. + +```js +import "date" + +from(bucket: "example-bucket") + |> range(start: -7d) + |> filter(fn: (r) => r["_measurement"] == "example-measurement") + |> filter(fn: (r) => r["_field"] == "example-field") + |> filter(fn: (r) => date.hour(t: r["_time"]) > 9 and date.hour(t: r["_time"]) < 17) + +This solution also applies if you to select data from certain seconds in a minute, minutes in an hour, days in the month, months in the year, etc. Use the [Flux `date` package](/flux/v0.x/stdlib/date/) to assign integer representations to your data and filter for your desired schedule. \ No newline at end of file diff --git a/content/resources/how-to-guides/state-changes-across-task-executions.md b/content/resources/how-to-guides/state-changes-across-task-executions.md new file mode 100644 index 000000000..bf3ac6c3b --- /dev/null +++ b/content/resources/how-to-guides/state-changes-across-task-executions.md @@ -0,0 +1,170 @@ +--- +title: Track state changes across task executions +description: > + Learn how to monitor state changes across task executions, so you don't miss changes across subsequent task runs. +menu: + resources: + parent: How-to guides +weight: 103 +--- + +## Problem + +It's common to use [InfluxDB tasks](/influxdb/cloud/process-data/) to evaluate and assign states to your time series data and then detect changes in those states. Tasks process data in batches, but what happens if there is a state change across the batch boundary? The task won't recognize it without knowing the final state of the previous task execution. This guide walks through creating a task that assigns a state to rows and then uses results from the previous task execution to detect any state changes across the batch boundary so you don’t miss any state changes. + +## Solution + +Explicitly assign levels to your data based on thresholds. + +### Solution Advantages +This is the easiest solution to understand if you have never written a task with the [`monitor` package](/flux/v0.x/stdlib/influxdata/influxdb/monitor/). + +### Solution Disadvantages +You have to explicitly define your thresholds, which potentially requires more code. + +### Solution Overview +Create a task where you: + +1. Boilerplate. Import packages and define task options. +2. Query your data. +3. Assign states to your data based on thresholds. Store this data in a variable, i.e. “states”. +4. Write the “states” to a bucket. +5. Find the latest value from the previous task run and store it in a variable “last_state_previous_task”. +6. Union “states” and “last_state_previous_task”. Store this data in a variable “unioned_states”. +7. Discover state changes in “unioned_states”. Store this data in a variable “state_changes”. +8. Notify on state changes that span across the last two tasks to catch any state changes that occur across task executions. + +### Solution Explained +1. Import packages and define task options and secrets. Import the following packages: + - [Flux Telegram package](/flux/v0.x/stdlib/contrib/sranka/telegram/): This package + - [Flux InfluxDB secrets package](/flux/v0.x/stdlib/influxdata/influxdb/secrets/): This package contains the [secrets.get()](/flux/v0.x/stdlib/influxdata/influxdb/secrets/get/) function which allows you to retrieve secrets from the InfluxDB secret store. Learn how to [manage secrets](/influxdb/v2.2/security/secrets/) in InfluxDB to use this package. + - [Flux InfluxDB monitoring package](https://docs.influxdata.com/flux/v0.x/stdlib/influxdata/influxdb/monitor/): This package contains functions and tools for monitoring your data. + + + ```js + import "contrib/sranka/telegram" + import "influxdata/influxdb/secrets" + import "influxdata/influxdb/monitor" + + option task = {name: "State changes across tasks", every: 30m, offset: 5m} + + telegram_token = secrets.get(key: "telegram_token") + telegram_channel_ID = secrets.get(key: "telegram_channel_ID") + ``` + +2. Query the data you want to monitor. + + ```js + data = from(bucket: "example-bucket") + // Query for data from the last successful task run or from the 1 every duration ago. + // This ensures that you won’t miss any data. + |> range(start: tasks.lastSuccess(orTime: -task.every)) + |> filter(fn: (r) => r._measurement == "example-measurement") + |> filter(fn: (r) => r.tagKey1 == "example-tag-value") + |> filter(fn: (r) => r._field == "example-field") + ``` + + Where `data` might look like: + + | _measurement | tagKey1 | _field | _value | _time | + | :------------------ | :---------------- | :------------ | -----: | :------------------- | + | example-measurement | example-tag-value | example-field | 30.0 | 2022-01-01T00:00:00Z | + | example-measurement | example-tag-value | example-field | 50.0 | 2022-01-01T00:00:00Z | + + +3. Assign states to your data based on thresholds. Store this data in a variable, i.e. “states”. To simplify this example, there are only two states: "ok" and "crit." Store states in the `_level` column (required by the `monitor` package). + + ```js + states = + data + |> map(fn: (r) => ({r with _level: if r._value > 40.0 then "crit" else "ok"})) + ``` + + Where `states` might look like: + + | _measurement | tagKey1 | _field | _value | _level | _time | + | :------------------ | :---------------- | :------------ | -----: | :----- | :------------------- | + | example-measurement | example-tag-value | example-field | 30.0 | ok | 2022-01-01T00:00:00Z | + | example-measurement | example-tag-value | example-field | 50.0 | crit | 2022-01-01T00:01:00Z | + + +4. Write “states” back to InfluxDB. You can write the data to a new measurement or to a new bucket. To write the data to a new measurement, use [`set()`](/flux/v0.x/stdlib/universe/set/) to update the value of the `_measurement` column in your “states” data. + + ```js + states + // (Optional) Change the measurement name to write the data to a new measurement + |> set(key: "_measurement", value: "new-measurement") + |> to(bucket : "example-bucket") + ``` + +5. Find the latest value from the previous task run and store it in a variable “last_state_previous_task”, + + ```js + last_state_previous_task = + from(bucket: "example-bucket") + |> range(start: date.sub(d: task.every, from: tasks.lastSuccess(orTime: -task.every)) + |> filter(fn: (r) => r._measurement == "example-measurement") + |> filter(fn: (r) => r.tagKey == "example-tag-value") + |> filter(fn: (r) => r._field == "example-field") + |> last() + ``` + + Where `last_state_previous_task` might look like: + + | _measurement | tagKey1 | _field | _value | _level | _time | + | :------------------ | :---------------- | :------------ | -----: | :----- | :------------------- | + | example-measurement | example-tag-value | example-field | 55.0 | crit | 2021-12-31T23:59:00Z | + +6. Union “states” and “last_state_previous_task”. Store this data in a variable “unioned_states”. Use [`sort()`](/flux/v0.x/stdlib/universe/sort/) to ensure rows are ordered by time. + + ```js + unioned_states = + union(tables: [states, last_state_previous_task]) + |> sort(columns: ["_time"], desc: true) + ``` + + Where `unioned_states` might look like: + + | _measurement | tagKey1 | _field | _value | _level | _time | + | :------------------ | :---------------- | :------------ | -----: | :----- | :------------------- | + | example-measurement | example-tag-value | example-field | 55.0 | crit | 2021-12-31T23:59:00Z | + | example-measurement | example-tag-value | example-field | 30.0 | ok | 2022-01-01T00:00:00Z | + | example-measurement | example-tag-value | example-field | 50.0 | crit | 2022-01-01T00:01:00Z | + +7. Use [`monitor.stateChangesOnly()`](/flux/v0.x/stdlib/influxdata/influxdb/monitor/statechangesonly/) to return only rows where the state changed in “unioned_states”. Store this data in a variable, “state_changes”. + + ```js + state_changes = + unioned_states + |> monitor.stateChangesOnly() + ``` + + Where `state_changes` might look like: + + | _measurement | tagKey1 | _field | _value | _level | _time | + | :------------------ | :---------------- | :------------ | -----: | :----- | :------------------- | + | example-measurement | example-tag-value | example-field | 30.0 | ok | 2022-01-01T00:00:00Z | + | example-measurement | example-tag-value | example-field | 50.0 | crit | 2022-01-01T00:01:00Z | + +8. Notify on state changes that span across the last two tasks to catch any state changes that occur across task executions. + + ```js + state_changes = + data + |> map( + fn: (r) => + ({ + _value: + telegram.message( + token: telegram_token, + channel: telegram_channel_ID, + text: "state change at ${r._value} at ${r._time}", + ), + }), + ) + ``` + + Using the unioned data, the following alerts would be sent to Telegram: + + - `state change at 30.0 at 2022-01-01T00:00:00Z` + - `state change at 50.0 at 2022-01-01T00:01:00Z`