From 11f4402511a82c46fefac53671c5e3fa999c13d9 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Mon, 21 Jan 2019 23:12:12 -0700 Subject: [PATCH] create flux guides --- .../query-data/flux/get-started/_index.md | 2 +- .../flux/get-started/transform-data.md | 2 +- content/v2.0/query-data/flux/guides/_index.md | 28 + .../query-data/flux/guides/execute-queries.md | 85 +++ .../v2.0/query-data/flux/guides/group-data.md | 671 ++++++++++++++++++ .../v2.0/query-data/flux/guides/histograms.md | 139 ++++ content/v2.0/query-data/flux/guides/join.md | 300 ++++++++ .../flux/guides/regular-expressions.md | 85 +++ .../v2.0/query-data/flux/guides/sort-limit.md | 47 ++ .../flux/guides/window-aggregate.md | 331 +++++++++ 10 files changed, 1688 insertions(+), 2 deletions(-) create mode 100644 content/v2.0/query-data/flux/guides/_index.md create mode 100644 content/v2.0/query-data/flux/guides/execute-queries.md create mode 100644 content/v2.0/query-data/flux/guides/group-data.md create mode 100644 content/v2.0/query-data/flux/guides/histograms.md create mode 100644 content/v2.0/query-data/flux/guides/join.md create mode 100644 content/v2.0/query-data/flux/guides/regular-expressions.md create mode 100644 content/v2.0/query-data/flux/guides/sort-limit.md create mode 100644 content/v2.0/query-data/flux/guides/window-aggregate.md diff --git a/content/v2.0/query-data/flux/get-started/_index.md b/content/v2.0/query-data/flux/get-started/_index.md index fafaaf34e..afe2148cf 100644 --- a/content/v2.0/query-data/flux/get-started/_index.md +++ b/content/v2.0/query-data/flux/get-started/_index.md @@ -51,7 +51,7 @@ are unique to each row. ## Tools for working with Flux -You have multiple [options for writing and running Flux queries](/v2.0/reference/flux/guides/executing-queries), +You have multiple [options for writing and running Flux queries](/v2.0/reference/flux/guides/execute-queries), but as you're getting started, we recommend using the following: ### 1. Data Explorer diff --git a/content/v2.0/query-data/flux/get-started/transform-data.md b/content/v2.0/query-data/flux/get-started/transform-data.md index e0e9eba49..b075ca8d7 100644 --- a/content/v2.0/query-data/flux/get-started/transform-data.md +++ b/content/v2.0/query-data/flux/get-started/transform-data.md @@ -166,7 +166,7 @@ and your own custom functions, but this is a good introduction into the basic sy --- _For a deeper dive into windowing and aggregating data with example data output for each transformation, -view the [Windowing and aggregating data](/v2.0/reference/flux/guides/windowing-aggregating) guide._ +view the [Windowing and aggregating data](/v2.0/reference/flux/guides/window-aggregate) guide._ --- diff --git a/content/v2.0/query-data/flux/guides/_index.md b/content/v2.0/query-data/flux/guides/_index.md new file mode 100644 index 000000000..77be5ea8d --- /dev/null +++ b/content/v2.0/query-data/flux/guides/_index.md @@ -0,0 +1,28 @@ +--- +title: Flux how-to guides +description: Helpful guides that walk through both common and complex tasks and use cases for Flux. +menu: + v2_0: + name: How-to guides + parent: Flux + weight: 3 +--- + +## [Different ways to execute Flux queries](/v2.0/query-data/flux/guides/execute-queries) +A guide that covers the different options for executing Flux queries with InfluxDB and Chronograf v1.7+. + +## [How to window and aggregate data with Flux](/v2.0/query-data/flux/guides/window-aggregate) +This guide walks through windowing and aggregating data with Flux and demonstrates +how data is shaped in the process. + +## [How to group data with Flux](/v2.0/query-data/flux/guides/group-data) +This guide walks through grouping data in Flux with examples of how data is shaped in the process. + +## [How to join data with Flux](/v2.0/query-data/flux/guides/join) +This guide walks through joining data with Flux, illustrating how joined data is output and how it can be used. + +## [How to sort and limit data with Flux](/v2.0/query-data/flux/guides/sort-limit) +This guide walks through sorting and limiting data with Flux. + +## [Regular expressions in Flux](/v2.0/query-data/flux/guides/regular-expressions) +This guide walks through using regular expressions in evaluation logic in Flux functions. diff --git a/content/v2.0/query-data/flux/guides/execute-queries.md b/content/v2.0/query-data/flux/guides/execute-queries.md new file mode 100644 index 000000000..1e94bb6f1 --- /dev/null +++ b/content/v2.0/query-data/flux/guides/execute-queries.md @@ -0,0 +1,85 @@ +--- +title: Execute Flux queries +seotitle: Different ways to execute Flux queries +description: There are multiple ways to execute Flux queries include the InfluxDB user interface, CLI, and API. +menu: + v2_0: + name: Execute Flux queries + parent: How-to guides + weight: 1 +--- + +There are multiple ways to execute Flux queries with InfluxDB and Chronograf v1.7+. +This guide covers the different options: + +1. [Data Explorer](#data-explorer) +2. [Influx REPL](#influx-repl) +3. [Influx query command](#influx-query-command) +5. [InfluxDB API](#influxdb-api) + +## Data Explorer +Flux queries can be built, executed, and visualized in InfluxDB UI's Data Explorer. + +![Data Explorer with Flux](/img/flux-data-explorer.png) + +## Influx REPL +The [`influx repl` command](/v2.0/reference/cli/influx/repl) starts an interactive +read-eval-print-loop (REPL) where you can write and execute Flux queries. + +```bash +influx repl --org org-name +``` + +## Influx query command +You can pass Flux queries to the [`influx query` command](/v2.0/reference/cli/influx/query) +as either a file or raw Flux via stdin. + +###### Run a Flux query from a file +```bash +influx query @/path/to/query.flux +``` + +###### Pass raw Flux via stdin pipe +```bash +influx query - # Return to open the pipe + +data = from(bucket: "example-bucket") |> range(start: -10m) # ... +# ctrl-d to close the pipe and submit the query +``` + +## InfluxDB API +Flux can be used to query InfluxDB through InfluxDB's `/api/v2/query` endpoint. +Queried data is returned in annotated CSV format. + +In your request, set the following: + +- `accept` header to `application/csv` +- `content-type` header to `application/vnd.flux` + +This allows you to POST the Flux query in plain text and receive the annotated CSV response. + +Below is an example `curl` command that queries InfluxDB using Flux: + +{{< code-tabs-wrapper >}} +{{% code-tabs %}} +[Multi-line](#) +[Single-line](#) +{{% /code-tabs %}} + +{{% code-tab-content %}} +```bash +curl localhost:8086/api/v2/query -XPOST -sS \ +-H 'accept:application/csv' \ +-H 'content-type:application/vnd.flux' \ +-d 'from(bucket:"telegraf") + |> range(start:-5m) + |> filter(fn:(r) => r._measurement == "cpu")' +``` +{{% /code-tab-content %}} + +{{% code-tab-content %}} +```bash +curl localhost:8086/api/v2/query -XPOST -sS -H 'accept:application/csv' -H 'content-type:application/vnd.flux' -d 'from(bucket:"telegraf") |> range(start:-5m) |> filter(fn:(r) => r._measurement == "cpu")' +``` +{{% /code-tab-content %}} +{{< /code-tabs-wrapper >}} diff --git a/content/v2.0/query-data/flux/guides/group-data.md b/content/v2.0/query-data/flux/guides/group-data.md new file mode 100644 index 000000000..1b6e6191f --- /dev/null +++ b/content/v2.0/query-data/flux/guides/group-data.md @@ -0,0 +1,671 @@ +--- +title: Group data with Flux +seotitle: How to group data with Flux +description: > + This guide walks through grouping data with Flux by providing examples and + illustrating how data is shaped throughout the process. +menu: + v2_0: + name: Group data + parent: How-to guides + weight: 3 +--- + +With Flux, data can be grouped by any column in your queried data set. +"Grouping" is accomplished by partitioning data into tables where each row shares a common value for specified columns. +This guide walks through grouping data in Flux with examples of how data is shaped in the process. + +## Group keys +Every table has a **group key** – a list of columns which for which every row in the table has the same value. + +###### Example group key +```js +[_start, _stop, _field, _measurement, host] +``` + +Grouping data in Flux is essentially defining the group key of output tables. +Understanding how modifying group keys shapes output data is key to successfully +grouping and transforming data into your desired output. + +## group() Function +Flux's [`group()` function](/v2.0/reference/flux/functions/transformations/group) defines the +group key for output tables, i.e. grouping records based on values for specific columns. + +###### group() example +```js +dataStream + |> group(columns: ["cpu", "host"]) +``` + +###### Resulting group key +```js +[cpu, host] +``` + +The `group()` function has the following parameters: + +### by +An explicit method for defining the group key with an array of strings. +Only columns specified are included in the output group key. + +### except +An implicit method for defining the group key with an array of strings. +All columns **except** those specified are included in the output group key. + +### none +A boolean that removes all grouping and outputs everything as a single table. + +--- + +_For more information, see the [`group()` function](/v2.0/reference/flux/functions/transformations/group)._ + +--- + +## Example grouping operations +To illustrate how grouping works, define a `dataSet` variable that queries System +CPU usage from the `telegraf/autogen` bucket. +Filter the `cpu` tag so it only returns results for each numbered CPU core. + +### Data set +CPU used by system operations for all numbered CPU cores. +It uses a regular expression to filter only numbered cores. + +```js +dataSet = from(bucket: "telegraf/autogen") + |> range(start: -2m) + |> filter(fn: (r) => + r._field == "usage_system" and + r.cpu =~ /cpu[0-9*]/ + ) + |> drop(columns: ["host"]) +``` + +> This example drops the `host` column from the returned data since the CPU data +> is only tracked for a single host and it simplifies the output tables. +> Don't drop the `host` column if monitoring multiple hosts. + +{{% truncate %}} +``` +Table: keys: [_start, _stop, _field, _measurement, cpu] + _start:time _stop:time _field:string _measurement:string cpu:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:34:00.000000000Z 7.892107892107892 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:34:10.000000000Z 7.2 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:34:20.000000000Z 7.4 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:34:30.000000000Z 5.5 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:34:40.000000000Z 7.4 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:34:50.000000000Z 7.5 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:35:00.000000000Z 10.3 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:35:10.000000000Z 9.2 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:35:20.000000000Z 8.4 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:35:30.000000000Z 8.5 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:35:40.000000000Z 8.6 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:35:50.000000000Z 10.2 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:36:00.000000000Z 10.6 + +Table: keys: [_start, _stop, _field, _measurement, cpu] + _start:time _stop:time _field:string _measurement:string cpu:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:34:00.000000000Z 0.7992007992007992 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:34:10.000000000Z 0.7 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:34:20.000000000Z 0.7 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:34:30.000000000Z 0.4 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:34:40.000000000Z 0.7 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:34:50.000000000Z 0.7 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:35:00.000000000Z 1.4 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:35:10.000000000Z 1.2 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:35:20.000000000Z 0.8 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:35:30.000000000Z 0.8991008991008991 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:35:40.000000000Z 0.8008008008008008 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:35:50.000000000Z 0.999000999000999 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:36:00.000000000Z 1.1022044088176353 + +Table: keys: [_start, _stop, _field, _measurement, cpu] + _start:time _stop:time _field:string _measurement:string cpu:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:34:00.000000000Z 4.1 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:34:10.000000000Z 3.6 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:34:20.000000000Z 3.5 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:34:30.000000000Z 2.6 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:34:40.000000000Z 4.5 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:34:50.000000000Z 4.895104895104895 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:35:00.000000000Z 6.906906906906907 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:35:10.000000000Z 5.7 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:35:20.000000000Z 5.1 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:35:30.000000000Z 4.7 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:35:40.000000000Z 5.1 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:35:50.000000000Z 5.9 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:36:00.000000000Z 6.4935064935064934 + +Table: keys: [_start, _stop, _field, _measurement, cpu] + _start:time _stop:time _field:string _measurement:string cpu:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:34:00.000000000Z 0.5005005005005005 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:34:10.000000000Z 0.5 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:34:20.000000000Z 0.5 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:34:30.000000000Z 0.3 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:34:40.000000000Z 0.6 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:34:50.000000000Z 0.6 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:35:00.000000000Z 1.3986013986013985 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:35:10.000000000Z 0.9 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:35:20.000000000Z 0.5005005005005005 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:35:30.000000000Z 0.7 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:35:40.000000000Z 0.6 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:35:50.000000000Z 0.8 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:36:00.000000000Z 0.9 +``` +{{% /truncate %}} + +**Note that the group key is output with each table: `Table: keys: `.** + +![Group example data set](/img/flux/grouping-data-set.png) + +### Group by CPU +Group the `dataSet` stream by the `cpu` column. + +```js +dataSet + |> group(columns: ["cpu"]) +``` + +This won't actually change the structure of the data since it already has `cpu` in the group key and is therefore grouped by `cpu`. +However, notice that it does change the group key: + +{{% truncate %}} +###### Group by CPU output tables +``` +Table: keys: [cpu] + cpu:string _stop:time _time:time _value:float _field:string _measurement:string _start:time +---------------------- ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:00.000000000Z 7.892107892107892 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:10.000000000Z 7.2 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:20.000000000Z 7.4 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:30.000000000Z 5.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:40.000000000Z 7.4 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:50.000000000Z 7.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:00.000000000Z 10.3 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:10.000000000Z 9.2 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:20.000000000Z 8.4 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:30.000000000Z 8.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:40.000000000Z 8.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:50.000000000Z 10.2 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:36:00.000000000Z 10.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [cpu] + cpu:string _stop:time _time:time _value:float _field:string _measurement:string _start:time +---------------------- ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:00.000000000Z 0.7992007992007992 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:10.000000000Z 0.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:20.000000000Z 0.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:30.000000000Z 0.4 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:40.000000000Z 0.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:50.000000000Z 0.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:00.000000000Z 1.4 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:10.000000000Z 1.2 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:20.000000000Z 0.8 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:30.000000000Z 0.8991008991008991 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:40.000000000Z 0.8008008008008008 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:50.000000000Z 0.999000999000999 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:36:00.000000000Z 1.1022044088176353 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [cpu] + cpu:string _stop:time _time:time _value:float _field:string _measurement:string _start:time +---------------------- ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:00.000000000Z 4.1 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:10.000000000Z 3.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:20.000000000Z 3.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:30.000000000Z 2.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:40.000000000Z 4.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:50.000000000Z 4.895104895104895 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:00.000000000Z 6.906906906906907 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:10.000000000Z 5.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:20.000000000Z 5.1 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:30.000000000Z 4.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:40.000000000Z 5.1 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:50.000000000Z 5.9 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:36:00.000000000Z 6.4935064935064934 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [cpu] + cpu:string _stop:time _time:time _value:float _field:string _measurement:string _start:time +---------------------- ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:00.000000000Z 0.5005005005005005 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:10.000000000Z 0.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:20.000000000Z 0.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:30.000000000Z 0.3 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:40.000000000Z 0.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:50.000000000Z 0.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:00.000000000Z 1.3986013986013985 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:10.000000000Z 0.9 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:20.000000000Z 0.5005005005005005 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:30.000000000Z 0.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:40.000000000Z 0.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:50.000000000Z 0.8 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.9 usage_system cpu 2018-11-05T21:34:00.000000000Z +``` +{{% /truncate %}} + +The visualization remains the same. + +![Group by CPU](/img/flux/grouping-data-set.png) + +### Group by time +Grouping data by the `_time` column is a good illustration of how grouping changes the structure of your data. + +```js +dataSet + |> group(columns: ["_time"]) +``` + +When grouping by `_time`, all records that share a common `_time` value are grouped into individual tables. +So each output table represents a single point in time. + +{{% truncate %}} +###### Group by time output tables +``` +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:34:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 7.892107892107892 usage_system cpu cpu0 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.7992007992007992 usage_system cpu cpu1 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 4.1 usage_system cpu cpu2 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.5005005005005005 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:34:10.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 7.2 usage_system cpu cpu0 +2018-11-05T21:34:10.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.7 usage_system cpu cpu1 +2018-11-05T21:34:10.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 3.6 usage_system cpu cpu2 +2018-11-05T21:34:10.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.5 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:34:20.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 7.4 usage_system cpu cpu0 +2018-11-05T21:34:20.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.7 usage_system cpu cpu1 +2018-11-05T21:34:20.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 3.5 usage_system cpu cpu2 +2018-11-05T21:34:20.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.5 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:34:30.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 5.5 usage_system cpu cpu0 +2018-11-05T21:34:30.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.4 usage_system cpu cpu1 +2018-11-05T21:34:30.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 2.6 usage_system cpu cpu2 +2018-11-05T21:34:30.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.3 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:34:40.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 7.4 usage_system cpu cpu0 +2018-11-05T21:34:40.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.7 usage_system cpu cpu1 +2018-11-05T21:34:40.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 4.5 usage_system cpu cpu2 +2018-11-05T21:34:40.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.6 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:34:50.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 7.5 usage_system cpu cpu0 +2018-11-05T21:34:50.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.7 usage_system cpu cpu1 +2018-11-05T21:34:50.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 4.895104895104895 usage_system cpu cpu2 +2018-11-05T21:34:50.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.6 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:35:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 10.3 usage_system cpu cpu0 +2018-11-05T21:35:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 1.4 usage_system cpu cpu1 +2018-11-05T21:35:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 6.906906906906907 usage_system cpu cpu2 +2018-11-05T21:35:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 1.3986013986013985 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:35:10.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 9.2 usage_system cpu cpu0 +2018-11-05T21:35:10.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 1.2 usage_system cpu cpu1 +2018-11-05T21:35:10.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 5.7 usage_system cpu cpu2 +2018-11-05T21:35:10.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.9 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:35:20.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 8.4 usage_system cpu cpu0 +2018-11-05T21:35:20.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.8 usage_system cpu cpu1 +2018-11-05T21:35:20.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 5.1 usage_system cpu cpu2 +2018-11-05T21:35:20.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.5005005005005005 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:35:30.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 8.5 usage_system cpu cpu0 +2018-11-05T21:35:30.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.8991008991008991 usage_system cpu cpu1 +2018-11-05T21:35:30.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 4.7 usage_system cpu cpu2 +2018-11-05T21:35:30.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.7 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:35:40.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 8.6 usage_system cpu cpu0 +2018-11-05T21:35:40.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.8008008008008008 usage_system cpu cpu1 +2018-11-05T21:35:40.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 5.1 usage_system cpu cpu2 +2018-11-05T21:35:40.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.6 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:35:50.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 10.2 usage_system cpu cpu0 +2018-11-05T21:35:50.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.999000999000999 usage_system cpu cpu1 +2018-11-05T21:35:50.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 5.9 usage_system cpu cpu2 +2018-11-05T21:35:50.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.8 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 10.6 usage_system cpu cpu0 +2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 1.1022044088176353 usage_system cpu cpu1 +2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 6.4935064935064934 usage_system cpu cpu2 +2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.9 usage_system cpu cpu3 +``` +{{% /truncate %}} + +Because each timestamp is a structured as a separate table, when visualized, they appear as individual, unconnected points. +Even though there are multiple records per timestamp, it will only visualize the last record of the table. + +![Group by time](/img/flux/grouping-by-time.png) + +> With some further processing, you could calculate the average CPU usage across all CPUs per point +> of time and group them into a single table, but we won't cover that in this example. +> If you're interested in running and visualizing this yourself, here's what the query would look like: +> +```js +dataSet + |> group(columns: ["_time"]) + |> mean() + |> group(columns: ["_value", "_time"], mode: "except") +``` + +## Group by CPU and time +Group by the `cpu` and `_time` columns. + +```js +dataSet + |> group(columns: ["cpu", "_time"]) +``` + +This outputs a table for every unique `cpu` and `_time` combination: + +{{% truncate %}} +###### Group by CPU and time output tables +``` +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:00.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 7.892107892107892 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:00.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 0.7992007992007992 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:00.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 4.1 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:00.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.5005005005005005 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:10.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 7.2 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:10.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 0.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:10.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 3.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:10.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:20.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 7.4 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:20.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 0.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:20.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 3.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:20.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:30.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 5.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:30.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 0.4 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:30.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 2.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:30.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.3 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:40.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 7.4 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:40.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 0.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:40.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 4.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:40.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:50.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 7.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:50.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 0.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:50.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 4.895104895104895 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:50.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:00.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 10.3 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:00.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 1.4 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:00.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 6.906906906906907 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:00.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 1.3986013986013985 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:10.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 9.2 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:10.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 1.2 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:10.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 5.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:10.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.9 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:20.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 8.4 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:20.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 0.8 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:20.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 5.1 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:20.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.5005005005005005 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:30.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 8.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:30.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 0.8991008991008991 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:30.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 4.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:30.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:40.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 8.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:40.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 0.8008008008008008 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:40.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 5.1 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:40.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:50.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 10.2 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:50.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 0.999000999000999 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:50.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 5.9 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:50.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.8 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:36:00.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 10.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:36:00.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 1.1022044088176353 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:36:00.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 6.4935064935064934 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:36:00.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.9 usage_system cpu 2018-11-05T21:34:00.000000000Z +``` +{{% /truncate %}} + +When visualized, tables appear as individual, unconnected points. + +![Group by CPU and time](/img/flux/grouping-by-cpu-time.png) + +Grouping by `cpu` and `_time` is a good illustration of how grouping works. + +## In conclusion +Grouping is a powerful way to shape your data into your desired output format. +It modifies the group keys of output tables, grouping records into tables that +all share common values within specified columns. diff --git a/content/v2.0/query-data/flux/guides/histograms.md b/content/v2.0/query-data/flux/guides/histograms.md new file mode 100644 index 000000000..ff6f46543 --- /dev/null +++ b/content/v2.0/query-data/flux/guides/histograms.md @@ -0,0 +1,139 @@ +--- +title: Create histograms with Flux +seotitle: How to create histograms with Flux +description: This guide walks through using the histogram() function to create cumulative histograms with Flux. +menu: + v2_0: + name: Create histograms + parent: How-to guides + weight: 7 +--- + + +Histograms provide valuable insight into the distribution of your data. +This guide walks through using Flux's `histogram()` function to transform your data into a **cumulative histogram**. + +## histgram() function +The [`histogram()` function](/v2.0/reference/flux/functions/transformations/histogram) approximates the +cumulative distribution of a dataset by counting data frequencies for a list of "bins." +A **bin** is simply a range in which a data point falls. +All data points that are less than or equal to the bound are counted in the bin. +In the histogram output, a column is added (le) that represents the upper bounds of of each bin. +Bin counts are cumulative. + +```js +from(bucket:"telegraf/autogen") + |> range(start: -5m) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field == "used_percent" + ) + |> histogram(bins: [0.0, 10.0, 20.0, 30.0]) +``` + +> Values output by the `histogram` function represent points of data aggregated over time. +> Since values do not represent single points in time, there is no `_time` column in the output table. + +## Bin helper functions +Flux provides two helper functions for generating histogram bins. +Each generates and outputs an array of floats designed to be used in the `histogram()` function's `bins` parameter. + +### linearBins() +The [`linearBins()` function](/v2.0/reference/flux/functions/misc/linearbins) generates a list of linearly separated floats. + +```js +linearBins(start: 0.0, width: 10.0, count: 10) + +// Generated list: [0, 10, 20, 30, 40, 50, 60, 70, 80, 90, +Inf] +``` + +### logarithmicBins() +The [`logarithmicBins()` function](/v2.0/reference/flux/functions/misc/logarithmicbins) generates a list of exponentially separated floats. + +```js +logarithmicBins(start: 1.0, factor: 2.0, count: 10, infinty: true) + +// Generated list: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512, +Inf] +``` + +## Examples + +### Generating a histogram with linear bins +```js +from(bucket:"telegraf/autogen") + |> range(start: -5m) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field == "used_percent" + ) + |> histogram( + bins: linearBins( + start:65.5, + width: 0.5, + count: 20, + infinity:false + ) + ) +``` + +###### Output table +``` +Table: keys: [_start, _stop, _field, _measurement, host] + _start:time _stop:time _field:string _measurement:string host:string le:float _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------ ---------------------------- ---------------------------- +2018-11-07T22:19:58.423358000Z 2018-11-07T22:24:58.423358000Z used_percent mem Scotts-MacBook-Pro.local 65.5 5 +2018-11-07T22:19:58.423358000Z 2018-11-07T22:24:58.423358000Z used_percent mem Scotts-MacBook-Pro.local 66 6 +2018-11-07T22:19:58.423358000Z 2018-11-07T22:24:58.423358000Z used_percent mem Scotts-MacBook-Pro.local 66.5 8 +2018-11-07T22:19:58.423358000Z 2018-11-07T22:24:58.423358000Z used_percent mem Scotts-MacBook-Pro.local 67 9 +2018-11-07T22:19:58.423358000Z 2018-11-07T22:24:58.423358000Z used_percent mem Scotts-MacBook-Pro.local 67.5 9 +2018-11-07T22:19:58.423358000Z 2018-11-07T22:24:58.423358000Z used_percent mem Scotts-MacBook-Pro.local 68 10 +2018-11-07T22:19:58.423358000Z 2018-11-07T22:24:58.423358000Z used_percent mem Scotts-MacBook-Pro.local 68.5 12 +2018-11-07T22:19:58.423358000Z 2018-11-07T22:24:58.423358000Z used_percent mem Scotts-MacBook-Pro.local 69 12 +2018-11-07T22:19:58.423358000Z 2018-11-07T22:24:58.423358000Z used_percent mem Scotts-MacBook-Pro.local 69.5 15 +2018-11-07T22:19:58.423358000Z 2018-11-07T22:24:58.423358000Z used_percent mem Scotts-MacBook-Pro.local 70 23 +2018-11-07T22:19:58.423358000Z 2018-11-07T22:24:58.423358000Z used_percent mem Scotts-MacBook-Pro.local 70.5 30 +2018-11-07T22:19:58.423358000Z 2018-11-07T22:24:58.423358000Z used_percent mem Scotts-MacBook-Pro.local 71 30 +2018-11-07T22:19:58.423358000Z 2018-11-07T22:24:58.423358000Z used_percent mem Scotts-MacBook-Pro.local 71.5 30 +2018-11-07T22:19:58.423358000Z 2018-11-07T22:24:58.423358000Z used_percent mem Scotts-MacBook-Pro.local 72 30 +2018-11-07T22:19:58.423358000Z 2018-11-07T22:24:58.423358000Z used_percent mem Scotts-MacBook-Pro.local 72.5 30 +2018-11-07T22:19:58.423358000Z 2018-11-07T22:24:58.423358000Z used_percent mem Scotts-MacBook-Pro.local 73 30 +2018-11-07T22:19:58.423358000Z 2018-11-07T22:24:58.423358000Z used_percent mem Scotts-MacBook-Pro.local 73.5 30 +2018-11-07T22:19:58.423358000Z 2018-11-07T22:24:58.423358000Z used_percent mem Scotts-MacBook-Pro.local 74 30 +2018-11-07T22:19:58.423358000Z 2018-11-07T22:24:58.423358000Z used_percent mem Scotts-MacBook-Pro.local 74.5 30 +2018-11-07T22:19:58.423358000Z 2018-11-07T22:24:58.423358000Z used_percent mem Scotts-MacBook-Pro.local 75 30 +``` + +### Generating a histogram with logarithmic bins +```js +from(bucket:"telegraf/autogen") + |> range(start: -5m) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field == "used_percent" + ) + |> histogram( + bins: logarithmicBins( + start:0.5, + factor: 2.0, + count: 10, + infinity:false + ) + ) +``` + +###### Output table +``` +Table: keys: [_start, _stop, _field, _measurement, host] + _start:time _stop:time _field:string _measurement:string host:string le:float _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------ ---------------------------- ---------------------------- +2018-11-07T22:23:36.860664000Z 2018-11-07T22:28:36.860664000Z used_percent mem Scotts-MacBook-Pro.local 0.5 0 +2018-11-07T22:23:36.860664000Z 2018-11-07T22:28:36.860664000Z used_percent mem Scotts-MacBook-Pro.local 1 0 +2018-11-07T22:23:36.860664000Z 2018-11-07T22:28:36.860664000Z used_percent mem Scotts-MacBook-Pro.local 2 0 +2018-11-07T22:23:36.860664000Z 2018-11-07T22:28:36.860664000Z used_percent mem Scotts-MacBook-Pro.local 4 0 +2018-11-07T22:23:36.860664000Z 2018-11-07T22:28:36.860664000Z used_percent mem Scotts-MacBook-Pro.local 8 0 +2018-11-07T22:23:36.860664000Z 2018-11-07T22:28:36.860664000Z used_percent mem Scotts-MacBook-Pro.local 16 0 +2018-11-07T22:23:36.860664000Z 2018-11-07T22:28:36.860664000Z used_percent mem Scotts-MacBook-Pro.local 32 0 +2018-11-07T22:23:36.860664000Z 2018-11-07T22:28:36.860664000Z used_percent mem Scotts-MacBook-Pro.local 64 2 +2018-11-07T22:23:36.860664000Z 2018-11-07T22:28:36.860664000Z used_percent mem Scotts-MacBook-Pro.local 128 30 +2018-11-07T22:23:36.860664000Z 2018-11-07T22:28:36.860664000Z used_percent mem Scotts-MacBook-Pro.local 256 30 +``` diff --git a/content/v2.0/query-data/flux/guides/join.md b/content/v2.0/query-data/flux/guides/join.md new file mode 100644 index 000000000..521bf799b --- /dev/null +++ b/content/v2.0/query-data/flux/guides/join.md @@ -0,0 +1,300 @@ +--- +title: Join data with Flux +seotitle: How to join data with Flux +description: This guide walks through joining data with Flux and outlines how it shapes your data in the process. +menu: + v2_0: + name: Join data + parent: How-to guides + weight: 5 +--- + +The [`join()` function](/v2.0/reference/flux/functions/transformations/join) merges two or more +input streams whose values are equal on a set of common columns into a single output stream. +Flux allows you to join on any columns common between two data streams and opens the door +for operations such as cross-measurement joins and math across measurements. + +To illustrate a join operation, use data captured by Telegraf and and stored in +InfluxDB with a default TICK stack installation - memory usage and processes. + +In this guide, we'll join two data streams, one representing memory usage and the other representing the +total number of running processes, then calculate the average memory usage per running process. + +## Define stream variables +In order to perform a join, you must have two streams of data. +Assign a variable to each data stream. + +### Memory used variable +Define a `memUsed` variable that filters on the `mem` measurement and the `used` field. +This returns the amount of memory (in bytes) used. + +###### memUsed stream definition +```js +memUsed = from(bucket: "telegraf/autogen") + |> range(start: -5m) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field == "used" + ) +``` + +{{% truncate %}} +###### memUsed data output +``` +Table: keys: [_start, _stop, _field, _measurement, host] + _start:time _stop:time _field:string _measurement:string host:string _time:time _value:int +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------ ------------------------------ -------------------------- +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:50:00.000000000Z 10956333056 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:50:10.000000000Z 11014008832 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:50:20.000000000Z 11373428736 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:50:30.000000000Z 11001421824 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:50:40.000000000Z 10985852928 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:50:50.000000000Z 10992279552 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:51:00.000000000Z 11053568000 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:51:10.000000000Z 11092242432 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:51:20.000000000Z 11612774400 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:51:30.000000000Z 11131961344 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:51:40.000000000Z 11124805632 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:51:50.000000000Z 11332464640 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:52:00.000000000Z 11176923136 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:52:10.000000000Z 11181068288 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:52:20.000000000Z 11182579712 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:52:30.000000000Z 11238862848 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:52:40.000000000Z 11275296768 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:52:50.000000000Z 11225411584 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:53:00.000000000Z 11252690944 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:53:10.000000000Z 11227029504 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:53:20.000000000Z 11201646592 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:53:30.000000000Z 11227897856 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:53:40.000000000Z 11330428928 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:53:50.000000000Z 11347976192 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:54:00.000000000Z 11368271872 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:54:10.000000000Z 11269623808 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:54:20.000000000Z 11295637504 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:54:30.000000000Z 11354423296 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:54:40.000000000Z 11379687424 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:54:50.000000000Z 11248926720 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:55:00.000000000Z 11292524544 +``` +{{% /truncate %}} + +### Total processes variable +Define a `procTotal` variable that filters on the `processes` measurement and the `total` field. +This returns the number of running processes. + +###### procTotal stream definition +```js +procTotal = from(bucket: "telegraf/autogen") + |> range(start: -5m) + |> filter(fn: (r) => + r._measurement == "processes" and + r._field == "total" + ) +``` + +{{% truncate %}} +###### procTotal data output +``` +Table: keys: [_start, _stop, _field, _measurement, host] + _start:time _stop:time _field:string _measurement:string host:string _time:time _value:int +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------ ------------------------------ -------------------------- +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:50:00.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:50:10.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:50:20.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:50:30.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:50:40.000000000Z 469 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:50:50.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:51:00.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:51:10.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:51:20.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:51:30.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:51:40.000000000Z 469 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:51:50.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:52:00.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:52:10.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:52:20.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:52:30.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:52:40.000000000Z 472 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:52:50.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:53:00.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:53:10.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:53:20.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:53:30.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:53:40.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:53:50.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:54:00.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:54:10.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:54:20.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:54:30.000000000Z 473 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:54:40.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:54:50.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:55:00.000000000Z 471 +``` +{{% /truncate %}} + +## Join the two data streams +With the two data streams defined, use the `join()` function to join them together. +`join()` requires two parameters: + +##### `tables` +A map of tables to join with keys by which they will be aliased. +In the example below, `mem` is the alias for `memUsed` and `proc` is the alias for `procTotal`. + +##### `on` +An array of strings defining the columns on which the tables will be joined. +_**Both tables must have all columns defined in this list.**_ + +```js +join( + tables: {mem:memUsed, proc:procTotal}, + on: ["_time", "_stop", "_start", "host"] +) +``` + +{{% truncate %}} +###### Joined output table +``` +Table: keys: [_field_mem, _field_proc, _measurement_mem, _measurement_proc, _start, _stop, host] + _field_mem:string _field_proc:string _measurement_mem:string _measurement_proc:string _start:time _stop:time host:string _time:time _value_mem:int _value_proc:int +---------------------- ---------------------- ----------------------- ------------------------ ------------------------------ ------------------------------ ------------------------ ------------------------------ -------------------------- -------------------------- + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:00.000000000Z 10956333056 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:10.000000000Z 11014008832 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:20.000000000Z 11373428736 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:30.000000000Z 11001421824 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:40.000000000Z 10985852928 469 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:50.000000000Z 10992279552 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:00.000000000Z 11053568000 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:10.000000000Z 11092242432 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:20.000000000Z 11612774400 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:30.000000000Z 11131961344 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:40.000000000Z 11124805632 469 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:50.000000000Z 11332464640 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:00.000000000Z 11176923136 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:10.000000000Z 11181068288 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:20.000000000Z 11182579712 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:30.000000000Z 11238862848 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:40.000000000Z 11275296768 472 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:50.000000000Z 11225411584 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:00.000000000Z 11252690944 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:10.000000000Z 11227029504 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:20.000000000Z 11201646592 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:30.000000000Z 11227897856 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:40.000000000Z 11330428928 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:50.000000000Z 11347976192 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:00.000000000Z 11368271872 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:10.000000000Z 11269623808 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:20.000000000Z 11295637504 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:30.000000000Z 11354423296 473 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:40.000000000Z 11379687424 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:50.000000000Z 11248926720 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:55:00.000000000Z 11292524544 471 +``` +{{% /truncate %}} + +Notice the output table includes the following columns: + +- `_field_mem` +- `_field_proc` +- `_measurement_mem` +- `_measurement_proc` +- `_value_mem` +- `_value_proc` + +These represent the columns with values unique to the two input tables. + +## Calculate and create a new table +With the two streams of data joined into a single table, use the [`map()` function](/v2.0/reference/flux/functions/transformations/map) +to build a new table by mapping the existing `_time` column to a new `_time` column and dividing `_value_mem` by `_value_proc` +and mapping it to a new `_value` column. + +```js +join(tables: {mem:memUsed, proc:procTotal}, on: ["_time", "_stop", "_start", "host"]) + |> map(fn: (r) => ({ + _time: r._time, + _value: r._value_mem / r._value_proc + })) +``` + +{{% truncate %}} +###### Mapped table +``` +Table: keys: [_field_mem, _field_proc, _measurement_mem, _measurement_proc, _start, _stop, host] + _field_mem:string _field_proc:string _measurement_mem:string _measurement_proc:string _start:time _stop:time host:string _time:time _value:int +---------------------- ---------------------- ----------------------- ------------------------ ------------------------------ ------------------------------ ------------------------ ------------------------------ -------------------------- + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:00.000000000Z 23311346 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:10.000000000Z 23434061 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:20.000000000Z 24147407 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:30.000000000Z 23407280 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:40.000000000Z 23423993 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:50.000000000Z 23338173 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:00.000000000Z 23518229 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:10.000000000Z 23600515 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:20.000000000Z 24708030 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:30.000000000Z 23685024 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:40.000000000Z 23720267 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:50.000000000Z 24060434 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:00.000000000Z 23730197 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:10.000000000Z 23789506 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:20.000000000Z 23792722 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:30.000000000Z 23861704 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:40.000000000Z 23888340 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:50.000000000Z 23833145 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:00.000000000Z 23941895 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:10.000000000Z 23887296 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:20.000000000Z 23833290 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:30.000000000Z 23838424 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:40.000000000Z 24056112 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:50.000000000Z 24093367 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:00.000000000Z 24136458 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:10.000000000Z 23977922 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:20.000000000Z 23982245 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:30.000000000Z 24005123 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:40.000000000Z 24160695 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:50.000000000Z 23883071 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:55:00.000000000Z 23975635 +``` +{{% /truncate %}} + +This table represents the average amount of memory in bytes per running process. + + +## Real world example +The following function calculates the batch sizes written to an InfluxDB cluster by joining +fields from `httpd` and `write` measurements in order to compare `pointReq` and `writeReq`. +The results are grouped by cluster ID so you can make comparisons across clusters. + +```js +batchSize = (cluster_id, start=-1m, interval=10s) => { + httpd = from(bucket:"telegraf") + |> range(start:start) + |> filter(fn:(r) => + r._measurement == "influxdb_httpd" and + r._field == "writeReq" and + r.cluster_id == cluster_id + ) + |> aggregateWindow(every: interval, fn: mean) + |> derivative(nonNegative:true,unit:60s) + + write = from(bucket:"telegraf") + |> range(start:start) + |> filter(fn:(r) => + r._measurement == "influxdb_write" and + r._field == "pointReq" and + r.cluster_id == cluster_id + ) + |> aggregateWindow(every: interval, fn: max) + |> derivative(nonNegative:true,unit:60s) + + return join( + tables:{httpd:httpd, write:write}, + on:["_time","_stop","_start","host"] + ) + |> map(fn:(r) => ({ + _time: r._time, + _value: r._value_httpd / r._value_write, + })) + |> group(columns: cluster_id) +} + +batchSize(cluster_id: "enter cluster id here") +``` diff --git a/content/v2.0/query-data/flux/guides/regular-expressions.md b/content/v2.0/query-data/flux/guides/regular-expressions.md new file mode 100644 index 000000000..de0862fd2 --- /dev/null +++ b/content/v2.0/query-data/flux/guides/regular-expressions.md @@ -0,0 +1,85 @@ +--- +title: Use regular expressions in Flux +seotitle: How to use regular expressions in Flux +description: This guide walks through using regular expressions in evaluation logic in Flux functions. +menu: + v2_0: + name: Regular expressions + parent: How-to guides + weight: 7 +--- + +Regular expressions (regexes) are incredibly powerful when matching patterns in large collections of data. +With Flux, regular expressions are primarily used for evaluation logic in operations such as filtering rows, +dropping and keeping columns, state detection, etc. +This guide shows how to use regular expressions in your Flux scripts. + +## Go regular expression syntax +Flux uses Go's [regexp package](https://golang.org/pkg/regexp/) for regular expression search. +The links [below](#helpful-links) provide information about Go's regular expression syntax. + +## Regular expression operators +Flux provides two comparison operators for use with regular expressions. + +#### `=~` +When the expression on the left **MATCHES** the regular expression on the right, this evaluates to `true`. + +#### `!~` +When the expression on the left **DOES NOT MATCH** the regular expression on the right, this evaluates to `true`. + +## Regular expressions in Flux +When using regex matching in your Flux scripts, enclose your regular expressions with `/`. +The following is the basic regex comparison syntax: + +###### Basic regex comparison syntax +```js +expression =~ /regex/ +expression !~ /regex/ +``` +## Examples + +### Use a regex to filter by tag value +The following example filters records by the `cpu` tag. +It only keeps records for which the `cpu` is either `cpu0`, `cpu1`, or `cpu2`. + +```js +from(bucket: "telegraf/autogen") + |> range(start: -15m) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_user" and + r.cpu =~ /cpu[0-2]/ + ) +``` + +### Use a regex to filter by field key +The following example excludes records that do not have `_percent` in a field key. + +```js +from(bucket: "telegraf/autogen") + |> range(start: -15m) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field =~ /_percent/ + ) +``` + +### Drop columns matching a regex +The following example drops columns whose names do not being with `_`. + +```js +from(bucket: "telegraf/autogen") + |> range(start: -15m) + |> filter(fn: (r) => r._measurement == "mem") + |> drop(fn: (col) => col !~ /_.*/) +``` + +## Helpful links + +##### Syntax documentation +[regexp Syntax GoDoc](https://godoc.org/regexp/syntax) +[RE2 Syntax Overview](https://github.com/google/re2/wiki/Syntax) + +##### Go regex testers +[Regex Tester - Golang](https://regex-golang.appspot.com/assets/html/index.html) +[Regex101](https://regex101.com/) diff --git a/content/v2.0/query-data/flux/guides/sort-limit.md b/content/v2.0/query-data/flux/guides/sort-limit.md new file mode 100644 index 000000000..c0da4e5ff --- /dev/null +++ b/content/v2.0/query-data/flux/guides/sort-limit.md @@ -0,0 +1,47 @@ +--- +title: Sort and limit data with Flux +seotitle: How to sort and limit data with Flux +description: > + This guide walks through sorting and limiting data with Flux and outlines how + it shapes your data in the process. +menu: + v2_0: + name: Sort and limit data + parent: How-to guides + weight: 6 +--- + +The [`sort()`function](/v2.0/reference/flux/functions/transformations/sort) orders the records within each table. The following example orders system uptime first by region, then host, then value. + +```js +from(bucket:"telegraf/autogen") + |> range(start:-12h) + |> filter(fn: (r) => + r._measurement == "system" and + r._field == "uptime" + ) + |> sort(columns:["region", "host", "_value"]) +``` + +The [`limit()` function](/v2.0/reference/flux/functions/transformations/limit) limit the number of records in output tables to a fixed number (n). The following example shows up to 10 records from the past hour. + +```js +from(bucket:"telegraf/autogen") + |> range(start:-1h) + |> limit(n:10) +``` + +You can use `sort()` and `limit()` together to show the top N records. The example below returns the 10 top system uptime values sorted first by region, then host, then value. + +```js +from(bucket:"telegraf/autogen") + |> range(start:-12h) + |> filter(fn: (r) => + r._measurement == "system" and + r._field == "uptime" + ) + |> sort(columns:["region", "host", "_value"]) + |> limit(n:10) +``` + +You now have created a Flux query that sorts and limits data. Flux also provides the [`top()`](/v2.0/reference/flux/functions/transformations/selectors/top) and [`bottom()`](/v2.0/reference/flux/functions/transformations/selectors/bottom) functions to perform both of these functions at the same time. diff --git a/content/v2.0/query-data/flux/guides/window-aggregate.md b/content/v2.0/query-data/flux/guides/window-aggregate.md new file mode 100644 index 000000000..c0c3fda8e --- /dev/null +++ b/content/v2.0/query-data/flux/guides/window-aggregate.md @@ -0,0 +1,331 @@ +--- +title: Window and aggregate data with Flux +seotitle: How to window and aggregate data with Flux +description: > + This guide walks through windowing and aggregating data with Flux and outlines + how it shapes your data in the process. +menu: + v2_0: + name: Window and aggregate data + parent: How-to guides + weight: 2 +--- + +A common operation performed with time series data is grouping data into windows of time, +or "windowing" data, then aggregating windowed values into a new value. +This guide walks through windowing and aggregating data with Flux and demonstrates +how data is shaped in the process. + +> The following example is an in-depth walk through of the steps required to window and aggregate data. +> The [`aggregateWindow()` function](#summing-up) performs these operations for you, but understanding +> how data is shaped in the process helps to successfully create your desired output. + +## Data set +For the purposes of this guide, define a variable that represents your base data set. +The following example queries the memory usage of the host machine. + +```js +dataSet = from(bucket: "telegraf/autogen") + |> range(start: -5m) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field == "used_percent" + ) + |> drop(columns: ["host"]) +``` + +> This example drops the `host` column from the returned data since the memory data +> is only tracked for a single host and it simplifies the output tables. +> Dropping the `host` column is column is optional and not recommended if monitoring memory +> on multiple hosts. + +`dataSet` can now be used to represent your base data, which will look similar to the following: + +{{% truncate %}} +``` +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:00.000000000Z 71.11611366271973 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:10.000000000Z 67.39630699157715 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:20.000000000Z 64.16666507720947 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:30.000000000Z 64.19951915740967 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:40.000000000Z 64.2122745513916 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:50.000000000Z 64.22209739685059 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:00.000000000Z 64.6336555480957 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:10.000000000Z 64.16516304016113 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:20.000000000Z 64.18349742889404 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:30.000000000Z 64.20474052429199 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:40.000000000Z 68.65062713623047 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:50.000000000Z 67.20139980316162 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:00.000000000Z 70.9143877029419 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:10.000000000Z 64.14549350738525 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:20.000000000Z 64.15379047393799 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:30.000000000Z 64.1592264175415 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:40.000000000Z 64.18190002441406 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:50.000000000Z 64.28837776184082 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:00.000000000Z 64.29731845855713 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:10.000000000Z 64.36963081359863 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:20.000000000Z 64.37397003173828 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:30.000000000Z 64.44413661956787 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:40.000000000Z 64.42906856536865 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:50.000000000Z 64.44573402404785 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:00.000000000Z 64.48912620544434 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:10.000000000Z 64.49522972106934 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:20.000000000Z 64.48652744293213 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:30.000000000Z 64.49949741363525 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:40.000000000Z 64.4949197769165 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:50.000000000Z 64.49787616729736 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49816226959229 +``` +{{% /truncate %}} + +## Windowing data +Use the [`window()` function](/v2.0/reference/flux/functions/transformations/window) to group your data based on time bounds. +The most common parameter passed with the `window()` is `every` which defines the duration of time between windows. +Other parameters are available, but for this example, window the base data set into one minute windows. + +```js +dataSet + |> window(every: 1m) +``` + +Each window of time is output in its own table containing all records that fall within the window. + +{{% truncate %}} +###### window() output tables +``` +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:00.000000000Z 71.11611366271973 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:10.000000000Z 67.39630699157715 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:20.000000000Z 64.16666507720947 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:30.000000000Z 64.19951915740967 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:40.000000000Z 64.2122745513916 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:50.000000000Z 64.22209739685059 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:00.000000000Z 64.6336555480957 +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:10.000000000Z 64.16516304016113 +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:20.000000000Z 64.18349742889404 +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:30.000000000Z 64.20474052429199 +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:40.000000000Z 68.65062713623047 +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:50.000000000Z 67.20139980316162 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:00.000000000Z 70.9143877029419 +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:10.000000000Z 64.14549350738525 +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:20.000000000Z 64.15379047393799 +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:30.000000000Z 64.1592264175415 +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:40.000000000Z 64.18190002441406 +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:50.000000000Z 64.28837776184082 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:00.000000000Z 64.29731845855713 +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:10.000000000Z 64.36963081359863 +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:20.000000000Z 64.37397003173828 +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:30.000000000Z 64.44413661956787 +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:40.000000000Z 64.42906856536865 +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:50.000000000Z 64.44573402404785 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:00.000000000Z 64.48912620544434 +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:10.000000000Z 64.49522972106934 +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:20.000000000Z 64.48652744293213 +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:30.000000000Z 64.49949741363525 +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:40.000000000Z 64.4949197769165 +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:50.000000000Z 64.49787616729736 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:55:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49816226959229 +``` +{{% /truncate %}} + +When visualized in [Chronograf](/chronograf/latest/), each window table is displayed in a different color. + +![Windowed data](/img/flux/simple-windowed-data.png) + +## Aggregate data +[Aggregate functions](/v2.0/reference/flux/functions/transformations/aggregates) take the values +of all rows in a table and use them to perform an aggregate operation. +The result is output as a new value in a single-row table. + +Since windowed data is split into separate tables, aggregate operations run against +each table separately and output new tables containing only the aggregated value. + +For this example, use the [`mean()` function](/v2.0/reference/flux/functions/transformations/aggregates/mean) +to output the average of each window: + +```js +dataSet + |> window(every: 1m) + |> mean() +``` + +{{% truncate %}} +###### mean() output tables +``` +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 65.88549613952637 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 65.50651391347249 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 65.30719598134358 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 64.39330975214641 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 64.49386278788249 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:55:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 64.49816226959229 +``` +{{% /truncate %}} + +Because each data point is contained in its own table, when visualized, +they appear as single, unconnected points. + +![Aggregated windowed data](/img/flux/simple-windowed-aggregate-data.png) + +### Recreate the time column +**Notice the `_time` column is not in the [aggregated output tables](#mean-output-tables).** +Because records in each table are aggregated together, their timestamps no longer +apply and the column is removed from the group key and table. + +Also notice the `_start` and `_stop` columns still exist. +These represent the lower and upper bounds of the time window. + +Many Flux functions rely on the `_time` column. +To further process your data after an aggregate function, you need to add `_time` back in. +Use the [`duplicate()` function](/v2.0/reference/flux/functions/transformations/duplicate) to +duplicate either the `_start` or `_stop` column as a new `_time` column. + +```js +dataSet + |> window(every: 1m) + |> mean() + |> duplicate(column: "_stop", as: "_time") +``` + +{{% truncate %}} +###### duplicate() output tables +``` +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:51:00.000000000Z 65.88549613952637 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:52:00.000000000Z 65.50651391347249 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:53:00.000000000Z 65.30719598134358 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:54:00.000000000Z 64.39330975214641 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49386278788249 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:55:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49816226959229 +``` +{{% /truncate %}} + +## "Unwindow" aggregate tables +Keeping aggregate values in separate tables generally isn't the format in which you want your data. +Use the `window()` function to "unwindow" your data into a single infinite (`inf`) window. + +```js +dataSet + |> window(every: 1m) + |> mean() + |> duplicate(column: "_stop", as: "_time") + |> window(every: inf) +``` + +> Windowing requires a `_time` column which is why it's necessary to +> [recreate the `_time` column](#recreate-the-time-column) after an aggregation. + +###### Unwindowed output table +``` +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:00.000000000Z 65.88549613952637 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:00.000000000Z 65.50651391347249 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:00.000000000Z 65.30719598134358 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:00.000000000Z 64.39330975214641 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49386278788249 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49816226959229 +``` + +With the aggregate values in a single table, data points in the visualization are connected. + +![Unwindowed aggregate data](/img/flux/simple-unwindowed-data.png) + +## Summing up +You have now created a Flux query that windows and aggregates data. +The data transformation process outlined in this guide should be used for all aggregation operations. + +Flux also provides the [`aggregateWindow()` function](/v2.0/reference/flux/functions/transformations/aggregates/aggregatewindow) +which performs all these separate functions for you. + +The following Flux query will return the same results: + +###### aggregateWindow function +```js +dataSet + |> aggregateWindow(every: 1m, fn: mean) +```