updated flux guides with v2 information
parent
11f4402511
commit
1f3708d13d
|
@ -11,9 +11,9 @@ menu:
|
|||
weight: 3
|
||||
---
|
||||
|
||||
With Flux, data can be grouped by any column in your queried data set.
|
||||
"Grouping" is accomplished by partitioning data into tables where each row shares a common value for specified columns.
|
||||
This guide walks through grouping data in Flux with examples of how data is shaped in the process.
|
||||
With Flux, you can group data by any column in your queried data set.
|
||||
"Grouping" partitions data into tables in which each row shares a common value for specified columns.
|
||||
This guide walks through grouping data in Flux and provides examples of how data is shaped in the process.
|
||||
|
||||
## Group keys
|
||||
Every table has a **group key** – a list of columns which for which every row in the table has the same value.
|
||||
|
@ -44,22 +44,13 @@ dataStream
|
|||
|
||||
The `group()` function has the following parameters:
|
||||
|
||||
### by
|
||||
An explicit method for defining the group key with an array of strings.
|
||||
Only columns specified are included in the output group key.
|
||||
### columns
|
||||
The list of columns to include or exclude (depending on the [mode](#mode)) in the grouping operation.
|
||||
|
||||
### except
|
||||
An implicit method for defining the group key with an array of strings.
|
||||
All columns **except** those specified are included in the output group key.
|
||||
### mode
|
||||
The method used to define the group and resulting group key.
|
||||
Possible values include `by` and `except`.
|
||||
|
||||
### none
|
||||
A boolean that removes all grouping and outputs everything as a single table.
|
||||
|
||||
---
|
||||
|
||||
_For more information, see the [`group()` function](/v2.0/reference/flux/functions/transformations/group)._
|
||||
|
||||
---
|
||||
|
||||
## Example grouping operations
|
||||
To illustrate how grouping works, define a `dataSet` variable that queries System
|
||||
|
@ -80,9 +71,11 @@ dataSet = from(bucket: "telegraf/autogen")
|
|||
|> drop(columns: ["host"])
|
||||
```
|
||||
|
||||
> This example drops the `host` column from the returned data since the CPU data
|
||||
> is only tracked for a single host and it simplifies the output tables.
|
||||
> Don't drop the `host` column if monitoring multiple hosts.
|
||||
{{% note %}}
|
||||
This example drops the `host` column from the returned data since the CPU data
|
||||
is only tracked for a single host and it simplifies the output tables.
|
||||
Don't drop the `host` column if monitoring multiple hosts.
|
||||
{{% /note %}}
|
||||
|
||||
{{% truncate %}}
|
||||
```
|
||||
|
@ -158,7 +151,7 @@ Table: keys: [_start, _stop, _field, _measurement, cpu]
|
|||
|
||||
**Note that the group key is output with each table: `Table: keys: <group-key>`.**
|
||||
|
||||
![Group example data set](/img/flux/grouping-data-set.png)
|
||||
![Group example data set](/img/grouping-data-set.png)
|
||||
|
||||
### Group by CPU
|
||||
Group the `dataSet` stream by the `cpu` column.
|
||||
|
@ -168,7 +161,8 @@ dataSet
|
|||
|> group(columns: ["cpu"])
|
||||
```
|
||||
|
||||
This won't actually change the structure of the data since it already has `cpu` in the group key and is therefore grouped by `cpu`.
|
||||
This won't actually change the structure of the data since it already has `cpu`
|
||||
in the group key and is therefore grouped by `cpu`.
|
||||
However, notice that it does change the group key:
|
||||
|
||||
{{% truncate %}}
|
||||
|
@ -246,7 +240,7 @@ Table: keys: [cpu]
|
|||
|
||||
The visualization remains the same.
|
||||
|
||||
![Group by CPU](/img/flux/grouping-data-set.png)
|
||||
![Group by CPU](/img/grouping-data-set.png)
|
||||
|
||||
### Group by time
|
||||
Grouping data by the `_time` column is a good illustration of how grouping changes the structure of your data.
|
||||
|
@ -369,20 +363,22 @@ Table: keys: [_time]
|
|||
{{% /truncate %}}
|
||||
|
||||
Because each timestamp is a structured as a separate table, when visualized, they appear as individual, unconnected points.
|
||||
Even though there are multiple records per timestamp, it will only visualize the last record of the table.
|
||||
Even though there are multiple records per timestamp, it will only visualize the last record of group's table.
|
||||
|
||||
![Group by time](/img/flux/grouping-by-time.png)
|
||||
![Group by time](/img/grouping-by-time.png)
|
||||
|
||||
{{% note %}}
|
||||
With some further processing, you could calculate the average CPU usage across all CPUs per point
|
||||
of time and group them into a single table, but we won't cover that in this example.
|
||||
If you're interested in running and visualizing this yourself, here's what the query would look like:
|
||||
|
||||
> With some further processing, you could calculate the average CPU usage across all CPUs per point
|
||||
> of time and group them into a single table, but we won't cover that in this example.
|
||||
> If you're interested in running and visualizing this yourself, here's what the query would look like:
|
||||
>
|
||||
```js
|
||||
dataSet
|
||||
|> group(columns: ["_time"])
|
||||
|> mean()
|
||||
|> group(columns: ["_value", "_time"], mode: "except")
|
||||
```
|
||||
{{% /note %}}
|
||||
|
||||
## Group by CPU and time
|
||||
Group by the `cpu` and `_time` columns.
|
||||
|
@ -661,7 +657,7 @@ Table: keys: [_time, cpu]
|
|||
|
||||
When visualized, tables appear as individual, unconnected points.
|
||||
|
||||
![Group by CPU and time](/img/flux/grouping-by-cpu-time.png)
|
||||
![Group by CPU and time](/img/grouping-by-cpu-time.png)
|
||||
|
||||
Grouping by `cpu` and `_time` is a good illustration of how grouping works.
|
||||
|
||||
|
|
|
@ -18,7 +18,7 @@ The [`histogram()` function](/v2.0/reference/flux/functions/transformations/hist
|
|||
cumulative distribution of a dataset by counting data frequencies for a list of "bins."
|
||||
A **bin** is simply a range in which a data point falls.
|
||||
All data points that are less than or equal to the bound are counted in the bin.
|
||||
In the histogram output, a column is added (le) that represents the upper bounds of of each bin.
|
||||
In the histogram output, a column is added (`le`) that represents the upper bounds of of each bin.
|
||||
Bin counts are cumulative.
|
||||
|
||||
```js
|
||||
|
@ -31,12 +31,14 @@ from(bucket:"telegraf/autogen")
|
|||
|> histogram(bins: [0.0, 10.0, 20.0, 30.0])
|
||||
```
|
||||
|
||||
> Values output by the `histogram` function represent points of data aggregated over time.
|
||||
> Since values do not represent single points in time, there is no `_time` column in the output table.
|
||||
{{% note %}}
|
||||
Values output by the `histogram` function represent points of data aggregated over time.
|
||||
Since values do not represent single points in time, there is no `_time` column in the output table.
|
||||
{{% /note %}}
|
||||
|
||||
## Bin helper functions
|
||||
Flux provides two helper functions for generating histogram bins.
|
||||
Each generates and outputs an array of floats designed to be used in the `histogram()` function's `bins` parameter.
|
||||
Each generates an array of floats designed to be used in the `histogram()` function's `bins` parameter.
|
||||
|
||||
### linearBins()
|
||||
The [`linearBins()` function](/v2.0/reference/flux/functions/misc/linearbins) generates a list of linearly separated floats.
|
||||
|
|
|
@ -10,12 +10,12 @@ menu:
|
|||
---
|
||||
|
||||
The [`join()` function](/v2.0/reference/flux/functions/transformations/join) merges two or more
|
||||
input streams whose values are equal on a set of common columns into a single output stream.
|
||||
input streams, whose values are equal on a set of common columns, into a single output stream.
|
||||
Flux allows you to join on any columns common between two data streams and opens the door
|
||||
for operations such as cross-measurement joins and math across measurements.
|
||||
|
||||
To illustrate a join operation, use data captured by Telegraf and and stored in
|
||||
InfluxDB with a default TICK stack installation - memory usage and processes.
|
||||
InfluxDB - memory usage and processes.
|
||||
|
||||
In this guide, we'll join two data streams, one representing memory usage and the other representing the
|
||||
total number of running processes, then calculate the average memory usage per running process.
|
||||
|
@ -142,7 +142,7 @@ In the example below, `mem` is the alias for `memUsed` and `proc` is the alias f
|
|||
|
||||
##### `on`
|
||||
An array of strings defining the columns on which the tables will be joined.
|
||||
_**Both tables must have all columns defined in this list.**_
|
||||
_**Both tables must have all columns specified in this list.**_
|
||||
|
||||
```js
|
||||
join(
|
||||
|
@ -203,9 +203,11 @@ Notice the output table includes the following columns:
|
|||
These represent the columns with values unique to the two input tables.
|
||||
|
||||
## Calculate and create a new table
|
||||
With the two streams of data joined into a single table, use the [`map()` function](/v2.0/reference/flux/functions/transformations/map)
|
||||
to build a new table by mapping the existing `_time` column to a new `_time` column and dividing `_value_mem` by `_value_proc`
|
||||
and mapping it to a new `_value` column.
|
||||
With the two streams of data joined into a single table, use the
|
||||
[`map()` function](/v2.0/reference/flux/functions/transformations/map)
|
||||
to build a new table by mapping the existing `_time` column to a new `_time`
|
||||
column and dividing `_value_mem` by `_value_proc` and mapping it to a
|
||||
new `_value` column.
|
||||
|
||||
```js
|
||||
join(tables: {mem:memUsed, proc:procTotal}, on: ["_time", "_stop", "_start", "host"])
|
||||
|
|
|
@ -10,8 +10,8 @@ menu:
|
|||
---
|
||||
|
||||
Regular expressions (regexes) are incredibly powerful when matching patterns in large collections of data.
|
||||
With Flux, regular expressions are primarily used for evaluation logic in operations such as filtering rows,
|
||||
dropping and keeping columns, state detection, etc.
|
||||
With Flux, regular expressions are primarily used for evaluation logic in predicate functions for things
|
||||
such as filtering rows, dropping and keeping columns, state detection, etc.
|
||||
This guide shows how to use regular expressions in your Flux scripts.
|
||||
|
||||
## Go regular expression syntax
|
||||
|
@ -71,7 +71,7 @@ The following example drops columns whose names do not being with `_`.
|
|||
from(bucket: "telegraf/autogen")
|
||||
|> range(start: -15m)
|
||||
|> filter(fn: (r) => r._measurement == "mem")
|
||||
|> drop(fn: (col) => col !~ /_.*/)
|
||||
|> drop(fn: (column) => column !~ /_.*/)
|
||||
```
|
||||
|
||||
## Helpful links
|
||||
|
|
|
@ -11,7 +11,9 @@ menu:
|
|||
weight: 6
|
||||
---
|
||||
|
||||
The [`sort()`function](/v2.0/reference/flux/functions/transformations/sort) orders the records within each table. The following example orders system uptime first by region, then host, then value.
|
||||
The [`sort()`function](/v2.0/reference/flux/functions/transformations/sort)
|
||||
orders the records within each table.
|
||||
The following example orders system uptime first by region, then host, then value.
|
||||
|
||||
```js
|
||||
from(bucket:"telegraf/autogen")
|
||||
|
@ -23,7 +25,9 @@ from(bucket:"telegraf/autogen")
|
|||
|> sort(columns:["region", "host", "_value"])
|
||||
```
|
||||
|
||||
The [`limit()` function](/v2.0/reference/flux/functions/transformations/limit) limit the number of records in output tables to a fixed number (n). The following example shows up to 10 records from the past hour.
|
||||
The [`limit()` function](/v2.0/reference/flux/functions/transformations/limit)
|
||||
limits the number of records in output tables to a fixed number, `n`.
|
||||
The following example shows up to 10 records from the past hour.
|
||||
|
||||
```js
|
||||
from(bucket:"telegraf/autogen")
|
||||
|
@ -31,7 +35,9 @@ from(bucket:"telegraf/autogen")
|
|||
|> limit(n:10)
|
||||
```
|
||||
|
||||
You can use `sort()` and `limit()` together to show the top N records. The example below returns the 10 top system uptime values sorted first by region, then host, then value.
|
||||
You can use `sort()` and `limit()` together to show the top N records.
|
||||
The example below returns the 10 top system uptime values sorted first by
|
||||
region, then host, then value.
|
||||
|
||||
```js
|
||||
from(bucket:"telegraf/autogen")
|
||||
|
@ -44,4 +50,7 @@ from(bucket:"telegraf/autogen")
|
|||
|> limit(n:10)
|
||||
```
|
||||
|
||||
You now have created a Flux query that sorts and limits data. Flux also provides the [`top()`](/v2.0/reference/flux/functions/transformations/selectors/top) and [`bottom()`](/v2.0/reference/flux/functions/transformations/selectors/bottom) functions to perform both of these functions at the same time.
|
||||
You now have created a Flux query that sorts and limits data.
|
||||
Flux also provides the [`top()`](/v2.0/reference/flux/functions/transformations/selectors/top)
|
||||
and [`bottom()`](/v2.0/reference/flux/functions/transformations/selectors/bottom)
|
||||
functions to perform both of these functions at the same time.
|
||||
|
|
|
@ -16,9 +16,11 @@ or "windowing" data, then aggregating windowed values into a new value.
|
|||
This guide walks through windowing and aggregating data with Flux and demonstrates
|
||||
how data is shaped in the process.
|
||||
|
||||
> The following example is an in-depth walk through of the steps required to window and aggregate data.
|
||||
> The [`aggregateWindow()` function](#summing-up) performs these operations for you, but understanding
|
||||
> how data is shaped in the process helps to successfully create your desired output.
|
||||
{{% note %}}
|
||||
The following example is an in-depth walk-through of the steps required to window and aggregate data.
|
||||
The [`aggregateWindow()` function](#summing-up) performs these operations for you, but understanding
|
||||
how data is shaped in the process helps to successfully create your desired output.
|
||||
{{% /note %}}
|
||||
|
||||
## Data set
|
||||
For the purposes of this guide, define a variable that represents your base data set.
|
||||
|
@ -34,10 +36,12 @@ dataSet = from(bucket: "telegraf/autogen")
|
|||
|> drop(columns: ["host"])
|
||||
```
|
||||
|
||||
> This example drops the `host` column from the returned data since the memory data
|
||||
> is only tracked for a single host and it simplifies the output tables.
|
||||
> Dropping the `host` column is column is optional and not recommended if monitoring memory
|
||||
> on multiple hosts.
|
||||
{{% note %}}
|
||||
This example drops the `host` column from the returned data since the memory data
|
||||
is only tracked for a single host and it simplifies the output tables.
|
||||
Dropping the `host` column is column is optional and not recommended if monitoring memory
|
||||
on multiple hosts.
|
||||
{{% /note %}}
|
||||
|
||||
`dataSet` can now be used to represent your base data, which will look similar to the following:
|
||||
|
||||
|
@ -81,9 +85,12 @@ Table: keys: [_start, _stop, _field, _measurement]
|
|||
{{% /truncate %}}
|
||||
|
||||
## Windowing data
|
||||
Use the [`window()` function](/v2.0/reference/flux/functions/transformations/window) to group your data based on time bounds.
|
||||
The most common parameter passed with the `window()` is `every` which defines the duration of time between windows.
|
||||
Other parameters are available, but for this example, window the base data set into one minute windows.
|
||||
Use the [`window()` function](/v2.0/reference/flux/functions/transformations/window)
|
||||
to group your data based on time bounds.
|
||||
The most common parameter passed with the `window()` is `every` which
|
||||
defines the duration of time between windows.
|
||||
Other parameters are available, but for this example, window the base data
|
||||
set into one minute windows.
|
||||
|
||||
```js
|
||||
dataSet
|
||||
|
@ -157,9 +164,9 @@ Table: keys: [_start, _stop, _field, _measurement]
|
|||
```
|
||||
{{% /truncate %}}
|
||||
|
||||
When visualized in [Chronograf](/chronograf/latest/), each window table is displayed in a different color.
|
||||
When visualized in the InfluxDB UI, each window table is displayed in a different color.
|
||||
|
||||
![Windowed data](/img/flux/simple-windowed-data.png)
|
||||
![Windowed data](/img/simple-windowed-data.png)
|
||||
|
||||
## Aggregate data
|
||||
[Aggregate functions](/v2.0/reference/flux/functions/transformations/aggregates) take the values
|
||||
|
@ -221,7 +228,7 @@ Table: keys: [_start, _stop, _field, _measurement]
|
|||
Because each data point is contained in its own table, when visualized,
|
||||
they appear as single, unconnected points.
|
||||
|
||||
![Aggregated windowed data](/img/flux/simple-windowed-aggregate-data.png)
|
||||
![Aggregated windowed data](/img/simple-windowed-aggregate-data.png)
|
||||
|
||||
### Recreate the time column
|
||||
**Notice the `_time` column is not in the [aggregated output tables](#mean-output-tables).**
|
||||
|
@ -232,7 +239,7 @@ Also notice the `_start` and `_stop` columns still exist.
|
|||
These represent the lower and upper bounds of the time window.
|
||||
|
||||
Many Flux functions rely on the `_time` column.
|
||||
To further process your data after an aggregate function, you need to add `_time` back in.
|
||||
To further process your data after an aggregate function, you need to re-add `_time`.
|
||||
Use the [`duplicate()` function](/v2.0/reference/flux/functions/transformations/duplicate) to
|
||||
duplicate either the `_start` or `_stop` column as a new `_time` column.
|
||||
|
||||
|
@ -295,8 +302,10 @@ dataSet
|
|||
|> window(every: inf)
|
||||
```
|
||||
|
||||
> Windowing requires a `_time` column which is why it's necessary to
|
||||
> [recreate the `_time` column](#recreate-the-time-column) after an aggregation.
|
||||
{{% note %}}
|
||||
Windowing requires a `_time` column which is why it's necessary to
|
||||
[recreate the `_time` column](#recreate-the-time-column) after an aggregation.
|
||||
{{% /note %}}
|
||||
|
||||
###### Unwindowed output table
|
||||
```
|
||||
|
@ -313,7 +322,7 @@ Table: keys: [_start, _stop, _field, _measurement]
|
|||
|
||||
With the aggregate values in a single table, data points in the visualization are connected.
|
||||
|
||||
![Unwindowed aggregate data](/img/flux/simple-unwindowed-data.png)
|
||||
![Unwindowed aggregate data](/img/simple-unwindowed-data.png)
|
||||
|
||||
## Summing up
|
||||
You have now created a Flux query that windows and aggregates data.
|
||||
|
|
|
@ -5,7 +5,7 @@ description: >
|
|||
which is designed for querying, analyzing, and acting on data.
|
||||
menu:
|
||||
v2_0_ref:
|
||||
name: Flux language specification
|
||||
name: Flux specification
|
||||
parent: Flux query language
|
||||
weight: 5
|
||||
---
|
||||
|
|
|
@ -3,7 +3,7 @@ title: Assignment and scope
|
|||
description: An assignment binds an identifier to a variable, option, or function. Every identifier in a program must be assigned.
|
||||
menu:
|
||||
v2_0_ref:
|
||||
parent: Flux language specification
|
||||
parent: Flux specification
|
||||
name: Assignment and scope
|
||||
weight: 20
|
||||
---
|
||||
|
|
|
@ -3,7 +3,7 @@ title: Blocks
|
|||
description: A block is a possibly empty sequence of statements within matching braces ({}).
|
||||
menu:
|
||||
v2_0_ref:
|
||||
parent: Flux language specification
|
||||
parent: Flux specification
|
||||
name: Blocks
|
||||
weight: 30
|
||||
---
|
||||
|
|
|
@ -6,7 +6,7 @@ description: >
|
|||
menu:
|
||||
v2_0_ref:
|
||||
name: Built-ins
|
||||
parent: Flux language specification
|
||||
parent: Flux specification
|
||||
weight: 80
|
||||
---
|
||||
|
||||
|
|
|
@ -4,7 +4,7 @@ description: Flux employs a basic data model built from basic data types. The da
|
|||
menu:
|
||||
v2_0_ref:
|
||||
name: Data model
|
||||
parent: Flux language specification
|
||||
parent: Flux specification
|
||||
weight: 1
|
||||
---
|
||||
|
||||
|
|
|
@ -3,7 +3,7 @@ title: Expressions
|
|||
description: An expression specifies the computation of a value by applying the operators and functions to operands.
|
||||
menu:
|
||||
v2_0_ref:
|
||||
parent: Flux language specification
|
||||
parent: Flux specification
|
||||
name: Expressions
|
||||
weight: 40
|
||||
---
|
||||
|
|
|
@ -3,7 +3,7 @@ title: Lexical elements
|
|||
description: Descriptions of Flux comments, tokens, identifiers, keywords, and other lexical elements.
|
||||
menu:
|
||||
v2_0_ref:
|
||||
parent: Flux language specification
|
||||
parent: Flux specification
|
||||
name: Lexical elements
|
||||
weight: 50
|
||||
---
|
||||
|
|
|
@ -3,7 +3,7 @@ title: Notation
|
|||
description: Notation principles for the Flux functional data scripting language.
|
||||
menu:
|
||||
v2_0_ref:
|
||||
parent: Flux language specification
|
||||
parent: Flux specification
|
||||
name: Notation
|
||||
weight: 60
|
||||
---
|
||||
|
|
|
@ -4,7 +4,7 @@ description: Flux supports many types of operators including arithmetic operator
|
|||
menu:
|
||||
v2_0_ref:
|
||||
name: Operators
|
||||
parent: Flux language specification
|
||||
parent: Flux specification
|
||||
weight: 130
|
||||
---
|
||||
|
||||
|
|
|
@ -3,7 +3,7 @@ title: Options
|
|||
description: placeholder
|
||||
menu:
|
||||
v2_0_ref:
|
||||
parent: Flux language specification
|
||||
parent: Flux specification
|
||||
name: Options
|
||||
weight: 110
|
||||
---
|
||||
|
|
|
@ -8,7 +8,7 @@ aliases:
|
|||
- /v2.0/reference/flux/language/programs
|
||||
menu:
|
||||
v2_0_ref:
|
||||
parent: Flux language specification
|
||||
parent: Flux specification
|
||||
name: Packages
|
||||
weight: 70
|
||||
---
|
||||
|
|
|
@ -3,7 +3,7 @@ title: Representation
|
|||
description: Source code is encoded in UTF-8. The text need not be canonicalized.
|
||||
menu:
|
||||
v2_0_ref:
|
||||
parent: Flux language specification
|
||||
parent: Flux specification
|
||||
name: Representation
|
||||
weight: 80
|
||||
---
|
||||
|
|
|
@ -3,7 +3,7 @@ title: Side effects
|
|||
description: A summary of side effects in the Flux functional data scripting language.
|
||||
menu:
|
||||
v2_0_ref:
|
||||
parent: Flux language specification
|
||||
parent: Flux specification
|
||||
name: Side effects
|
||||
weight: 90
|
||||
---
|
||||
|
|
|
@ -3,7 +3,7 @@ title: Statements
|
|||
description: Statements control execution in the Flux functional data scripting language.
|
||||
menu:
|
||||
v2_0_ref:
|
||||
parent: Flux language specification
|
||||
parent: Flux specification
|
||||
name: Statements
|
||||
weight: 100
|
||||
---
|
||||
|
|
|
@ -3,7 +3,7 @@ title: Types
|
|||
description: A type defines the set of values and operations on those values. Types are never explicitly declared as part of the syntax. Types are always inferred from the usage of the value.
|
||||
menu:
|
||||
v2_0_ref:
|
||||
parent: Flux language specification
|
||||
parent: Flux specification
|
||||
name: Types
|
||||
weight: 110
|
||||
---
|
||||
|
|
|
@ -3,7 +3,7 @@ title: Variables
|
|||
description: Flux variables hold values. A variable can only hold values defined by its type.
|
||||
menu:
|
||||
v2_0_ref:
|
||||
parent: Flux language specification
|
||||
parent: Flux specification
|
||||
name: Variables
|
||||
weight: 120
|
||||
---
|
||||
|
|
Binary file not shown.
After Width: | Height: | Size: 13 KiB |
Binary file not shown.
After Width: | Height: | Size: 12 KiB |
Binary file not shown.
After Width: | Height: | Size: 23 KiB |
Binary file not shown.
After Width: | Height: | Size: 14 KiB |
Binary file not shown.
After Width: | Height: | Size: 12 KiB |
Binary file not shown.
After Width: | Height: | Size: 19 KiB |
Loading…
Reference in New Issue