+ Query InfluxDB
+ Syntax basics
+
diff --git a/content/influxdb/v1.7/flux/guides/_index.md b/content/influxdb/v1.7/flux/guides/_index.md
new file mode 100644
index 000000000..965cafaf7
--- /dev/null
+++ b/content/influxdb/v1.7/flux/guides/_index.md
@@ -0,0 +1,35 @@
+---
+title: Query data with Flux
+description: Guides that walk through both common and complex queries and use cases for Flux.
+weight: 3
+menu:
+ influxdb_1_7:
+ name: Query with Flux
+ parent: Flux
+---
+
+The following guides walk through both common and complex queries and use cases for Flux.
+
+{{% note %}}
+#### Example data variable
+Many of the examples provided in the following guides use a `data` variable,
+which represents a basic query that filters data by measurement and field.
+`data` is defined as:
+
+```js
+data = from(bucket: "db/rp")
+ |> range(start: -1h)
+ |> filter(fn: (r) =>
+ r._measurement == "example-measurement" and
+ r._field == "example-field"
+ )
+```
+{{% /note %}}
+
+## Flux query guides
+
+{{< children type="anchored-list" pages="all" >}}
+
+---
+
+{{< children pages="all" readmore="true" hr="true" >}}
diff --git a/content/influxdb/v1.7/flux/guides/calculate-percentages.md b/content/influxdb/v1.7/flux/guides/calculate-percentages.md
new file mode 100644
index 000000000..9db9b7d3f
--- /dev/null
+++ b/content/influxdb/v1.7/flux/guides/calculate-percentages.md
@@ -0,0 +1,207 @@
+---
+title: Calculate percentages with Flux
+list_title: Calculate percentages
+description: >
+ Use `pivot()` or `join()` and the `map()` function to align operand values into rows and calculate a percentage.
+menu:
+ influxdb_1_7:
+ name: Calculate percentages
+ parent: Query with Flux
+weight: 6
+list_query_example: percentages
+---
+
+Calculating percentages from queried data is a common use case for time series data.
+To calculate a percentage in Flux, operands must be in each row.
+Use `map()` to re-map values in the row and calculate a percentage.
+
+**To calculate percentages**
+
+1. Use [`from()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/inputs/from/),
+ [`range()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/range/) and
+ [`filter()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/filter/) to query operands.
+2. Use [`pivot()` or `join()`](/influxdb/v1.7/flux/guides/mathematic-operations/#pivot-vs-join)
+ to align operand values into rows.
+3. Use [`map()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/map/)
+ to divide the numerator operand value by the denominator operand value and multiply by 100.
+
+{{% note %}}
+The following examples use `pivot()` to align operands into rows because
+`pivot()` works in most cases and is more performant than `join()`.
+_See [Pivot vs join](/influxdb/v1.7/flux/guides/mathematic-operations/#pivot-vs-join)._
+{{% /note %}}
+
+```js
+from(bucket: "db/rp")
+ |> range(start: -1h)
+ |> filter(fn: (r) => r._measurement == "m1" and r._field =~ /field[1-2]/ )
+ |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
+ |> map(fn: (r) => ({ r with _value: r.field1 / r.field2 * 100.0 }))
+```
+
+## GPU monitoring example
+The following example queries data from the gpu-monitor bucket and calculates the
+percentage of GPU memory used over time.
+Data includes the following:
+
+- **`gpu` measurement**
+- **`mem_used` field**: used GPU memory in bytes
+- **`mem_total` field**: total GPU memory in bytes
+
+### Query mem_used and mem_total fields
+```js
+from(bucket: "gpu-monitor")
+ |> range(start: 2020-01-01T00:00:00Z)
+ |> filter(fn: (r) => r._measurement == "gpu" and r._field =~ /mem_/)
+```
+
+###### Returns the following stream of tables:
+
+| _time | _measurement | _field | _value |
+|:----- |:------------:|:------: | ------: |
+| 2020-01-01T00:00:00Z | gpu | mem_used | 2517924577 |
+| 2020-01-01T00:00:10Z | gpu | mem_used | 2695091978 |
+| 2020-01-01T00:00:20Z | gpu | mem_used | 2576980377 |
+| 2020-01-01T00:00:30Z | gpu | mem_used | 3006477107 |
+| 2020-01-01T00:00:40Z | gpu | mem_used | 3543348019 |
+| 2020-01-01T00:00:50Z | gpu | mem_used | 4402341478 |
+
+
+
+| _time | _measurement | _field | _value |
+|:----- |:------------:|:------: | ------: |
+| 2020-01-01T00:00:00Z | gpu | mem_total | 8589934592 |
+| 2020-01-01T00:00:10Z | gpu | mem_total | 8589934592 |
+| 2020-01-01T00:00:20Z | gpu | mem_total | 8589934592 |
+| 2020-01-01T00:00:30Z | gpu | mem_total | 8589934592 |
+| 2020-01-01T00:00:40Z | gpu | mem_total | 8589934592 |
+| 2020-01-01T00:00:50Z | gpu | mem_total | 8589934592 |
+
+### Pivot fields into columns
+Use `pivot()` to pivot the `mem_used` and `mem_total` fields into columns.
+Output includes `mem_used` and `mem_total` columns with values for each corresponding `_time`.
+
+```js
+// ...
+ |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
+```
+
+###### Returns the following:
+
+| _time | _measurement | mem_used | mem_total |
+|:----- |:------------:| --------: | ---------: |
+| 2020-01-01T00:00:00Z | gpu | 2517924577 | 8589934592 |
+| 2020-01-01T00:00:10Z | gpu | 2695091978 | 8589934592 |
+| 2020-01-01T00:00:20Z | gpu | 2576980377 | 8589934592 |
+| 2020-01-01T00:00:30Z | gpu | 3006477107 | 8589934592 |
+| 2020-01-01T00:00:40Z | gpu | 3543348019 | 8589934592 |
+| 2020-01-01T00:00:50Z | gpu | 4402341478 | 8589934592 |
+
+### Map new values
+Each row now contains the values necessary to calculate a percentage.
+Use `map()` to re-map values in each row.
+Divide `mem_used` by `mem_total` and multiply by 100 to return the percentage.
+
+{{% note %}}
+To return a precise float percentage value that includes decimal points, the example
+below casts integer field values to floats and multiplies by a float value (`100.0`).
+{{% /note %}}
+
+```js
+// ...
+ |> map(fn: (r) => ({
+ _time: r._time,
+ _measurement: r._measurement,
+ _field: "mem_used_percent",
+ _value: float(v: r.mem_used) / float(v: r.mem_total) * 100.0
+ }))
+```
+##### Query results:
+
+| _time | _measurement | _field | _value |
+|:----- |:------------:|:------: | ------: |
+| 2020-01-01T00:00:00Z | gpu | mem_used_percent | 29.31 |
+| 2020-01-01T00:00:10Z | gpu | mem_used_percent | 31.37 |
+| 2020-01-01T00:00:20Z | gpu | mem_used_percent | 30.00 |
+| 2020-01-01T00:00:30Z | gpu | mem_used_percent | 35.00 |
+| 2020-01-01T00:00:40Z | gpu | mem_used_percent | 41.25 |
+| 2020-01-01T00:00:50Z | gpu | mem_used_percent | 51.25 |
+
+### Full query
+```js
+from(bucket: "gpu-monitor")
+ |> range(start: 2020-01-01T00:00:00Z)
+ |> filter(fn: (r) => r._measurement == "gpu" and r._field =~ /mem_/ )
+ |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
+ |> map(fn: (r) => ({
+ _time: r._time,
+ _measurement: r._measurement,
+ _field: "mem_used_percent",
+ _value: float(v: r.mem_used) / float(v: r.mem_total) * 100.0
+ }))
+```
+
+## Examples
+
+#### Calculate percentages using multiple fields
+```js
+from(bucket: "db/rp")
+ |> range(start: -1h)
+ |> filter(fn: (r) => r._measurement == "example-measurement")
+ |> filter(fn: (r) =>
+ r._field == "used_system" or
+ r._field == "used_user" or
+ r._field == "total"
+ )
+ |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
+ |> map(fn: (r) => ({ r with
+ _value: float(v: r.used_system + r.used_user) / float(v: r.total) * 100.0
+ }))
+```
+
+#### Calculate percentages using multiple measurements
+
+1. Ensure measurements are in the same [bucket](/influxdb/v1.7/flux/get-started/#buckets).
+2. Use `filter()` to include data from both measurements.
+3. Use `group()` to ungroup data and return a single table.
+4. Use `pivot()` to pivot fields into columns.
+5. Use `map()` to re-map rows and perform the percentage calculation.
+
+
+```js
+from(bucket: "db/rp")
+ |> range(start: -1h)
+ |> filter(fn: (r) =>
+ (r._measurement == "m1" or r._measurement == "m2") and
+ (r._field == "field1" or r._field == "field2")
+ )
+ |> group()
+ |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
+ |> map(fn: (r) => ({ r with _value: r.field1 / r.field2 * 100.0 }))
+```
+
+#### Calculate percentages using multiple data sources
+```js
+import "sql"
+import "influxdata/influxdb/secrets"
+
+pgUser = secrets.get(key: "POSTGRES_USER")
+pgPass = secrets.get(key: "POSTGRES_PASSWORD")
+pgHost = secrets.get(key: "POSTGRES_HOST")
+
+t1 = sql.from(
+ driverName: "postgres",
+ dataSourceName: "postgresql://${pgUser}:${pgPass}@${pgHost}",
+ query:"SELECT id, name, available FROM exampleTable"
+)
+
+t2 = from(bucket: "db/rp")
+ |> range(start: -1h)
+ |> filter(fn: (r) =>
+ r._measurement == "example-measurement" and
+ r._field == "example-field"
+ )
+
+join(tables: {t1: t1, t2: t2}, on: ["id"])
+ |> map(fn: (r) => ({ r with _value: r._value_t2 / r.available_t1 * 100.0 }))
+```
diff --git a/content/influxdb/v1.7/flux/guides/conditional-logic.md b/content/influxdb/v1.7/flux/guides/conditional-logic.md
new file mode 100644
index 000000000..fef58a079
--- /dev/null
+++ b/content/influxdb/v1.7/flux/guides/conditional-logic.md
@@ -0,0 +1,195 @@
+---
+title: Query using conditional logic
+seotitle: Query using conditional logic in Flux
+list_title: Conditional logic
+description: >
+ This guide describes how to use Flux conditional expressions, such as `if`,
+ `else`, and `then`, to query and transform data. **Flux evaluates statements from left to right and stops evaluating once a condition matches.**
+menu:
+ influxdb_1_7:
+ name: Conditional logic
+ parent: Query with Flux
+weight: 20
+list_code_example: |
+ ```js
+ if color == "green" then "008000" else "ffffff"
+ ```
+---
+
+Flux provides `if`, `then`, and `else` conditional expressions that allow for powerful and flexible Flux queries.
+
+##### Conditional expression syntax
+```js
+// Pattern
+if 
+
+### Script pane
+The Script pane is where you write your Flux script.
+In its default state, the **Script** pane includes an optional [Script Wizard](/chronograf/v1.7/guides/querying-data/#explore-data-with-flux)
+that uses selected options to build a Flux query for you.
+The generated query includes all the relevant functions and [template variables](#template-variables-in-flux)
+required to return your desired data.
+
+### Functions pane
+The Functions pane provides a list of functions available in your Flux queries.
+Clicking on a function will add it to the end of the script in the Script pane.
+Hovering over a function provides documentation for the function as well as links
+to deep documentation.
+
+### Dynamic sources
+Chronograf can be configured with multiple data sources.
+The **Sources dropdown** allows you to select a specific data source to connect to,
+but a **Dynamic Source** options is also available.
+With a dynamic source, the cell will query data from whatever data source to which
+Chronograf is currently connected.
+Connections are managed under Chronograf's **Configuration** tab.
+
+### View raw data
+As you're building your Flux scripts, each function processes or transforms your
+data is ways specific to the function.
+It can be helpful to view the actual data in order to see how it is being shaped.
+The **View Raw Data** toggle above the data visualization switches between graphed
+data and raw data shown in table form.
+
+
+
+_The **View Raw Data** toggle is only available when using Flux._
+
+## Template variables in Flux
+Chronograf [template variables](/{{< latest "chronograf" >}}/guides/dashboard-template-variables/)
+allow you to alter specific components of cells’ queries using elements provided in the
+Chronograf user interface.
+
+In your Flux query, reference template variables just as you would reference defined Flux variables.
+The following example uses Chronograf's [predefined template variables](#predefined-template-variables),
+`dashboardTime`, `upperDashboardTime`, and `autoInterval`:
+
+```js
+from(bucket: "telegraf/autogen")
+ |> filter(fn: (r) => r._measurement == "cpu")
+ |> range(
+ start: dashboardTime,
+ stop: upperDashboardTime
+ )
+ window(every: autoInterval)
+```
+
+### Predefined template variables
+
+#### dashboardTime
+The `dashboardTime` template variable represents the lower time bound of ranged data.
+It's value is controlled by the time dropdown in your dashboard.
+It should be used to define the `start` parameter of the `range()` function.
+
+```js
+dataSet
+ |> range(
+ start: dashboardTime
+ )
+```
+
+#### upperDashboardTime
+The `upperDashboardTime` template variable represents the upper time bound of ranged data.
+It's value is modified by the time dropdown in your dashboard when using an absolute time range.
+It should be used to define the `stop` parameter of the `range()` function.
+
+```js
+dataSet
+ |> range(
+ start: dashboardTime,
+ stop: upperDashboardTime
+ )
+```
+> As a best practice, always set the `stop` parameter of the `range()` function to `upperDashboardTime` in cell queries.
+> Without it, `stop` defaults to "now" and the absolute upper range bound selected in the time dropdown is not honored,
+> potentially causing unnecessary load on InfluxDB.
+
+#### autoInterval
+The `autoInterval` template variable represents the refresh interval of the dashboard
+and is controlled by the refresh interval dropdown.
+It's typically used to align window intervals created in
+[windowing and aggregation](/influxdb/v1.7/flux/guides/window-aggregate) operations with dashboard refreshes.
+
+```js
+dataSet
+ |> range(
+ start: dashboardTime,
+ stop: upperDashboardTime
+ )
+ |> aggregateWindow(
+ every: autoInterval,
+ fn: mean
+ )
+```
+
+### Custom template variables
+See Flux read-eval-print-loop (REPL).
+ +{{% note %}} +#### Operands must be the same type +Operands in Flux mathematic operations must be the same data type. +For example, integers cannot be used in operations with floats. +Otherwise, you will get an error similar to: + +``` +Error: type error: float != int +``` + +To convert operands to the same type, use [type-conversion functions](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/type-conversions/) +or manually format operands. +The operand data type determines the output data type. +For example: + +```js +100 // Parsed as an integer +100.0 // Parsed as a float + +// Example evaluations +> 20 / 8 +2 + +> 20.0 / 8.0 +2.5 +``` +{{% /note %}} + +## Custom mathematic functions +Flux lets you [create custom functions](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/custom-functions) that use mathematic operations. +View the examples below. + +###### Custom multiplication function +```js +multiply = (x, y) => x * y + +multiply(x: 10, y: 12) +// Returns 120 +``` + +###### Custom percentage function +```js +percent = (sample, total) => (sample / total) * 100.0 + +percent(sample: 20.0, total: 80.0) +// Returns 25.0 +``` + +### Transform values in a data stream +To transform multiple values in an input stream, your function needs to: + +- [Handle piped-forward data](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/custom-functions/#functions-that-manipulate-piped-forward-data). +- Each operand necessary for the calculation exists in each row _(see [Pivot vs join](#pivot-vs-join) below)_. +- Use the [`map()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/map) to iterate over each row. + +The example `multiplyByX()` function below includes: + +- A `tables` parameter that represents the input data stream (`<-`). +- An `x` parameter which is the number by which values in the `_value` column are multiplied. +- A `map()` function that iterates over each row in the input stream. + It uses the `with` operator to preserve existing columns in each row. + It also multiples the `_value` column by `x`. + +```js +multiplyByX = (x, tables=<-) => + tables + |> map(fn: (r) => ({ + r with + _value: r._value * x + }) + ) + +data + |> multiplyByX(x: 10) +``` + +## Examples + +### Convert bytes to gigabytes +To convert active memory from bytes to gigabytes (GB), divide the `active` field +in the `mem` measurement by 1,073,741,824. + +The `map()` function iterates over each row in the piped-forward data and defines +a new `_value` by dividing the original `_value` by 1073741824. + +```js +from(bucket: "db/rp") + |> range(start: -10m) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field == "active" + ) + |> map(fn: (r) => ({ + r with + _value: r._value / 1073741824 + }) + ) +``` + +You could turn that same calculation into a function: + +```js +bytesToGB = (tables=<-) => + tables + |> map(fn: (r) => ({ + r with + _value: r._value / 1073741824 + }) + ) + +data + |> bytesToGB() +``` + +#### Include partial gigabytes +Because the original metric (bytes) is an integer, the output of the operation is an integer and does not include partial GBs. +To calculate partial GBs, convert the `_value` column and its values to floats using the +[`float()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/type-conversions/float) +and format the denominator in the division operation as a float. + +```js +bytesToGB = (tables=<-) => + tables + |> map(fn: (r) => ({ + r with + _value: float(v: r._value) / 1073741824.0 + }) + ) +``` + +### Calculate a percentage +To calculate a percentage, use simple division, then multiply the result by 100. + +```js +> 1.0 / 4.0 * 100.0 +25.0 +``` + +_For an in-depth look at calculating percentages, see [Calculate percentates](/influxdb/v1.7/flux/guides/calculate-percentages)._ + +## Pivot vs join +To query and use values in mathematical operations in Flux, operand values must +exists in a single row. +Both `pivot()` and `join()` will do this, but there are important differences between the two: + +#### Pivot is more performant +`pivot()` reads and operates on a single stream of data. +`join()` requires two streams of data and the overhead of reading and combining +both streams can be significant, especially for larger data sets. + +#### Use join for multiple data sources +Use `join()` when querying data from different buckets or data sources. + +##### Pivot fields into columns for mathematic calculations +```js +data + |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value") + |> map(fn: (r) => ({ r with + _value: (r.field1 + r.field2) / r.field3 * 100.0 + })) +``` + +##### Join multiple data sources for mathematic calculations +```js +import "sql" +import "influxdata/influxdb/secrets" + +pgUser = secrets.get(key: "POSTGRES_USER") +pgPass = secrets.get(key: "POSTGRES_PASSWORD") +pgHost = secrets.get(key: "POSTGRES_HOST") + +t1 = sql.from( + driverName: "postgres", + dataSourceName: "postgresql://${pgUser}:${pgPass}@${pgHost}", + query:"SELECT id, name, available FROM exampleTable" +) + +t2 = from(bucket: "db/rp") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "example-measurement" and + r._field == "example-field" + ) + +join(tables: {t1: t1, t2: t2}, on: ["id"]) + |> map(fn: (r) => ({ r with _value: r._value_t2 / r.available_t1 * 100.0 })) +``` diff --git a/content/influxdb/v1.7/flux/guides/median.md b/content/influxdb/v1.7/flux/guides/median.md new file mode 100644 index 000000000..170decb46 --- /dev/null +++ b/content/influxdb/v1.7/flux/guides/median.md @@ -0,0 +1,144 @@ +--- +title: Find median values +seotitle: Find median values in Flux +list_title: Median +description: > + Use the `median()` function to return a value representing the `0.5` quantile (50th percentile) or median of input data. +weight: 10 +menu: + influxdb_1_7: + parent: Query with Flux + name: Median +list_query_example: median +--- + +Use the [`median()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/median/) +to return a value representing the `0.5` quantile (50th percentile) or median of input data. + +## Select a method for calculating the median +Select one of the following methods to calculate the median: + +- [estimate_tdigest](#estimate-tdigest) +- [exact_mean](#exact-mean) +- [exact_selector](#exact-selector) + +### estimate_tdigest +**(Default)** An aggregate method that uses a [t-digest data structure](https://github.com/tdunning/t-digest) +to compute an accurate `0.5` quantile estimate on large data sources. +Output tables consist of a single row containing the calculated median. + +{{< flex >}} +{{% flex-content %}} +**Given the following input table:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.0 | +| 2020-01-01T00:03:00Z | 2.0 | +| 2020-01-01T00:04:00Z | 3.0 | +{{% /flex-content %}} +{{% flex-content %}} +**`estimate_tdigest` returns:** + +| _value | +|:------:| +| 1.5 | +{{% /flex-content %}} +{{< /flex >}} + +### exact_mean +An aggregate method that takes the average of the two points closest to the `0.5` quantile value. +Output tables consist of a single row containing the calculated median. + +{{< flex >}} +{{% flex-content %}} +**Given the following input table:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.0 | +| 2020-01-01T00:03:00Z | 2.0 | +| 2020-01-01T00:04:00Z | 3.0 | +{{% /flex-content %}} +{{% flex-content %}} +**`exact_mean` returns:** + +| _value | +|:------:| +| 1.5 | +{{% /flex-content %}} +{{< /flex >}} + +### exact_selector +A selector method that returns the data point for which at least 50% of points are less than. +Output tables consist of a single row containing the calculated median. + +{{< flex >}} +{{% flex-content %}} +**Given the following input table:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.0 | +| 2020-01-01T00:03:00Z | 2.0 | +| 2020-01-01T00:04:00Z | 3.0 | +{{% /flex-content %}} +{{% flex-content %}} +**`exact_selector` returns:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:02:00Z | 1.0 | +{{% /flex-content %}} +{{< /flex >}} + +{{% note %}} +The examples below use the [example data variable](/influxdb/v1.7/flux/guides/#example-data-variable). +{{% /note %}} + +## Find the value that represents the median +Use the default method, `"estimate_tdigest"`, to return all rows in a table that +contain values in the 50th percentile of data in the table. + +```js +data + |> median() +``` + +## Find the average of values closest to the median +Use the `exact_mean` method to return a single row per input table containing the +average of the two values closest to the mathematical median of data in the table. + +```js +data + |> median(method: "exact_mean") +``` + +## Find the point with the median value +Use the `exact_selector` method to return a single row per input table containing the +value that 50% of values in the table are less than. + +```js +data + |> median(method: "exact_selector") +``` + +## Use median() with aggregateWindow() +[`aggregateWindow()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/aggregatewindow/) +segments data into windows of time, aggregates data in each window into a single +point, and then removes the time-based segmentation. +It is primarily used to downsample data. + +To specify the [median calculation method](#select-a-method-for-calculating-the-median) in `aggregateWindow()`, use the +[full function syntax](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/aggregatewindow/#specify-parameters-of-the-aggregate-function): + +```js +data + |> aggregateWindow( + every: 5m, + fn: (tables=<-, column) => tables |> median(method: "exact_selector") + ) +``` diff --git a/content/influxdb/v1.7/flux/guides/monitor-states.md b/content/influxdb/v1.7/flux/guides/monitor-states.md new file mode 100644 index 000000000..270e7bb80 --- /dev/null +++ b/content/influxdb/v1.7/flux/guides/monitor-states.md @@ -0,0 +1,149 @@ +--- +title: Monitor states +seotitle: Monitor states and state changes in your events and metrics with Flux. +description: Flux provides several functions to help monitor states and state changes in your data. +menu: + influxdb_1_7: + name: Monitor states + parent: Query with Flux +weight: 20 +--- + +Flux helps you monitor states in your metrics and events: + +- [Find how long a state persists](#find-how-long-a-state-persists) +- [Count the number of consecutive states](#count-the-number-of-consecutive-states) +- [Detect state changes](#example-query-to-count-machine-state) + +If you're just getting started with Flux queries, check out the following: + +- [Get started with Flux](/influxdb/v1.7/flux/get-started/) for a conceptual overview of Flux. +- [Execute queries](/influxdb/v1.7/flux/guides/executing-queries/) to discover a variety of ways to run your queries. + +## Find how long a state persists + +1. Use the [`stateDuration()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/stateduration/) function to calculate how long a column value has remained the same value (or state). Include the following information: + + - **Column to search:** any tag key, tag value, field key, field value, or measurement. + - **Value:** the value (or state) to search for in the specified column. + - **State duration column:** a new column to store the state duration─the length of time that the specified value persists. + - **Unit:** the unit of time (`1s` (by default), `1m`, `1h`) used to increment the state duration. + + + ```js + |> stateDuration( + fn: (r) => + r._column_to_search == "value_to_search_for", + column: "state_duration", + unit: 1s + ) + ``` + +2. Use `stateDuration()` to search each point for the specified value: + + - For the first point that evaluates `true`, the state duration is set to `0`. For each consecutive point that evaluates `true`, the state duration increases by the time interval between each consecutive point (in specified units). + - If the state is `false`, the state duration is reset to `-1`. + +### Example query with stateDuration() + +The following query searches the `doors` bucket over the past 5 minutes to find how many seconds a door has been `closed`. + +```js +from(bucket: "doors") + |> range(start: -5m) + |> stateDuration( + fn: (r) => + r._value == "closed", + column: "door_closed", + unit: 1s + ) +``` + +In this example, `door_closed` is the **State duration** column. If you write data to the `doors` bucket every minute, the state duration increases by `60s` for each consecutive point where `_value` is `closed`. If `_value` is not `closed`, the state duration is reset to `0`. + +#### Query results + +Results for the example query above may look like this (for simplicity, we've omitted the measurement, tag, and field columns): + +```bash +_time _value door_closed +2019-10-26T17:39:16Z closed 0 +2019-10-26T17:40:16Z closed 60 +2019-10-26T17:41:16Z closed 120 +2019-10-26T17:42:16Z open -1 +2019-10-26T17:43:16Z closed 0 +2019-10-26T17:44:27Z closed 60 +``` + +## Count the number of consecutive states + +1. Use the `stateCount()` function and include the following information: + + - **Column to search:** any tag key, tag value, field key, field value, or measurement. + - **Value:** to search for in the specified column. + - **State count column:** a new column to store the state count─the number of consecutive records in which the specified value exists. + + + ```js + |> stateCount + (fn: (r) => + r._column_to_search == "value_to_search_for", + column: "state_count" + ) + ``` + +2. Use `stateCount()` to search each point for the specified value: + + - For the first point that evaluates `true`, the state count is set to `1`. For each consecutive point that evaluates `true`, the state count increases by 1. + - If the state is `false`, the state count is reset to `-1`. + +### Example query with stateCount() + +The following query searches the `doors` bucket over the past 5 minutes and +calculates how many points have `closed` as their `_value`. + +```js +from(bucket: "doors") + |> range(start: -5m) + |> stateDuration( + fn: (r) => + r._value == "closed", + column: "door_closed") +``` + +This example stores the **state count** in the `door_closed` column. +If you write data to the `doors` bucket every minute, the state count increases +by `1` for each consecutive point where `_value` is `closed`. +If `_value` is not `closed`, the state count is reset to `-1`. + +#### Query results + +Results for the example query above may look like this (for simplicity, we've omitted the measurement, tag, and field columns): + +```bash +_time _value door_closed +2019-10-26T17:39:16Z closed 1 +2019-10-26T17:40:16Z closed 2 +2019-10-26T17:41:16Z closed 3 +2019-10-26T17:42:16Z open -1 +2019-10-26T17:43:16Z closed 1 +2019-10-26T17:44:27Z closed 2 +``` + +#### Example query to count machine state + +The following query checks the machine state every minute (idle, assigned, or busy). +InfluxDB searches the `servers` bucket over the past hour and counts records with a machine state of `idle`, `assigned` or `busy`. + +```js +from(bucket: "servers") + |> range(start: -1h) + |> filter(fn: (r) => + r.machine_state == "idle" or + r.machine_state == "assigned" or + r.machine_state == "busy" + ) + |> stateCount(fn: (r) => r.machine_state == "busy", column: "_count") + |> stateCount(fn: (r) => r.machine_state == "assigned", column: "_count") + |> stateCount(fn: (r) => r.machine_state == "idle", column: "_count") +``` diff --git a/content/influxdb/v1.7/flux/guides/moving-average.md b/content/influxdb/v1.7/flux/guides/moving-average.md new file mode 100644 index 000000000..a57862570 --- /dev/null +++ b/content/influxdb/v1.7/flux/guides/moving-average.md @@ -0,0 +1,113 @@ +--- +title: Calculate the moving average +seotitle: Calculate the moving average in Flux +list_title: Moving Average +description: > + Use the `movingAverage()` or `timedMovingAverage()` functions to return the moving average of data. +weight: 10 +menu: + influxdb_1_7: + parent: Query with Flux + name: Moving Average +list_query_example: moving_average +--- + +Use the [`movingAverage()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/movingaverage/) +or [`timedMovingAverage()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/timedmovingaverage/) +functions to return the moving average of data. + +```js +data + |> movingAverage(n: 5) + +// OR + +data + |> timedMovingAverage(every: 5m, period: 10m) +``` + +### movingAverage() +For each row in a table, `movingAverage()` returns the average of the current value and +**previous** values where `n` is the total number of values used to calculate the average. + +If `n = 3`: + +| Row # | Calculation | +|:-----:|:----------- | +| 1 | _Insufficient number of rows_ | +| 2 | _Insufficient number of rows_ | +| 3 | (Row1 + Row2 + Row3) / 3 | +| 4 | (Row2 + Row3 + Row4) / 3 | +| 5 | (Row3 + Row4 + Row5) / 3 | + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.2 | +| 2020-01-01T00:03:00Z | 1.8 | +| 2020-01-01T00:04:00Z | 0.9 | +| 2020-01-01T00:05:00Z | 1.4 | +| 2020-01-01T00:06:00Z | 2.0 | +{{% /flex-content %}} +{{% flex-content %}} +**The following would return:** + +```js +|> movingAverage(n: 3) +``` + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:03:00Z | 1.33 | +| 2020-01-01T00:04:00Z | 1.30 | +| 2020-01-01T00:05:00Z | 1.36 | +| 2020-01-01T00:06:00Z | 1.43 | +{{% /flex-content %}} +{{< /flex >}} + +### timedMovingAverage() +For each row in a table, `timedMovingAverage()` returns the average of the +current value and all row values in the **previous** `period` (duration). +It returns moving averages at a frequency defined by the `every` parameter. + +Each color in the diagram below represents a period of time used to calculate an +average and the time a point representing the average is returned. +If `every = 30m` and `period = 1h`: + +{{< svg "/static/svgs/timed-moving-avg.svg" >}} + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.2 | +| 2020-01-01T00:03:00Z | 1.8 | +| 2020-01-01T00:04:00Z | 0.9 | +| 2020-01-01T00:05:00Z | 1.4 | +| 2020-01-01T00:06:00Z | 2.0 | +{{% /flex-content %}} +{{% flex-content %}} +**The following would return:** + +```js +|> timedMovingAverage( + every: 2m, + period: 4m +) +``` + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:02:00Z | 1.000 | +| 2020-01-01T00:04:00Z | 1.333 | +| 2020-01-01T00:06:00Z | 1.325 | +| 2020-01-01T00:06:00Z | 1.150 | +{{% /flex-content %}} +{{< /flex >}} diff --git a/content/influxdb/v1.7/flux/guides/optimize-queries.md b/content/influxdb/v1.7/flux/guides/optimize-queries.md new file mode 100644 index 000000000..c3689c94f --- /dev/null +++ b/content/influxdb/v1.7/flux/guides/optimize-queries.md @@ -0,0 +1,74 @@ +--- +title: Optimize Flux queries +description: > + Optimize your Flux queries to reduce their memory and compute (CPU) requirements. +weight: 30 +menu: + influxdb_1_7: + name: Optimize queries + parent: Query with Flux +--- + +Optimize your Flux queries to reduce their memory and compute (CPU) requirements. + +- [Start queries with pushdown functions](#start-queries-with-pushdown-functions) +- [Avoid short window durations](#avoid-short-window-durations) +- [Use "heavy" functions sparingly](#use-heavy-functions-sparingly) +- [Balance time range and data precision](#balance-time-range-and-data-precision) + +## Start queries with pushdown functions +Some Flux functions can push their data manipulation down to the underlying +data source rather than storing and manipulating data in memory. +These are known as "pushdown" functions and using them correctly can greatly +reduce the amount of memory necessary to run a query. + +#### Pushdown functions +- [range()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/range/) +- [filter()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/filter/) +- [group()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/group/) + +Use pushdown functions at the beginning of your query. +Once a non-pushdown function runs, Flux pulls data into memory and runs all +subsequent operations there. + +##### Pushdown functions in use +```js +from(bucket: "db/rp") + |> range(start: -1h) // + |> filter(fn: (r) => r.sensor == "abc123") // Pushed to the data source + |> group(columns: ["_field", "host"]) // + + |> aggregateWindow(every: 5m, fn: max) // + |> filter(fn: (r) => r._value >= 90.0) // Run in memory + |> top(n: 10) // +``` + +## Avoid short window durations +Windowing (grouping data based on time intervals) is commonly used to aggregate and downsample data. +Increase performance by avoiding short window durations. +More windows require more compute power to evaluate which window each row should be assigned to. +Reasonable window durations depend on the total time range queried. + +## Use "heavy" functions sparingly +The following functions use more memory or CPU than others. +Consider their necessity in your data processing before using them: + +- [map()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/map/) +- [reduce()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/reduce/) +- [window()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/window/) +- [join()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/join/) +- [union()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/union/) +- [pivot()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/pivot/) + +{{% note %}} +We're continually optimizing Flux and this list may not represent its current state. +{{% /note %}} + +## Balance time range and data precision +To ensure queries are performant, balance the time range and the precision of your data. +For example, if you query data stored every second and request six months worth of data, +results will include a minimum of ≈15.5 million points. +Flux must store these points in memory to generate a response. + +To query data over large periods of time, create a [continuous query](/influxdb/v1.7/query_language/continuous_queries/) +to downsample data, and then query the downsampled data instead. diff --git a/content/influxdb/v1.7/flux/guides/percentile-quantile.md b/content/influxdb/v1.7/flux/guides/percentile-quantile.md new file mode 100644 index 000000000..444a5c9be --- /dev/null +++ b/content/influxdb/v1.7/flux/guides/percentile-quantile.md @@ -0,0 +1,161 @@ +--- +title: Find percentile and quantile values +seotitle: Query percentile and quantile values in Flux +list_title: Percentile & quantile +description: > + Use the `quantile()` function to return all values within the `q` quantile or + percentile of input data. +weight: 10 +menu: + influxdb_1_7: + parent: Query with Flux + name: Percentile & quantile +list_query_example: quantile +--- + +Use the [`quantile()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/quantile/) +to return a value representing the `q` quantile or percentile of input data. + +## Percentile versus quantile +Percentiles and quantiles are very similar, differing only in the number used to calculate return values. +A percentile is calculated using numbers between `0` and `100`. +A quantile is calculated using numbers between `0.0` and `1.0`. +For example, the **`0.5` quantile** is the same as the **50th percentile**. + +## Select a method for calculating the quantile +Select one of the following methods to calculate the quantile: + +- [estimate_tdigest](#estimate-tdigest) +- [exact_mean](#exact-mean) +- [exact_selector](#exact-selector) + +### estimate_tdigest +**(Default)** An aggregate method that uses a [t-digest data structure](https://github.com/tdunning/t-digest) +to compute a quantile estimate on large data sources. +Output tables consist of a single row containing the calculated quantile. + +If calculating the `0.5` quantile or 50th percentile: + +{{< flex >}} +{{% flex-content %}} +**Given the following input table:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.0 | +| 2020-01-01T00:03:00Z | 2.0 | +| 2020-01-01T00:04:00Z | 3.0 | +{{% /flex-content %}} +{{% flex-content %}} +**`estimate_tdigest` returns:** + +| _value | +|:------:| +| 1.5 | +{{% /flex-content %}} +{{< /flex >}} + +### exact_mean +An aggregate method that takes the average of the two points closest to the quantile value. +Output tables consist of a single row containing the calculated quantile. + +If calculating the `0.5` quantile or 50th percentile: + +{{< flex >}} +{{% flex-content %}} +**Given the following input table:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.0 | +| 2020-01-01T00:03:00Z | 2.0 | +| 2020-01-01T00:04:00Z | 3.0 | +{{% /flex-content %}} +{{% flex-content %}} +**`exact_mean` returns:** + +| _value | +|:------:| +| 1.5 | +{{% /flex-content %}} +{{< /flex >}} + +### exact_selector +A selector method that returns the data point for which at least `q` points are less than. +Output tables consist of a single row containing the calculated quantile. + +If calculating the `0.5` quantile or 50th percentile: + +{{< flex >}} +{{% flex-content %}} +**Given the following input table:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.0 | +| 2020-01-01T00:03:00Z | 2.0 | +| 2020-01-01T00:04:00Z | 3.0 | +{{% /flex-content %}} +{{% flex-content %}} +**`exact_selector` returns:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:02:00Z | 1.0 | +{{% /flex-content %}} +{{< /flex >}} + +{{% note %}} +The examples below use the [example data variable](/influxdb/v1.7/flux/guides/#example-data-variable). +{{% /note %}} + +## Find the value representing the 99th percentile +Use the default method, `"estimate_tdigest"`, to return all rows in a table that +contain values in the 99th percentile of data in the table. + +```js +data + |> quantile(q: 0.99) +``` + +## Find the average of values closest to the quantile +Use the `exact_mean` method to return a single row per input table containing the +average of the two values closest to the mathematical quantile of data in the table. +For example, to calculate the `0.99` quantile: + +```js +data + |> quantile(q: 0.99, method: "exact_mean") +``` + +## Find the point with the quantile value +Use the `exact_selector` method to return a single row per input table containing the +value that `q * 100`% of values in the table are less than. +For example, to calculate the `0.99` quantile: + +```js +data + |> quantile(q: 0.99, method: "exact_selector") +``` + +## Use quantile() with aggregateWindow() +[`aggregateWindow()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/aggregatewindow/) +segments data into windows of time, aggregates data in each window into a single +point, and then removes the time-based segmentation. +It is primarily used to downsample data. + +To specify the [quantile calculation method](#select-a-method-for-calculating-the-quantile) in +`aggregateWindow()`, use the [full function syntax](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/aggregatewindow/#specify-parameters-of-the-aggregate-function): + +```js +data + |> aggregateWindow( + every: 5m, + fn: (tables=<-, column) => + tables + |> quantile(q: 0.99, method: "exact_selector") + ) +``` diff --git a/content/influxdb/v1.7/flux/guides/query-fields.md b/content/influxdb/v1.7/flux/guides/query-fields.md new file mode 100644 index 000000000..a77403a1b --- /dev/null +++ b/content/influxdb/v1.7/flux/guides/query-fields.md @@ -0,0 +1,76 @@ +--- +title: Query fields and tags +seotitle: Query fields and tags in InfluxDB using Flux +description: > + Use the `filter()` function to query data based on fields, tags, or any other column value. + `filter()` performs operations similar to the `SELECT` statement and the `WHERE` + clause in InfluxQL and other SQL-like query languages. +weight: 1 +menu: + influxdb_1_7: + parent: Query with Flux +list_code_example: | + ```js + from(bucket: "db/rp") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "example-measurement" and + r._field == "example-field" and + r.tag == "example-tag" + ) + ``` +--- + +Use the [`filter()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/filter/) +to query data based on fields, tags, or any other column value. +`filter()` performs operations similar to the `SELECT` statement and the `WHERE` +clause in InfluxQL and other SQL-like query languages. + +## The filter() function +`filter()` has an `fn` parameter that expects a **predicate function**, +an anonymous function comprised of one or more **predicate expressions**. +The predicate function evaluates each input row. +Rows that evaluate to `true` are **included** in the output data. +Rows that evaluate to `false` are **excluded** from the output data. + +```js +// ... + |> filter(fn: (r) => r._measurement == "example-measurement" ) +``` + +The `fn` predicate function requires an `r` argument, which represents each row +as `filter()` iterates over input data. +Key-value pairs in the row record represent columns and their values. +Use **dot notation** or **bracket notation** to reference specific column values in the predicate function. +Use [logical operators](/{{< latest "influxdb" "v2" >}}/reference/flux/language/operators/#logical-operators) +to chain multiple predicate expressions together. + +```js +// Row record +r = {foo: "bar", baz: "quz"} + +// Example predicate function +(r) => r.foo == "bar" and r["baz"] == "quz" + +// Evaluation results +(r) => true and true +``` + +## Filter by fields and tags +The combination of [`from()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/inputs/from), +[`range()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/range), +and `filter()` represent the most basic Flux query: + +1. Use `from()` to define your [bucket](/influxdb/v1.7/flux/get-started/#buckets). +2. Use `range()` to limit query results by time. +3. Use `filter()` to identify what rows of data to output. + +```js +from(bucket: "db/rp") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "example-measurement" and + r._field == "example-field" and + r.tag == "example-tag" + ) +``` diff --git a/content/influxdb/v1.7/flux/guides/rate.md b/content/influxdb/v1.7/flux/guides/rate.md new file mode 100644 index 000000000..27dabfe2e --- /dev/null +++ b/content/influxdb/v1.7/flux/guides/rate.md @@ -0,0 +1,109 @@ +--- +title: Calculate the rate of change +seotitle: Calculate the rate of change in Flux +list_title: Rate +description: > + Use the `derivative()` function to calculate the rate of change between subsequent values or the + `aggregate.rate()` function to calculate the average rate of change per window of time. + If time between points varies, these functions normalize points to a common time interval + making values easily comparable. +weight: 10 +menu: + influxdb_1_7: + parent: Query with Flux + name: Rate +list_query_example: rate_of_change +--- + + +Use the [`derivative()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/derivative/) +to calculate the rate of change between subsequent values or the +[`aggregate.rate()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/aggregate/rate/) +to calculate the average rate of change per window of time. +If time between points varies, these functions normalize points to a common time interval +making values easily comparable. + +## Rate of change between subsequent values +Use the [`derivative()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/derivative/) +to calculate the rate of change per unit of time between subsequent _non-null_ values. + +```js +data + |> derivative(unit: 1s) +``` + +By default, `derivative()` returns only positive derivative values and replaces negative values with _null_. +Cacluated values are returned as [floats](/{{< latest "influxdb" "v2" >}}/reference/flux/language/types/#numeric-types). + + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:00:00Z | 250 | +| 2020-01-01T00:04:00Z | 160 | +| 2020-01-01T00:12:00Z | 150 | +| 2020-01-01T00:19:00Z | 220 | +| 2020-01-01T00:32:00Z | 200 | +| 2020-01-01T00:51:00Z | 290 | +| 2020-01-01T01:00:00Z | 340 | +{{% /flex-content %}} +{{% flex-content %}} +**`derivative(unit: 1m)` returns:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:04:00Z | | +| 2020-01-01T00:12:00Z | | +| 2020-01-01T00:19:00Z | 10.0 | +| 2020-01-01T00:32:00Z | | +| 2020-01-01T00:51:00Z | 4.74 | +| 2020-01-01T01:00:00Z | 5.56 | +{{% /flex-content %}} +{{< /flex >}} + +Results represent the rate of change **per minute** between subsequent values with +negative values set to _null_. + +### Return negative derivative values +To return negative derivative values, set the `nonNegative` parameter to `false`, + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:00:00Z | 250 | +| 2020-01-01T00:04:00Z | 160 | +| 2020-01-01T00:12:00Z | 150 | +| 2020-01-01T00:19:00Z | 220 | +| 2020-01-01T00:32:00Z | 200 | +| 2020-01-01T00:51:00Z | 290 | +| 2020-01-01T01:00:00Z | 340 | +{{% /flex-content %}} +{{% flex-content %}} +**The following returns:** + +```js +|> derivative( + unit: 1m, + nonNegative: false +) +``` + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:04:00Z | -22.5 | +| 2020-01-01T00:12:00Z | -1.25 | +| 2020-01-01T00:19:00Z | 10.0 | +| 2020-01-01T00:32:00Z | -1.54 | +| 2020-01-01T00:51:00Z | 4.74 | +| 2020-01-01T01:00:00Z | 5.56 | +{{% /flex-content %}} +{{< /flex >}} + +Results represent the rate of change **per minute** between subsequent values and +include negative values. diff --git a/content/influxdb/v1.7/flux/guides/regular-expressions.md b/content/influxdb/v1.7/flux/guides/regular-expressions.md new file mode 100644 index 000000000..c1444099d --- /dev/null +++ b/content/influxdb/v1.7/flux/guides/regular-expressions.md @@ -0,0 +1,91 @@ +--- +title: Use regular expressions in Flux +list_title: Regular expressions +description: This guide walks through using regular expressions in evaluation logic in Flux functions. +menu: + influxdb_1_7: + name: Regular expressions + parent: Query with Flux +weight: 20 +list_query_example: regular_expressions +--- + +Regular expressions (regexes) are incredibly powerful when matching patterns in large collections of data. +With Flux, regular expressions are primarily used for evaluation logic in predicate functions for things +such as filtering rows, dropping and keeping columns, state detection, etc. +This guide shows how to use regular expressions in your Flux scripts. + +If you're just getting started with Flux queries, check out the following: + +- [Get started with Flux](/influxdb/v1.7/flux/get-started/) for a conceptual overview of Flux and parts of a Flux query. +- [Execute queries](/influxdb/v1.7/flux/guides/execute-queries/) to discover a variety of ways to run your queries. + +## Go regular expression syntax +Flux uses Go's [regexp package](https://golang.org/pkg/regexp/) for regular expression search. +The links [below](#helpful-links) provide information about Go's regular expression syntax. + +## Regular expression operators +Flux provides two comparison operators for use with regular expressions. + +#### `=~` +When the expression on the left **MATCHES** the regular expression on the right, this evaluates to `true`. + +#### `!~` +When the expression on the left **DOES NOT MATCH** the regular expression on the right, this evaluates to `true`. + +## Regular expressions in Flux +When using regex matching in your Flux scripts, enclose your regular expressions with `/`. +The following is the basic regex comparison syntax: + +###### Basic regex comparison syntax +```js +expression =~ /regex/ +expression !~ /regex/ +``` +## Examples + +### Use a regex to filter by tag value +The following example filters records by the `cpu` tag. +It only keeps records for which the `cpu` is either `cpu0`, `cpu1`, or `cpu2`. + +```js +from(bucket: "db/rp") + |> range(start: -15m) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_user" and + r.cpu =~ /cpu[0-2]/ + ) +``` + +### Use a regex to filter by field key +The following example excludes records that do not have `_percent` in a field key. + +```js +from(bucket: "db/rp") + |> range(start: -15m) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field =~ /_percent/ + ) +``` + +### Drop columns matching a regex +The following example drops columns whose names do not being with `_`. + +```js +from(bucket: "db/rp") + |> range(start: -15m) + |> filter(fn: (r) => r._measurement == "mem") + |> drop(fn: (column) => column !~ /_.*/) +``` + +## Helpful links + +##### Syntax documentation +[regexp Syntax GoDoc](https://godoc.org/regexp/syntax) +[RE2 Syntax Overview](https://github.com/google/re2/wiki/Syntax) + +##### Go regex testers +[Regex Tester - Golang](https://regex-golang.appspot.com/assets/html/index.html) +[Regex101](https://regex101.com/) diff --git a/content/influxdb/v1.7/flux/guides/scalar-values.md b/content/influxdb/v1.7/flux/guides/scalar-values.md new file mode 100644 index 000000000..010ce2f51 --- /dev/null +++ b/content/influxdb/v1.7/flux/guides/scalar-values.md @@ -0,0 +1,260 @@ +--- +title: Extract scalar values in Flux +list_title: Extract scalar values +description: > + Use Flux stream and table functions to extract scalar values from Flux query output. + This lets you, for example, dynamically set variables using query results. +menu: + influxdb_1_7: + name: Extract scalar values + parent: Query with Flux +weight: 20 +list_code_example: | + ```js + scalarValue = { + _record = + data + |> tableFind(fn: key => true) + |> getRecord(idx: 0) + return _record._value + } + ``` +--- + +Use Flux [stream and table functions](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/stream-table/) +to extract scalar values from Flux query output. +This lets you, for example, dynamically set variables using query results. + +**To extract scalar values from output:** + +1. [Extract a table](#extract-a-table). +2. [Extract a column from the table](#extract-a-column-from-the-table) + _**or**_ [extract a row from the table](#extract-a-row-from-the-table). + +_The samples on this page use the [sample data provided below](#sample-data)._ + +{{% warn %}} +#### Current limitations +- The InfluxDB user interface (UI) does not currently support raw scalar output. + Use [`map()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/map/) to add + scalar values to output data. +- The [Flux REPL](/influxdb/v1.7/flux/guides/execute-queries/#influx-cli) does not currently support + Flux stream and table functions (also known as "dynamic queries"). + See [#15321](https://github.com/influxdata/influxdb/issues/15231). +{{% /warn %}} + +## Extract a table +Flux formats query results as a stream of tables. +To extract a scalar value from a stream of tables, you must first extract a single table. + +to extract a single table from the stream of tables. + +{{% note %}} +If query results include only one table, it is still formatted as a stream of tables. +You still must extract that table from the stream. +{{% /note %}} + +Use [`tableFind()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/stream-table/tablefind/) +to extract the **first** table whose [group key](/influxdb/v1.7/flux/get-started/#group-keys) +values match the `fn` **predicate function**. +The predicate function requires a `key` record, which represents the group key of +each table. + +```js +sampleData + |> tableFind(fn: (key) => + key._field == "temp" and + key.location == "sfo" + ) +``` + +The example above returns a single table: + +| _time | location | _field | _value | +|:----- |:--------:|:------:| ------:| +| 2019-11-01T12:00:00Z | sfo | temp | 65.1 | +| 2019-11-01T13:00:00Z | sfo | temp | 66.2 | +| 2019-11-01T14:00:00Z | sfo | temp | 66.3 | +| 2019-11-01T15:00:00Z | sfo | temp | 66.8 | + +{{% note %}} +#### Extract the correct table +Flux functions do not guarantee table order and `tableFind()` returns only the +**first** table that matches the `fn` predicate. +To extract the table that includes the data you actually want, be very specific in +your predicate function or filter and transform your data to minimize the number +of tables piped-forward into `tableFind()`. +{{% /note %}} + +## Extract a column from the table +Use the [`getColumn()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/stream-table/getcolumn/) +to output an array of values from a specific column in the extracted table. + + +```js +sampleData + |> tableFind(fn: (key) => + key._field == "temp" and + key.location == "sfo" + ) + |> getColumn(column: "_value") + +// Returns [65.1, 66.2, 66.3, 66.8] +``` + +### Use extracted column values +Use a variable to store the array of values. +In the example below, `SFOTemps` represents the array of values. +Reference a specific index (integer starting from `0`) in the array to return the +value at that index. + +```js +SFOTemps = sampleData + |> tableFind(fn: (key) => + key._field == "temp" and + key.location == "sfo" + ) + |> getColumn(column: "_value") + +SFOTemps +// Returns [65.1, 66.2, 66.3, 66.8] + +SFOTemps[0] +// Returns 65.1 + +SFOTemps[2] +// Returns 66.3 +``` + +## Extract a row from the table +Use the [`getRecord()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/stream-table/getrecord/) +to output data from a single row in the extracted table. +Specify the index of the row to output using the `idx` parameter. +The function outputs a record with key-value pairs for each column. + +```js +sampleData + |> tableFind(fn: (key) => + key._field == "temp" and + key.location == "sfo" + ) + |> getRecord(idx: 0) + +// Returns { +// _time:2019-11-11T12:00:00Z, +// _field:"temp", +// location:"sfo", +// _value: 65.1 +// } +``` + +### Use an extracted row record +Use a variable to store the extracted row record. +In the example below, `tempInfo` represents the extracted row. +Use [dot notation](/influxdb/v1.7/flux/get-started/syntax-basics/#records) to reference +keys in the object. + +```js +tempInfo = sampleData + |> tableFind(fn: (key) => + key._field == "temp" and + key.location == "sfo" + ) + |> getRecord(idx: 0) + +tempInfo +// Returns { +// _time:2019-11-11T12:00:00Z, +// _field:"temp", +// location:"sfo", +// _value: 65.1 +// } + +tempInfo._time +// Returns 2019-11-11T12:00:00Z + +tempInfo.location +// Returns sfo +``` + +## Example helper functions +Create custom helper functions to extract scalar values from query output. + +##### Extract a scalar field value +```js +// Define a helper function to extract field values +getFieldValue = (tables=<-, field) => { + extract = tables + |> tableFind(fn: (key) => key._field == field) + |> getColumn(column: "_value") + return extract[0] +} + +// Use the helper function to define a variable +lastJFKTemp = sampleData + |> filter(fn: (r) => r.location == "kjfk") + |> last() + |> getFieldValue(field: "temp") + +lastJFKTemp +// Returns 71.2 +``` + +##### Extract scalar row data +```js +// Define a helper function to extract a row as a record +getRow = (tables=<-, field, idx=0) => { + extract = tables + |> tableFind(fn: (key) => true) + |> getRecord(idx: idx) + return extract +} + +// Use the helper function to define a variable +lastReported = sampleData + |> last() + |> getRow(idx: 0) + +"The last location to report was ${lastReported.location}. +The temperature was ${string(v: lastReported._value)}°F." + +// Returns: +// The last location to report was kord. +// The temperature was 38.9°F. +``` + +--- + +## Sample data + +The following sample data set represents fictional temperature metrics collected +from three locations. +It's formatted in [annotated CSV](https://v2.docs.influxdata.com/v2.0/reference/syntax/annotated-csv/) and imported +into the Flux query using the [`csv.from()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/csv/from/). + +Place the following at the beginning of your query to use the sample data: + +{{% truncate %}} +```js +import "csv" + +sampleData = csv.from(csv: " +#datatype,string,long,dateTime:RFC3339,string,string,double +#group,false,true,false,true,true,false +#default,,,,,, +,result,table,_time,location,_field,_value +,,0,2019-11-01T12:00:00Z,sfo,temp,65.1 +,,0,2019-11-01T13:00:00Z,sfo,temp,66.2 +,,0,2019-11-01T14:00:00Z,sfo,temp,66.3 +,,0,2019-11-01T15:00:00Z,sfo,temp,66.8 +,,1,2019-11-01T12:00:00Z,kjfk,temp,69.4 +,,1,2019-11-01T13:00:00Z,kjfk,temp,69.9 +,,1,2019-11-01T14:00:00Z,kjfk,temp,71.0 +,,1,2019-11-01T15:00:00Z,kjfk,temp,71.2 +,,2,2019-11-01T12:00:00Z,kord,temp,46.4 +,,2,2019-11-01T13:00:00Z,kord,temp,46.3 +,,2,2019-11-01T14:00:00Z,kord,temp,42.7 +,,2,2019-11-01T15:00:00Z,kord,temp,38.9 +") +``` +{{% /truncate %}} diff --git a/content/influxdb/v1.7/flux/guides/sort-limit.md b/content/influxdb/v1.7/flux/guides/sort-limit.md new file mode 100644 index 000000000..8a35dfff8 --- /dev/null +++ b/content/influxdb/v1.7/flux/guides/sort-limit.md @@ -0,0 +1,68 @@ +--- +title: Sort and limit data with Flux +seotitle: Sort and limit data in InfluxDB with Flux +list_title: Sort and limit +description: > + Use the `sort()`function to order records within each table by specific columns and the + `limit()` function to limit the number of records in output tables to a fixed number, `n`. +menu: + influxdb_1_7: + name: Sort and limit + parent: Query with Flux +weight: 3 +list_query_example: sort_limit +--- + +Use the [`sort()`function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/sort) +to order records within each table by specific columns and the +[`limit()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/limit) +to limit the number of records in output tables to a fixed number, `n`. + +If you're just getting started with Flux queries, check out the following: + +- [Get started with Flux](/influxdb/v1.7/flux/get-started/) for a conceptual overview of Flux and parts of a Flux query. +- [Execute queries](/influxdb/v1.7/flux/guides/execute-queries/) to discover a variety of ways to run your queries. + +##### Example sorting system uptime + +The following example orders system uptime first by region, then host, then value. + +```js +from(bucket:"db/rp") + |> range(start:-12h) + |> filter(fn: (r) => + r._measurement == "system" and + r._field == "uptime" + ) + |> sort(columns:["region", "host", "_value"]) +``` + +The [`limit()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/limit) +limits the number of records in output tables to a fixed number, `n`. +The following example shows up to 10 records from the past hour. + +```js +from(bucket:"db/rp") + |> range(start:-1h) + |> limit(n:10) +``` + +You can use `sort()` and `limit()` together to show the top N records. +The example below returns the 10 top system uptime values sorted first by +region, then host, then value. + +```js +from(bucket:"db/rp") + |> range(start:-12h) + |> filter(fn: (r) => + r._measurement == "system" and + r._field == "uptime" + ) + |> sort(columns:["region", "host", "_value"]) + |> limit(n:10) +``` + +You now have created a Flux query that sorts and limits data. +Flux also provides the [`top()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/top) +and [`bottom()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/bottom) +functions to perform both of these functions at the same time. diff --git a/content/influxdb/v1.7/flux/guides/sql.md b/content/influxdb/v1.7/flux/guides/sql.md new file mode 100644 index 000000000..f72d4a45c --- /dev/null +++ b/content/influxdb/v1.7/flux/guides/sql.md @@ -0,0 +1,196 @@ +--- +title: Query SQL data sources +seotitle: Query SQL data sources with InfluxDB +list_title: Query SQL data +description: > + The Flux `sql` package provides functions for working with SQL data sources. + Use `sql.from()` to query SQL databases like PostgreSQL and MySQL +menu: + influxdb_1_7: + parent: Query with Flux + list_title: SQL data +weight: 20 +list_code_example: | + ```js + import "sql" + + sql.from( + driverName: "postgres", + dataSourceName: "postgresql://user:password@localhost", + query: "SELECT * FROM example_table" + ) + ``` +--- + +The Flux `sql` package provides functions for working with SQL data sources. +[`sql.from()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/sql/from/) lets you query SQL data sources +like [PostgreSQL](https://www.postgresql.org/) and [MySQL](https://www.mysql.com/), +and use the results with InfluxDB dashboards, tasks, and other operations. + +- [Query a SQL data source](#query-a-sql-data-source) +- [Join SQL data with data in InfluxDB](#join-sql-data-with-data-in-influxdb) +- [Sample sensor data](#sample-sensor-data) + +## Query a SQL data source +To query a SQL data source: + +1. Import the `sql` package in your Flux query +2. Use the `sql.from()` function to specify the driver, data source name (DSN), + and query used to query data from your SQL data source: + +{{< code-tabs-wrapper >}} +{{% code-tabs %}} +[PostgreSQL](#) +[MySQL](#) +{{% /code-tabs %}} + +{{% code-tab-content %}} +```js +import "sql" + +sql.from( + driverName: "postgres", + dataSourceName: "postgresql://user:password@localhost", + query: "SELECT * FROM example_table" +) +``` +{{% /code-tab-content %}} + +{{% code-tab-content %}} +```js +import "sql" + +sql.from( + driverName: "mysql", + dataSourceName: "user:password@tcp(localhost:3306)/db", + query: "SELECT * FROM example_table" +) +``` +{{% /code-tab-content %}} +{{< /code-tabs-wrapper >}} + +_See the [`sql.from()` documentation](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/sql/from/) for +information about required function parameters._ + +## Join SQL data with data in InfluxDB +One of the primary benefits of querying SQL data sources from InfluxDB +is the ability to enrich query results with data stored outside of InfluxDB. + +Using the [air sensor sample data](#sample-sensor-data) below, the following query +joins air sensor metrics stored in InfluxDB with sensor information stored in PostgreSQL. +The joined data lets you query and filter results based on sensor information +that isn't stored in InfluxDB. + +```js +// Import the "sql" package +import "sql" + +// Query data from PostgreSQL +sensorInfo = sql.from( + driverName: "postgres", + dataSourceName: "postgresql://localhost?sslmode=disable", + query: "SELECT * FROM sensors" +) + +// Query data from InfluxDB +sensorMetrics = from(bucket: "telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "airSensors") + +// Join InfluxDB query results with PostgreSQL query results +join(tables: {metric: sensorMetrics, info: sensorInfo}, on: ["sensor_id"]) +``` + +--- + +## Sample sensor data +The [sample data generator](#download-and-run-the-sample-data-generator) and +[sample sensor information](#import-the-sample-sensor-information) simulate a +group of sensors that measure temperature, humidity, and carbon monoxide +in rooms throughout a building. +Each collected data point is stored in InfluxDB with a `sensor_id` tag that identifies +the specific sensor it came from. +Sample sensor information is stored in PostgreSQL. + +**Sample data includes:** + +- Simulated data collected from each sensor and stored in the `airSensors` measurement in **InfluxDB**: + - temperature + - humidity + - co + +- Information about each sensor stored in the `sensors` table in **PostgreSQL**: + - sensor_id + - location + - model_number + - last_inspected + +### Import and generate sample sensor data + +#### Download and run the sample data generator +`air-sensor-data.rb` is a script that generates air sensor data and stores the data in InfluxDB. +To use `air-sensor-data.rb`: + +1. [Create a database](/influxdb/v1.7/introduction/getting-started/#creating-a-database) to store the data. +2. Download the sample data generator. _This tool requires [Ruby](https://www.ruby-lang.org/en/)._ + + Download Air Sensor Generator + +3. Give `air-sensor-data.rb` executable permissions: + + ``` + chmod +x air-sensor-data.rb + ``` + +4. Start the generator. Specify your database. + + ``` + ./air-sensor-data.rb -d database-name + ``` + + The generator begins to write data to InfluxDB and will continue until stopped. + Use `ctrl-c` to stop the generator. + + _**Note:** Use the `--help` flag to view other configuration options._ + + +5. Query your target database to ensure the generated data is writing successfully. + The generator doesn't catch errors from write requests, so it will continue running + even if data is not writing to InfluxDB successfully. + + ``` + from(bucket: "database-name/autogen") + |> range(start: -1m) + |> filter(fn: (r) => r._measurement == "airSensors") + ``` + +#### Import the sample sensor information +1. [Download and install PostgreSQL](https://www.postgresql.org/download/). +2. Download the sample sensor information CSV. + + Download Sample Data + +3. Use a PostgreSQL client (`psql` or a GUI) to create the `sensors` table: + + ``` + CREATE TABLE sensors ( + sensor_id character varying(50), + location character varying(50), + model_number character varying(50), + last_inspected date + ); + ``` + +4. Import the downloaded CSV sample data. + _Update the `FROM` file path to the path of the downloaded CSV sample data._ + + ``` + COPY sensors(sensor_id,location,model_number,last_inspected) + FROM '/path/to/sample-sensor-info.csv' DELIMITER ',' CSV HEADER; + ``` + +5. Query the table to ensure the data was imported correctly: + + ``` + SELECT * FROM sensors; + ``` diff --git a/content/influxdb/v1.7/flux/guides/window-aggregate.md b/content/influxdb/v1.7/flux/guides/window-aggregate.md new file mode 100644 index 000000000..75f77d063 --- /dev/null +++ b/content/influxdb/v1.7/flux/guides/window-aggregate.md @@ -0,0 +1,352 @@ +--- +title: Window and aggregate data with Flux +seotitle: Window and aggregate data in InfluxDB with Flux +list_title: Window & aggregate +description: > + This guide walks through windowing and aggregating data with Flux and outlines + how it shapes your data in the process. +menu: + influxdb_1_7: + name: Window & aggregate + parent: Query with Flux +weight: 4 +list_query_example: aggregate_window +--- + +A common operation performed with time series data is grouping data into windows of time, +or "windowing" data, then aggregating windowed values into a new value. +This guide walks through windowing and aggregating data with Flux and demonstrates +how data is shaped in the process. + +If you're just getting started with Flux queries, check out the following: + +- [Get started with Flux](/influxdb/v1.7/flux/get-started/) for a conceptual overview of Flux and parts of a Flux query. +- [Execute queries](/influxdb/v1.7/flux/guides/execute-queries/) to discover a variety of ways to run your queries. + +{{% note %}} +The following example is an in-depth walk-through of the steps required to window and aggregate data. +The [`aggregateWindow()` function](#summing-up) performs these operations for you, but understanding +how data is shaped in the process helps to successfully create your desired output. +{{% /note %}} + +## Data set +For the purposes of this guide, define a variable that represents your base data set. +The following example queries the memory usage of the host machine. + +```js +dataSet = from(bucket: "db/rp") + |> range(start: -5m) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field == "used_percent" + ) + |> drop(columns: ["host"]) +``` + +{{% note %}} +This example drops the `host` column from the returned data since the memory data +is only tracked for a single host and it simplifies the output tables. +Dropping the `host` column is optional and not recommended if monitoring memory +on multiple hosts. +{{% /note %}} + +`dataSet` can now be used to represent your base data, which will look similar to the following: + +{{% truncate %}} +``` +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:00.000000000Z 71.11611366271973 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:10.000000000Z 67.39630699157715 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:20.000000000Z 64.16666507720947 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:30.000000000Z 64.19951915740967 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:40.000000000Z 64.2122745513916 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:50.000000000Z 64.22209739685059 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:00.000000000Z 64.6336555480957 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:10.000000000Z 64.16516304016113 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:20.000000000Z 64.18349742889404 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:30.000000000Z 64.20474052429199 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:40.000000000Z 68.65062713623047 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:50.000000000Z 67.20139980316162 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:00.000000000Z 70.9143877029419 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:10.000000000Z 64.14549350738525 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:20.000000000Z 64.15379047393799 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:30.000000000Z 64.1592264175415 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:40.000000000Z 64.18190002441406 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:50.000000000Z 64.28837776184082 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:00.000000000Z 64.29731845855713 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:10.000000000Z 64.36963081359863 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:20.000000000Z 64.37397003173828 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:30.000000000Z 64.44413661956787 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:40.000000000Z 64.42906856536865 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:50.000000000Z 64.44573402404785 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:00.000000000Z 64.48912620544434 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:10.000000000Z 64.49522972106934 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:20.000000000Z 64.48652744293213 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:30.000000000Z 64.49949741363525 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:40.000000000Z 64.4949197769165 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:50.000000000Z 64.49787616729736 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49816226959229 +``` +{{% /truncate %}} + +## Windowing data +Use the [`window()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/window) +to group your data based on time bounds. +The most common parameter passed with the `window()` is `every` which +defines the duration of time between windows. +Other parameters are available, but for this example, window the base data +set into one minute windows. + +```js +dataSet + |> window(every: 1m) +``` + +{{% note %}} +The `every` parameter supports all [valid duration units](/{{< latest "influxdb" "v2" >}}/reference/flux/language/types/#duration-types), +including **calendar months (`1mo`)** and **years (`1y`)**. +{{% /note %}} + +Each window of time is output in its own table containing all records that fall within the window. + +{{% truncate %}} +###### window() output tables +``` +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:00.000000000Z 71.11611366271973 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:10.000000000Z 67.39630699157715 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:20.000000000Z 64.16666507720947 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:30.000000000Z 64.19951915740967 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:40.000000000Z 64.2122745513916 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:50.000000000Z 64.22209739685059 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:00.000000000Z 64.6336555480957 +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:10.000000000Z 64.16516304016113 +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:20.000000000Z 64.18349742889404 +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:30.000000000Z 64.20474052429199 +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:40.000000000Z 68.65062713623047 +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:50.000000000Z 67.20139980316162 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:00.000000000Z 70.9143877029419 +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:10.000000000Z 64.14549350738525 +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:20.000000000Z 64.15379047393799 +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:30.000000000Z 64.1592264175415 +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:40.000000000Z 64.18190002441406 +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:50.000000000Z 64.28837776184082 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:00.000000000Z 64.29731845855713 +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:10.000000000Z 64.36963081359863 +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:20.000000000Z 64.37397003173828 +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:30.000000000Z 64.44413661956787 +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:40.000000000Z 64.42906856536865 +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:50.000000000Z 64.44573402404785 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:00.000000000Z 64.48912620544434 +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:10.000000000Z 64.49522972106934 +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:20.000000000Z 64.48652744293213 +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:30.000000000Z 64.49949741363525 +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:40.000000000Z 64.4949197769165 +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:50.000000000Z 64.49787616729736 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:55:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49816226959229 +``` +{{% /truncate %}} + +When visualized in the InfluxDB UI, each window table is displayed in a different color. + + + +## Aggregate data +[Aggregate functions](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates) take the values +of all rows in a table and use them to perform an aggregate operation. +The result is output as a new value in a single-row table. + +Since windowed data is split into separate tables, aggregate operations run against +each table separately and output new tables containing only the aggregated value. + +For this example, use the [`mean()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/mean) +to output the average of each window: + +```js +dataSet + |> window(every: 1m) + |> mean() +``` + +{{% truncate %}} +###### mean() output tables +``` +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 65.88549613952637 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 65.50651391347249 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 65.30719598134358 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 64.39330975214641 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 64.49386278788249 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:55:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 64.49816226959229 +``` +{{% /truncate %}} + +Because each data point is contained in its own table, when visualized, +they appear as single, unconnected points. + + + +### Recreate the time column +**Notice the `_time` column is not in the [aggregated output tables](#mean-output-tables).** +Because records in each table are aggregated together, their timestamps no longer +apply and the column is removed from the group key and table. + +Also notice the `_start` and `_stop` columns still exist. +These represent the lower and upper bounds of the time window. + +Many Flux functions rely on the `_time` column. +To further process your data after an aggregate function, you need to re-add `_time`. +Use the [`duplicate()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/duplicate) to +duplicate either the `_start` or `_stop` column as a new `_time` column. + +```js +dataSet + |> window(every: 1m) + |> mean() + |> duplicate(column: "_stop", as: "_time") +``` + +{{% truncate %}} +###### duplicate() output tables +``` +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:51:00.000000000Z 65.88549613952637 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:52:00.000000000Z 65.50651391347249 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:53:00.000000000Z 65.30719598134358 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:54:00.000000000Z 64.39330975214641 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49386278788249 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:55:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49816226959229 +``` +{{% /truncate %}} + +## "Unwindow" aggregate tables +Keeping aggregate values in separate tables generally isn't the format in which you want your data. +Use the `window()` function to "unwindow" your data into a single infinite (`inf`) window. + +```js +dataSet + |> window(every: 1m) + |> mean() + |> duplicate(column: "_stop", as: "_time") + |> window(every: inf) +``` + +{{% note %}} +Windowing requires a `_time` column which is why it's necessary to +[recreate the `_time` column](#recreate-the-time-column) after an aggregation. +{{% /note %}} + +###### Unwindowed output table +``` +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:00.000000000Z 65.88549613952637 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:00.000000000Z 65.50651391347249 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:00.000000000Z 65.30719598134358 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:00.000000000Z 64.39330975214641 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49386278788249 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49816226959229 +``` + +With the aggregate values in a single table, data points in the visualization are connected. + + + +## Summing up +You have now created a Flux query that windows and aggregates data. +The data transformation process outlined in this guide should be used for all aggregation operations. + +Flux also provides the [`aggregateWindow()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/aggregatewindow) +which performs all these separate functions for you. + +The following Flux query will return the same results: + +###### aggregateWindow function +```js +dataSet + |> aggregateWindow(every: 1m, fn: mean) +``` diff --git a/content/influxdb/v1.7/flux/installation.md b/content/influxdb/v1.7/flux/installation.md new file mode 100644 index 000000000..b7fab4dc2 --- /dev/null +++ b/content/influxdb/v1.7/flux/installation.md @@ -0,0 +1,33 @@ +--- +title: Enable Flux +description: Instructions for enabling Flux in your InfluxDB configuration. +menu: + influxdb_1_7: + name: Enable Flux + parent: Flux + weight: 1 +--- + +Flux is packaged with **InfluxDB v1.7+** and does not require any additional installation, +however it is **disabled by default and needs to be enabled**. + +## Enable Flux +Enable Flux by setting the `flux-enabled` option to `true` under the `[http]` section of your `influxdb.conf`: + +###### influxdb.conf +```toml +# ... + +[http] + + # ... + + flux-enabled = true + + # ... +``` + +> The default location of your `influxdb.conf` depends on your operating system. +> More information is available in the [Configuring InfluxDB](/influxdb/v1.7/administration/config/#using-the-configuration-file) guide. + +When InfluxDB starts, the Flux daemon starts as well and data can be queried using Flux. diff --git a/content/influxdb/v1.7/query_language/functions.md b/content/influxdb/v1.7/query_language/functions.md index f3761d909..c10759ade 100644 --- a/content/influxdb/v1.7/query_language/functions.md +++ b/content/influxdb/v1.7/query_language/functions.md @@ -4374,7 +4374,7 @@ InfluxDB then rounds those averages down to the nearest integer. _InfluxQL does not currently support histogram generation. For information about creating histograms with data stored in InfluxDB, see -[Flux's `histogram()` function](/flux/v0.7/functions/transformations/histogram)._ +[Flux's `histogram()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/transformations/histogram)._ ### LN() diff --git a/content/influxdb/v1.7/troubleshooting/query_management.md b/content/influxdb/v1.7/troubleshooting/query_management.md index 6afb60e32..dd6d160fc 100644 --- a/content/influxdb/v1.7/troubleshooting/query_management.md +++ b/content/influxdb/v1.7/troubleshooting/query_management.md @@ -14,7 +14,7 @@ Manage your InfluxQL queries using the following: - [KILL QUERIES](#stop-currently-running-queries-with-kill-query) to stop queries overloading your system - [Configuration settings](#configuration-settings-for-query-management) to prevent and halt the execution of inefficient queries -> The commands and configurations provided on this page are for **Influx Query Language (InfluxQL) only** -- **no equivalent set of Flux commands and configurations currently exists**. For the most current Flux documentation, see [Get started with Flux](/flux/v0.50/introduction/getting-started/). +> The commands and configurations provided on this page are for **Influx Query Language (InfluxQL) only** -- **no equivalent set of Flux commands and configurations currently exists**. For the most current Flux documentation, see [Get started with Flux](/influxdb/v1.8/flux/get-started/). ## List currently-running queries with `SHOW QUERIES` diff --git a/content/influxdb/v1.8/about_the_project/releasenotes-changelog.md b/content/influxdb/v1.8/about_the_project/releasenotes-changelog.md index ecc874618..2ec5899ce 100644 --- a/content/influxdb/v1.8/about_the_project/releasenotes-changelog.md +++ b/content/influxdb/v1.8/about_the_project/releasenotes-changelog.md @@ -33,7 +33,7 @@ menu: #### Flux v0.65 ready for production use -This release updates support for the Flux language and queries. To learn about Flux design principles and see how to get started with Flux, see [Introduction to Flux](/flux/v0.65/introduction/). +This release updates support for the Flux language and queries. To learn about Flux design principles and see how to get started with Flux, see [Introduction to Flux](/influxdb/v1.8/flux/). * Use the new [`influx -type=flux`](/influxdb/v1.8/tools/shell/#type) option to enable the Flux REPL shell for creating Flux queries. @@ -276,7 +276,7 @@ Support for the Flux language and queries has been added in this release. To beg * Enable Flux using the new configuration setting [`[http] flux-enabled = true`](/influxdb/v1.7/administration/config/#flux-enabled-false). * Use the new [`influx -type=flux`](/influxdb/v1.7/tools/shell/#type) option to enable the Flux REPL shell for creating Flux queries. -* Read about Flux and the Flux language, enabling Flux, or jump into the getting started and other guides in the [Flux v0.7 (technical preview) documentation](/flux/v0.7/). +* Read about Flux and the Flux language, enabling Flux, or jump into the getting started and other guides. #### Time Series Index (TSI) query performance and throughputs improvements diff --git a/content/influxdb/v1.8/flux/_index.md b/content/influxdb/v1.8/flux/_index.md new file mode 100644 index 000000000..3dd78c6f5 --- /dev/null +++ b/content/influxdb/v1.8/flux/_index.md @@ -0,0 +1,37 @@ +--- +title: Flux data scripting language +description: > + Flux is a functional data scripting language designed for querying, analyzing, and acting on time series data. +menu: + influxdb_1_8: + name: Flux + weight: 80 +--- + +Flux is a functional data scripting language designed for querying, analyzing, and acting on time series data. +Its takes the power of [InfluxQL](/{{< latest "influxdb" "v1" >}}/query_language/spec/) and the functionality of [TICKscript](/{{< latest "kapacitor" >}}/tick/introduction/) and combines them into a single, unified syntax. + +> Flux v0.65 is production-ready and included with [InfluxDB v1.8](/influxdb/v1.8). +> The InfluxDB v1.8 implementation of Flux is read-only and does not support +> writing data back to InfluxDB. + +## Flux design principles +Flux is designed to be usable, readable, flexible, composable, testable, contributable, and shareable. +Its syntax is largely inspired by [2018's most popular scripting language](https://insights.stackoverflow.com/survey/2018#technology), +Javascript, and takes a functional approach to data exploration and processing. + +The following example illustrates pulling data from a bucket (similar to an InfluxQL database) for the last five minutes, +filtering that data by the `cpu` measurement and the `cpu=cpu-total` tag, windowing the data in 1 minute intervals, +and calculating the average of each window: + +```js +from(bucket:"telegraf/autogen") + |> range(start:-1h) + |> filter(fn:(r) => + r._measurement == "cpu" and + r.cpu == "cpu-total" + ) + |> aggregateWindow(every: 1m, fn: mean) +``` + +{{< children >}} \ No newline at end of file diff --git a/content/influxdb/v1.8/flux/flux-vs-influxql.md b/content/influxdb/v1.8/flux/flux-vs-influxql.md new file mode 100644 index 000000000..c98b89cd3 --- /dev/null +++ b/content/influxdb/v1.8/flux/flux-vs-influxql.md @@ -0,0 +1,419 @@ +--- +title: Flux vs InfluxQL +description: +menu: + influxdb_1_8: + name: Flux vs InfluxQL + parent: Flux + weight: 5 +--- + +Flux is an alternative to [InfluxQL](/influxdb/v1.8/query_language/) and other SQL-like query languages for querying and analyzing data. +Flux uses functional language patterns making it incredibly powerful, flexible, and able to overcome many of the limitations of InfluxQL. +This article outlines many of the tasks possible with Flux but not InfluxQL and provides information about Flux and InfluxQL parity. + +- [Possible with Flux](#possible-with-flux) +- [InfluxQL and Flux parity](#influxql-and-flux-parity) + +## Possible with Flux + +- [Joins](#joins) +- [Math across measurements](#math-across-measurements) +- [Sort by tags](#sort-by-tags) +- [Group by any column](#group-by-any-column) +- [Window by calendar months and years](#window-by-calendar-months-and-years) +- [Work with multiple data sources](#work-with-multiple-data-sources) +- [DatePart-like queries](#datepart-like-queries) +- [Pivot](#pivot) +- [Histograms](#histograms) +- [Covariance](#covariance) +- [Cast booleans to integers](#cast-booleans-to-integers) +- [String manipulation and data shaping](#string-manipulation-and-data-shaping) +- [Work with geo-temporal data](#work-with-geo-temporal-data) + +### Joins +InfluxQL has never supported joins. They can be accomplished using [TICKscript](/{{< latest "kapacitor" >}}/tick/introduction/), +but even TICKscript's join capabilities are limited. +Flux's [`join()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/join/) allows you +to join data **from any bucket, any measurement, and on any columns** as long as +each data set includes the columns on which they are to be joined. +This opens the door for really powerful and useful operations. + +```js +dataStream1 = from(bucket: "bucket1") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "network" and + r._field == "bytes-transferred" + ) + +dataStream2 = from(bucket: "bucket1") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "httpd" and + r._field == "requests-per-sec" + ) + +join( + tables: {d1:dataStream1, d2:dataStream2}, + on: ["_time", "_stop", "_start", "host"] + ) +``` + + +--- + +_For an in-depth walkthrough of using the `join()` function, see [How to join data with Flux](/influxdb/v1.8/flux/guides/join)._ + +--- + +### Math across measurements +Being able to perform cross-measurement joins also allows you to run calculations using +data from separate measurements – a highly requested feature from the InfluxData community. +The example below takes two data streams from separate measurements, `mem` and `processes`, +joins them, then calculates the average amount of memory used per running process: + +```js +// Memory used (in bytes) +memUsed = from(bucket: "telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field == "used" + ) + +// Total processes running +procTotal = from(bucket: "telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "processes" and + r._field == "total" + ) + +// Join memory used with total processes and calculate +// the average memory (in MB) used for running processes. +join( + tables: {mem:memUsed, proc:procTotal}, + on: ["_time", "_stop", "_start", "host"] + ) + |> map(fn: (r) => ({ + _time: r._time, + _value: (r._value_mem / r._value_proc) / 1000000 + }) +) +``` + +### Sort by tags +InfluxQL's sorting capabilities are very limited, allowing you only to control the +sort order of `time` using the `ORDER BY time` clause. +Flux's [`sort()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/sort) sorts records based on list of columns. +Depending on the column type, records are sorted lexicographically, numerically, or chronologically. + +```js +from(bucket:"telegraf/autogen") + |> range(start:-12h) + |> filter(fn: (r) => + r._measurement == "system" and + r._field == "uptime" + ) + |> sort(columns:["region", "host", "_value"]) +``` + +### Group by any column +InfluxQL lets you group by tags or by time intervals, but nothing else. +Flux lets you group by any column in the dataset, including `_value`. +Use the Flux [`group()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/group/) +to define which columns to group data by. + +```js +from(bucket:"telegraf/autogen") + |> range(start:-12h) + |> filter(fn: (r) => r._measurement == "system" and r._field == "uptime" ) + |> group(columns:["host", "_value"]) +``` + +### Window by calendar months and years +InfluxQL does not support windowing data by calendar months and years due to their varied lengths. +Flux supports calendar month and year duration units (`1mo`, `1y`) and lets you +window and aggregate data by calendar month and year. + +```js +from(bucket:"telegraf/autogen") + |> range(start:-1y) + |> filter(fn: (r) => r._measurement == "mem" and r._field == "used_percent" ) + |> aggregateWindow(every: 1mo, fn: mean) +``` + +### Work with multiple data sources +InfluxQL can only query data stored in InfluxDB. +Flux can query data from other data sources such as CSV, PostgreSQL, MySQL, Google BigTable, and more. +Join that data with data in InfluxDB to enrich query results. + +- [Flux CSV package](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/csv/) +- [Flux SQL package](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/sql/) +- [Flux BigTable package](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/bigtable/) + + +```js +import "csv" +import "sql" + +csvData = csv.from(csv: rawCSV) +sqlData = sql.from( + driverName: "postgres", + dataSourceName: "postgresql://user:password@localhost", + query:"SELECT * FROM example_table" +) +data = from(bucket: "telegraf/autogen") + |> range(start: -24h) + |> filter(fn: (r) => r._measurement == "sensor") + +auxData = join(tables: {csv: csvData, sql: sqlData}, on: ["sensor_id"]) +enrichedData = join(tables: {data: data, aux: auxData}, on: ["sensor_id"]) + +enrichedData + |> yeild(name: "enriched_data") +``` + +--- + +_For an in-depth walkthrough of querying SQL data, see [Query SQL data sources](/influxdb/v1.8/flux/guides/sql)._ + +--- + +### DatePart-like queries +InfluxQL doesn't support DatePart-like queries that only return results during specified hours of the day. +The Flux [`hourSelection` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/hourselection/) +returns only data with time values in a specified hour range. + +```js +from(bucket: "telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "cpu" and + r.cpu == "cpu-total" + ) + |> hourSelection(start: 9, stop: 17) +``` + +### Pivot +Pivoting data tables has never been supported in InfluxQL. +The Flux [`pivot()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/pivot) provides the ability +to pivot data tables by specifying `rowKey`, `columnKey`, and `valueColumn` parameters. + +```js +from(bucket: "telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "cpu" and + r.cpu == "cpu-total" + ) + |> pivot( + rowKey:["_time"], + columnKey: ["_field"], + valueColumn: "_value" + ) +``` + +### Histograms +The ability to generate histograms has been a highly requested feature for InfluxQL, but has never been supported. +Flux's [`histogram()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/histogram) uses input +data to generate a cumulative histogram with support for other histogram types coming in the future. + +```js +from(bucket: "telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field == "used_percent" + ) + |> histogram( + buckets: [10, 20, 30, 40, 50, 60, 70, 80, 90, 100] + ) +``` + +--- + +_For an example of using Flux to create a cumulative histogram, see [Create histograms](/influxdb/v1.8/flux/guides/histograms)._ + +--- + +### Covariance +Flux provides functions for simple covariance calculation. +The [`covariance()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/covariance) +calculates the covariance between two columns and the [`cov()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/cov) +calculates the covariance between two data streams. + +###### Covariance between two columns +```js +from(bucket: "telegraf/autogen") + |> range(start:-5m) + |> covariance(columns: ["x", "y"]) +``` + +###### Covariance between two streams of data +```js +table1 = from(bucket: "telegraf/autogen") + |> range(start: -15m) + |> filter(fn: (r) => + r._measurement == "measurement_1" + ) + +table2 = from(bucket: "telegraf/autogen") + |> range(start: -15m) + |> filter(fn: (r) => + r._measurement == "measurement_2" + ) + +cov(x: table1, y: table2, on: ["_time", "_field"]) +``` + +### Cast booleans to integers +InfluxQL supports type casting, but only for numeric data types (floats to integers and vice versa). +[Flux type conversion functions](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/type-conversions/) +provide much broader support for type conversions and let you perform some long-requested +operations like casting a boolean values to integers. + +##### Cast boolean field values to integers +```js +from(bucket: "telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "m" and + r._field == "bool_field" + ) + |> toInt() +``` + +### String manipulation and data shaping +InfluxQL doesn't support string manipulation when querying data. +The [Flux Strings package](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/strings/) is a collection of functions that operate on string data. +When combined with the [`map()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/map/), +functions in the string package allow for operations like string sanitization and normalization. + +```js +import "strings" + +from(bucket: "telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "weather" and + r._field == "temp" + ) + |> map(fn: (r) => ({ + r with + location: strings.toTitle(v: r.location), + sensor: strings.replaceAll(v: r.sensor, t: " ", u: "-"), + status: strings.substring(v: r.status, start: 0, end: 8) + })) +``` + +### Work with geo-temporal data +InfluxQL doesn't provide functionality for working with geo-temporal data. +The [Flux Geo package](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/geo/) is a collection of functions that +let you shape, filter, and group geo-temporal data. + +```js +import "experimental/geo" + +from(bucket: "geo/autogen") + |> range(start: -1w) + |> filter(fn: (r) => r._measurement == "taxi") + |> geo.shapeData(latField: "latitude", lonField: "longitude", level: 20) + |> geo.filterRows( + region: {lat: 40.69335938, lon: -73.30078125, radius: 20.0}, + strict: true + ) + |> geo.asTracks(groupBy: ["fare-id"]) +``` + + +## InfluxQL and Flux parity +Flux is working towards complete parity with InfluxQL and new functions are being added to that end. +The table below shows InfluxQL statements, clauses, and functions along with their equivalent Flux functions. + +_For a complete list of Flux functions, [view all Flux functions](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/all-functions)._ + +### InfluxQL and Flux parity + +| InfluxQL | Flux Functions | +| -------- | -------------- | +| [SELECT](/influxdb/v1.8/query_language/explore-data/#the-basic-select-statement) | [filter()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/filter/) | +| [WHERE](/influxdb/v1.8/query_language/explore-data/#the-where-clause) | [filter()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/filter/), [range()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/range/) | +| [GROUP BY](/influxdb/v1.8/query_language/explore-data/#the-group-by-clause) | [group()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/group/) | +| [INTO](/influxdb/v1.8/query_language/explore-data/#the-into-clause) | [to()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/outputs/to/) * | +| [ORDER BY](/influxdb/v1.8/query_language/explore-data/#order-by-time-desc) | [sort()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/sort/) | +| [LIMIT](/influxdb/v1.8/query_language/explore-data/#the-limit-clause) | [limit()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/limit/) | +| [SLIMIT](/influxdb/v1.8/query_language/explore-data/#the-slimit-clause) | -- | +| [OFFSET](/influxdb/v1.8/query_language/explore-data/#the-offset-clause) | -- | +| [SOFFSET](/influxdb/v1.8/query_language/explore-data/#the-soffset-clause) | -- | +| [SHOW DATABASES](/influxdb/v1.8/query_language/explore-schema/#show-databases) | [buckets()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/inputs/buckets/) | +| [SHOW MEASUREMENTS](/influxdb/v1.8/query_language/explore-schema/#show-measurements) | [v1.measurements](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/influxdb-v1/measurements) | +| [SHOW FIELD KEYS](/influxdb/v1.8/query_language/explore-schema/#show-field-keys) | [keys()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/keys/) | +| [SHOW RETENTION POLICIES](/influxdb/v1.8/query_language/explore-schema/#show-retention-policies) | [buckets()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/inputs/buckets/) | +| [SHOW TAG KEYS](/influxdb/v1.8/query_language/explore-schema/#show-tag-keys) | [v1.tagKeys()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/influxdb-v1/tagkeys), [v1.measurementTagKeys()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/influxdb-v1/measurementtagkeys) | +| [SHOW TAG VALUES](/influxdb/v1.8/query_language/explore-schema/#show-tag-values) | [v1.tagValues()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/influxdb-v1/tagvalues), [v1.measurementTagValues()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/influxdb-v1/measurementtagvalues) | +| [SHOW SERIES](/influxdb/v1.8/query_language/explore-schema/#show-series) | -- | +| [CREATE DATABASE](/influxdb/v1.8/query_language/database_management/#create-database) | -- | +| [DROP DATABASE](/influxdb/v1.8/query_language/database_management/#delete-a-database-with-drop-database) | -- | +| [DROP SERIES](/influxdb/v1.8/query_language/database_management/#drop-series-from-the-index-with-drop-serie) | -- | +| [DELETE](/influxdb/v1.8/query_language/database_management/#delete-series-with-delete) | -- | +| [DROP MEASUREMENT](/influxdb/v1.8/query_language/database_management/#delete-measurements-with-drop-measurement) | -- | +| [DROP SHARD](/influxdb/v1.8/query_language/database_management/#delete-a-shard-with-drop-shard) | -- | +| [CREATE RETENTION POLICY](/influxdb/v1.8/query_language/database_management/#create-retention-policies-with-create-retention-policy) | -- | +| [ALTER RETENTION POLICY](/influxdb/v1.8/query_language/database_management/#modify-retention-policies-with-alter-retention-policy) | -- | +| [DROP RETENTION POLICY](/influxdb/v1.8/query_language/database_management/#delete-retention-policies-with-drop-retention-policy) | -- | +| [COUNT](/influxdb/v1.8/query_language/functions#count) | [count()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/count/) | +| [DISTINCT](/influxdb/v1.8/query_language/functions#distinct) | [distinct()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/distinct/) | +| [INTEGRAL](/influxdb/v1.8/query_language/functions#integral) | [integral()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/integral/) | +| [MEAN](/influxdb/v1.8/query_language/functions#mean) | [mean()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/mean/) | +| [MEDIAN](/influxdb/v1.8/query_language/functions#median) | [median()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/median/) | +| [MODE](/influxdb/v1.8/query_language/functions#mode) | [mode()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/mode/) | +| [SPREAD](/influxdb/v1.8/query_language/functions#spread) | [spread()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/spread/) | +| [STDDEV](/influxdb/v1.8/query_language/functions#stddev) | [stddev()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/stddev/) | +| [SUM](/influxdb/v1.8/query_language/functions#sum) | [sum()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/sum/) | +| [BOTTOM](/influxdb/v1.8/query_language/functions#bottom) | [bottom()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/bottom/) | +| [FIRST](/influxdb/v1.8/query_language/functions#first) | [first()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/first/) | +| [LAST](/influxdb/v1.8/query_language/functions#last) | [last()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/last/) | +| [MAX](/influxdb/v1.8/query_language/functions#max) | [max()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/max/) | +| [MIN](/influxdb/v1.8/query_language/functions#min) | [min()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/min/) | +| [PERCENTILE](/influxdb/v1.8/query_language/functions#percentile) | [quantile()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/quantile/) | +| [SAMPLE](/influxdb/v1.8/query_language/functions#sample) | [sample()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/sample/) | +| [TOP](/influxdb/v1.8/query_language/functions#top) | [top()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/top/) | +| [ABS](/influxdb/v1.8/query_language/functions#abs) | [math.abs()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/abs/) | +| [ACOS](/influxdb/v1.8/query_language/functions#acos) | [math.acos()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/acos/) | +| [ASIN](/influxdb/v1.8/query_language/functions#asin) | [math.asin()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/asin/) | +| [ATAN](/influxdb/v1.8/query_language/functions#atan) | [math.atan()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/atan/) | +| [ATAN2](/influxdb/v1.8/query_language/functions#atan2) | [math.atan2()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/atan2/) | +| [CEIL](/influxdb/v1.8/query_language/functions#ceil) | [math.ceil()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/ceil/) | +| [COS](/influxdb/v1.8/query_language/functions#cos) | [math.cos()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/cos/) | +| [CUMULATIVE_SUM](/influxdb/v1.8/query_language/functions#cumulative-sum) | [cumulativeSum()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/cumulativesum/) | +| [DERIVATIVE](/influxdb/v1.8/query_language/functions#derivative) | [derivative()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/derivative/) | +| [DIFFERENCE](/influxdb/v1.8/query_language/functions#difference) | [difference()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/difference/) | +| [ELAPSED](/influxdb/v1.8/query_language/functions#elapsed) | [elapsed()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/elapsed/) | +| [EXP](/influxdb/v1.8/query_language/functions#exp) | [math.exp()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/exp/) | +| [FLOOR](/influxdb/v1.8/query_language/functions#floor) | [math.floor()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/floor/) | +| [HISTOGRAM](/influxdb/v1.8/query_language/functions#histogram) | [histogram()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/histogram/) | +| [LN](/influxdb/v1.8/query_language/functions#ln) | [math.log()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/log/) | +| [LOG](/influxdb/v1.8/query_language/functions#log) | [math.logb()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/logb/) | +| [LOG2](/influxdb/v1.8/query_language/functions#log2) | [math.log2()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/log2/) | +| [LOG10](/influxdb/v1.8/query_language/functions#logt10) | [math.log10()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/log10/) | +| [MOVING_AVERAGE](/influxdb/v1.8/query_language/functions#moving-average) | [movingAverage()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/movingaverage/) | +| [NON_NEGATIVE_DERIVATIVE](/influxdb/v1.8/query_language/functions#non-negative-derivative) | [derivative(nonNegative:true)](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/derivative/) | +| [NON_NEGATIVE_DIFFERENCE](/influxdb/v1.8/query_language/functions#non-negative-difference) | [difference(nonNegative:true)](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/derivative/) | +| [POW](/influxdb/v1.8/query_language/functions#pow) | [math.pow()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/pow/) | +| [ROUND](/influxdb/v1.8/query_language/functions#round) | [math.round()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/round/) | +| [SIN](/influxdb/v1.8/query_language/functions#sin) | [math.sin()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/sin/) | +| [SQRT](/influxdb/v1.8/query_language/functions#sqrt) | [math.sqrt()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/sqrt/) | +| [TAN](/influxdb/v1.8/query_language/functions#tan) | [math.tan()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/tan/) | +| [HOLT_WINTERS](/influxdb/v1.8/query_language/functions#holt-winters) | [holtWinters()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/holtwinters/) | +| [CHANDE_MOMENTUM_OSCILLATOR](/influxdb/v1.8/query_language/functions#chande-momentum-oscillator) | [chandeMomentumOscillator()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/chandemomentumoscillator/) | +| [EXPONENTIAL_MOVING_AVERAGE](/influxdb/v1.8/query_language/functions#exponential-moving-average) | [exponentialMovingAverage()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/exponentialmovingaverage/) | +| [DOUBLE_EXPONENTIAL_MOVING_AVERAGE](/influxdb/v1.8/query_language/functions#double-exponential-moving-average) | [doubleEMA()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/doubleema/) | +| [KAUFMANS_EFFICIENCY_RATIO](/influxdb/v1.8/query_language/functions#kaufmans-efficiency-ratio) | [kaufmansER()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/kaufmanser/) | +| [KAUFMANS_ADAPTIVE_MOVING_AVERAGE](/influxdb/v1.8/query_language/functions#kaufmans-adaptive-moving-average) | [kaufmansAMA()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/kaufmansama/) | +| [TRIPLE_EXPONENTIAL_MOVING_AVERAGE](/influxdb/v1.8/query_language/functions#triple-exponential-moving-average) | [tripleEMA()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/tripleema/) | +| [TRIPLE_EXPONENTIAL_DERIVATIVE](/influxdb/v1.8/query_language/functions#triple-exponential-derivative) | [tripleExponentialDerivative()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/tripleexponentialderivative/) | +| [RELATIVE_STRENGTH_INDEX](/influxdb/v1.8/query_language/functions#relative-strength-index) | [relativeStrengthIndex()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/relativestrengthindex/) | + +_* Theto() function only writes to InfluxDB 2.0._
diff --git a/content/influxdb/v1.8/flux/get-started/_index.md b/content/influxdb/v1.8/flux/get-started/_index.md
new file mode 100644
index 000000000..5a0426090
--- /dev/null
+++ b/content/influxdb/v1.8/flux/get-started/_index.md
@@ -0,0 +1,120 @@
+---
+title: Get started with Flux
+description: >
+ Get started with Flux, InfluxData's new functional data scripting language.
+ This step-by-step guide will walk you through the basics and get you on your way.
+menu:
+ influxdb_1_8:
+ name: Get started with Flux
+ identifier: get-started
+ parent: Flux
+ weight: 2
+---
+
+Flux is InfluxData's new functional data scripting language designed for querying,
+analyzing, and acting on data.
+
+This multi-part getting started guide walks through important concepts related to Flux.
+It covers querying time series data from InfluxDB using Flux, and introduces Flux syntax and functions.
+
+## What you will need
+
+##### InfluxDB v1.8+
+Flux v0.65 is built into InfluxDB v1.8 and can be used to query data stored in InfluxDB.
+InfluxDB v1.8's `influx` CLI also includes a `-type=` option that, when set to `flux`, will start an
+interactive Flux Read-Eval-Print-Loop (REPL) allowing you to write and run Flux queries from the command line.
+
+---
+
+_For information about downloading and installing InfluxDB, see [InfluxDB installation](/{{< latest "influxdb" "v1" >}}/introduction/installation)._
+
+---
+
+##### Chronograf v1.8+
+**Not required but strongly recommended**.
+Chronograf v1.8's Data Explorer provides a user interface (UI) for writing Flux scripts and visualizing results.
+Dashboards in Chronograf v1.8+ also support Flux queries.
+
+---
+
+_For information about downloading and installing Chronograf, see [Chronograf installation](/{{< latest "chronograf" >}}/introduction/installation)._
+
+---
+
+## Key concepts
+Flux introduces important new concepts you should understand as you get started.
+
+### Buckets
+Flux introduces "buckets," a new data storage concept for InfluxDB.
+A **bucket** is a named location where data is stored that has a retention policy.
+It's similar to an InfluxDB v1.x "database," but is a combination of both a database and a retention policy.
+When using multiple retention policies, each retention policy is treated as is its own bucket.
+
+Flux's [`from()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/inputs/from), which defines an InfluxDB data source, requires a `bucket` parameter.
+When using Flux with InfluxDB v1.x, use the following bucket naming convention which combines
+the database name and retention policy into a single bucket name:
+
+###### InfluxDB v1.x bucket naming convention
+```js
+// Pattern
+from(bucket:"
+ Query InfluxDB
+ Syntax basics
+
diff --git a/content/influxdb/v1.8/flux/guides/_index.md b/content/influxdb/v1.8/flux/guides/_index.md
new file mode 100644
index 000000000..5703ea31c
--- /dev/null
+++ b/content/influxdb/v1.8/flux/guides/_index.md
@@ -0,0 +1,35 @@
+---
+title: Query data with Flux
+description: Guides that walk through both common and complex queries and use cases for Flux.
+weight: 3
+menu:
+ influxdb_1_8:
+ name: Query with Flux
+ parent: Flux
+---
+
+The following guides walk through both common and complex queries and use cases for Flux.
+
+{{% note %}}
+#### Example data variable
+Many of the examples provided in the following guides use a `data` variable,
+which represents a basic query that filters data by measurement and field.
+`data` is defined as:
+
+```js
+data = from(bucket: "db/rp")
+ |> range(start: -1h)
+ |> filter(fn: (r) =>
+ r._measurement == "example-measurement" and
+ r._field == "example-field"
+ )
+```
+{{% /note %}}
+
+## Flux query guides
+
+{{< children type="anchored-list" pages="all" >}}
+
+---
+
+{{< children pages="all" readmore="true" hr="true" >}}
diff --git a/content/influxdb/v1.8/flux/guides/calculate-percentages.md b/content/influxdb/v1.8/flux/guides/calculate-percentages.md
new file mode 100644
index 000000000..d89bf18b7
--- /dev/null
+++ b/content/influxdb/v1.8/flux/guides/calculate-percentages.md
@@ -0,0 +1,208 @@
+---
+title: Calculate percentages with Flux
+list_title: Calculate percentages
+description: >
+ Use `pivot()` or `join()` and the `map()` function to align operand values into rows and calculate a percentage.
+menu:
+ influxdb_1_8:
+ name: Calculate percentages
+ identifier: flux-calc-perc
+ parent: Query with Flux
+weight: 6
+list_query_example: percentages
+---
+
+Calculating percentages from queried data is a common use case for time series data.
+To calculate a percentage in Flux, operands must be in each row.
+Use `map()` to re-map values in the row and calculate a percentage.
+
+**To calculate percentages**
+
+1. Use [`from()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/inputs/from/),
+ [`range()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/range/) and
+ [`filter()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/filter/) to query operands.
+2. Use [`pivot()` or `join()`](/influxdb/v1.8/flux/guides/mathematic-operations/#pivot-vs-join)
+ to align operand values into rows.
+3. Use [`map()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/map/)
+ to divide the numerator operand value by the denominator operand value and multiply by 100.
+
+{{% note %}}
+The following examples use `pivot()` to align operands into rows because
+`pivot()` works in most cases and is more performant than `join()`.
+_See [Pivot vs join](/influxdb/v1.8/flux/guides/mathematic-operations/#pivot-vs-join)._
+{{% /note %}}
+
+```js
+from(bucket: "db/rp")
+ |> range(start: -1h)
+ |> filter(fn: (r) => r._measurement == "m1" and r._field =~ /field[1-2]/ )
+ |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
+ |> map(fn: (r) => ({ r with _value: r.field1 / r.field2 * 100.0 }))
+```
+
+## GPU monitoring example
+The following example queries data from the gpu-monitor bucket and calculates the
+percentage of GPU memory used over time.
+Data includes the following:
+
+- **`gpu` measurement**
+- **`mem_used` field**: used GPU memory in bytes
+- **`mem_total` field**: total GPU memory in bytes
+
+### Query mem_used and mem_total fields
+```js
+from(bucket: "gpu-monitor")
+ |> range(start: 2020-01-01T00:00:00Z)
+ |> filter(fn: (r) => r._measurement == "gpu" and r._field =~ /mem_/)
+```
+
+###### Returns the following stream of tables:
+
+| _time | _measurement | _field | _value |
+|:----- |:------------:|:------: | ------: |
+| 2020-01-01T00:00:00Z | gpu | mem_used | 2517924577 |
+| 2020-01-01T00:00:10Z | gpu | mem_used | 2695091978 |
+| 2020-01-01T00:00:20Z | gpu | mem_used | 2576980377 |
+| 2020-01-01T00:00:30Z | gpu | mem_used | 3006477107 |
+| 2020-01-01T00:00:40Z | gpu | mem_used | 3543348019 |
+| 2020-01-01T00:00:50Z | gpu | mem_used | 4402341478 |
+
+
+
+| _time | _measurement | _field | _value |
+|:----- |:------------:|:------: | ------: |
+| 2020-01-01T00:00:00Z | gpu | mem_total | 8589934592 |
+| 2020-01-01T00:00:10Z | gpu | mem_total | 8589934592 |
+| 2020-01-01T00:00:20Z | gpu | mem_total | 8589934592 |
+| 2020-01-01T00:00:30Z | gpu | mem_total | 8589934592 |
+| 2020-01-01T00:00:40Z | gpu | mem_total | 8589934592 |
+| 2020-01-01T00:00:50Z | gpu | mem_total | 8589934592 |
+
+### Pivot fields into columns
+Use `pivot()` to pivot the `mem_used` and `mem_total` fields into columns.
+Output includes `mem_used` and `mem_total` columns with values for each corresponding `_time`.
+
+```js
+// ...
+ |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
+```
+
+###### Returns the following:
+
+| _time | _measurement | mem_used | mem_total |
+|:----- |:------------:| --------: | ---------: |
+| 2020-01-01T00:00:00Z | gpu | 2517924577 | 8589934592 |
+| 2020-01-01T00:00:10Z | gpu | 2695091978 | 8589934592 |
+| 2020-01-01T00:00:20Z | gpu | 2576980377 | 8589934592 |
+| 2020-01-01T00:00:30Z | gpu | 3006477107 | 8589934592 |
+| 2020-01-01T00:00:40Z | gpu | 3543348019 | 8589934592 |
+| 2020-01-01T00:00:50Z | gpu | 4402341478 | 8589934592 |
+
+### Map new values
+Each row now contains the values necessary to calculate a percentage.
+Use `map()` to re-map values in each row.
+Divide `mem_used` by `mem_total` and multiply by 100 to return the percentage.
+
+{{% note %}}
+To return a precise float percentage value that includes decimal points, the example
+below casts integer field values to floats and multiplies by a float value (`100.0`).
+{{% /note %}}
+
+```js
+// ...
+ |> map(fn: (r) => ({
+ _time: r._time,
+ _measurement: r._measurement,
+ _field: "mem_used_percent",
+ _value: float(v: r.mem_used) / float(v: r.mem_total) * 100.0
+ }))
+```
+##### Query results:
+
+| _time | _measurement | _field | _value |
+|:----- |:------------:|:------: | ------: |
+| 2020-01-01T00:00:00Z | gpu | mem_used_percent | 29.31 |
+| 2020-01-01T00:00:10Z | gpu | mem_used_percent | 31.37 |
+| 2020-01-01T00:00:20Z | gpu | mem_used_percent | 30.00 |
+| 2020-01-01T00:00:30Z | gpu | mem_used_percent | 35.00 |
+| 2020-01-01T00:00:40Z | gpu | mem_used_percent | 41.25 |
+| 2020-01-01T00:00:50Z | gpu | mem_used_percent | 51.25 |
+
+### Full query
+```js
+from(bucket: "gpu-monitor")
+ |> range(start: 2020-01-01T00:00:00Z)
+ |> filter(fn: (r) => r._measurement == "gpu" and r._field =~ /mem_/ )
+ |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
+ |> map(fn: (r) => ({
+ _time: r._time,
+ _measurement: r._measurement,
+ _field: "mem_used_percent",
+ _value: float(v: r.mem_used) / float(v: r.mem_total) * 100.0
+ }))
+```
+
+## Examples
+
+#### Calculate percentages using multiple fields
+```js
+from(bucket: "db/rp")
+ |> range(start: -1h)
+ |> filter(fn: (r) => r._measurement == "example-measurement")
+ |> filter(fn: (r) =>
+ r._field == "used_system" or
+ r._field == "used_user" or
+ r._field == "total"
+ )
+ |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
+ |> map(fn: (r) => ({ r with
+ _value: float(v: r.used_system + r.used_user) / float(v: r.total) * 100.0
+ }))
+```
+
+#### Calculate percentages using multiple measurements
+
+1. Ensure measurements are in the same [bucket](/influxdb/v1.8/flux/get-started/#buckets).
+2. Use `filter()` to include data from both measurements.
+3. Use `group()` to ungroup data and return a single table.
+4. Use `pivot()` to pivot fields into columns.
+5. Use `map()` to re-map rows and perform the percentage calculation.
+
+
+```js
+from(bucket: "db/rp")
+ |> range(start: -1h)
+ |> filter(fn: (r) =>
+ (r._measurement == "m1" or r._measurement == "m2") and
+ (r._field == "field1" or r._field == "field2")
+ )
+ |> group()
+ |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
+ |> map(fn: (r) => ({ r with _value: r.field1 / r.field2 * 100.0 }))
+```
+
+#### Calculate percentages using multiple data sources
+```js
+import "sql"
+import "influxdata/influxdb/secrets"
+
+pgUser = secrets.get(key: "POSTGRES_USER")
+pgPass = secrets.get(key: "POSTGRES_PASSWORD")
+pgHost = secrets.get(key: "POSTGRES_HOST")
+
+t1 = sql.from(
+ driverName: "postgres",
+ dataSourceName: "postgresql://${pgUser}:${pgPass}@${pgHost}",
+ query:"SELECT id, name, available FROM exampleTable"
+)
+
+t2 = from(bucket: "db/rp")
+ |> range(start: -1h)
+ |> filter(fn: (r) =>
+ r._measurement == "example-measurement" and
+ r._field == "example-field"
+ )
+
+join(tables: {t1: t1, t2: t2}, on: ["id"])
+ |> map(fn: (r) => ({ r with _value: r._value_t2 / r.available_t1 * 100.0 }))
+```
diff --git a/content/influxdb/v1.8/flux/guides/conditional-logic.md b/content/influxdb/v1.8/flux/guides/conditional-logic.md
new file mode 100644
index 000000000..d56ce090a
--- /dev/null
+++ b/content/influxdb/v1.8/flux/guides/conditional-logic.md
@@ -0,0 +1,195 @@
+---
+title: Query using conditional logic
+seotitle: Query using conditional logic in Flux
+list_title: Conditional logic
+description: >
+ This guide describes how to use Flux conditional expressions, such as `if`,
+ `else`, and `then`, to query and transform data. **Flux evaluates statements from left to right and stops evaluating once a condition matches.**
+menu:
+ influxdb_1_8:
+ name: Conditional logic
+ parent: Query with Flux
+weight: 20
+list_code_example: |
+ ```js
+ if color == "green" then "008000" else "ffffff"
+ ```
+---
+
+Flux provides `if`, `then`, and `else` conditional expressions that allow for powerful and flexible Flux queries.
+
+##### Conditional expression syntax
+```js
+// Pattern
+if
+
+### Script pane
+The Script pane is where you write your Flux script.
+In its default state, the **Script** pane includes an optional [Script Wizard](/chronograf/v1.8/guides/querying-data/#explore-data-with-flux)
+that uses selected options to build a Flux query for you.
+The generated query includes all the relevant functions and [template variables](#template-variables-in-flux)
+required to return your desired data.
+
+### Functions pane
+The Functions pane provides a list of functions available in your Flux queries.
+Clicking on a function will add it to the end of the script in the Script pane.
+Hovering over a function provides documentation for the function as well as links
+to deep documentation.
+
+### Dynamic sources
+Chronograf can be configured with multiple data sources.
+The **Sources dropdown** allows you to select a specific data source to connect to,
+but a **Dynamic Source** options is also available.
+With a dynamic source, the cell will query data from whatever data source to which
+Chronograf is currently connected.
+Connections are managed under Chronograf's **Configuration** tab.
+
+### View raw data
+As you're building your Flux scripts, each function processes or transforms your
+data is ways specific to the function.
+It can be helpful to view the actual data in order to see how it is being shaped.
+The **View Raw Data** toggle above the data visualization switches between graphed
+data and raw data shown in table form.
+
+
+
+_The **View Raw Data** toggle is only available when using Flux._
+
+## Template variables in Flux
+Chronograf [template variables](/{{< latest "chronograf" >}}/guides/dashboard-template-variables/)
+allow you to alter specific components of cells’ queries using elements provided in the
+Chronograf user interface.
+
+In your Flux query, reference template variables just as you would reference defined Flux variables.
+The following example uses Chronograf's [predefined template variables](#predefined-template-variables),
+`dashboardTime`, `upperDashboardTime`, and `autoInterval`:
+
+```js
+from(bucket: "telegraf/autogen")
+ |> filter(fn: (r) => r._measurement == "cpu")
+ |> range(
+ start: dashboardTime,
+ stop: upperDashboardTime
+ )
+ window(every: autoInterval)
+```
+
+### Predefined template variables
+
+#### dashboardTime
+The `dashboardTime` template variable represents the lower time bound of ranged data.
+It's value is controlled by the time dropdown in your dashboard.
+It should be used to define the `start` parameter of the `range()` function.
+
+```js
+dataSet
+ |> range(
+ start: dashboardTime
+ )
+```
+
+#### upperDashboardTime
+The `upperDashboardTime` template variable represents the upper time bound of ranged data.
+It's value is modified by the time dropdown in your dashboard when using an absolute time range.
+It should be used to define the `stop` parameter of the `range()` function.
+
+```js
+dataSet
+ |> range(
+ start: dashboardTime,
+ stop: upperDashboardTime
+ )
+```
+> As a best practice, always set the `stop` parameter of the `range()` function to `upperDashboardTime` in cell queries.
+> Without it, `stop` defaults to "now" and the absolute upper range bound selected in the time dropdown is not honored,
+> potentially causing unnecessary load on InfluxDB.
+
+#### autoInterval
+The `autoInterval` template variable represents the refresh interval of the dashboard
+and is controlled by the refresh interval dropdown.
+It's typically used to align window intervals created in
+[windowing and aggregation](/influxdb/v1.8/flux/guides/window-aggregate) operations with dashboard refreshes.
+
+```js
+dataSet
+ |> range(
+ start: dashboardTime,
+ stop: upperDashboardTime
+ )
+ |> aggregateWindow(
+ every: autoInterval,
+ fn: mean
+ )
+```
+
+### Custom template variables
+See Flux read-eval-print-loop (REPL).
+ +{{% note %}} +#### Operands must be the same type +Operands in Flux mathematic operations must be the same data type. +For example, integers cannot be used in operations with floats. +Otherwise, you will get an error similar to: + +``` +Error: type error: float != int +``` + +To convert operands to the same type, use [type-conversion functions](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/type-conversions/) +or manually format operands. +The operand data type determines the output data type. +For example: + +```js +100 // Parsed as an integer +100.0 // Parsed as a float + +// Example evaluations +> 20 / 8 +2 + +> 20.0 / 8.0 +2.5 +``` +{{% /note %}} + +## Custom mathematic functions +Flux lets you [create custom functions](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/custom-functions) that use mathematic operations. +View the examples below. + +###### Custom multiplication function +```js +multiply = (x, y) => x * y + +multiply(x: 10, y: 12) +// Returns 120 +``` + +###### Custom percentage function +```js +percent = (sample, total) => (sample / total) * 100.0 + +percent(sample: 20.0, total: 80.0) +// Returns 25.0 +``` + +### Transform values in a data stream +To transform multiple values in an input stream, your function needs to: + +- [Handle piped-forward data](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/custom-functions/#functions-that-manipulate-piped-forward-data). +- Each operand necessary for the calculation exists in each row _(see [Pivot vs join](#pivot-vs-join) below)_. +- Use the [`map()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/map) to iterate over each row. + +The example `multiplyByX()` function below includes: + +- A `tables` parameter that represents the input data stream (`<-`). +- An `x` parameter which is the number by which values in the `_value` column are multiplied. +- A `map()` function that iterates over each row in the input stream. + It uses the `with` operator to preserve existing columns in each row. + It also multiples the `_value` column by `x`. + +```js +multiplyByX = (x, tables=<-) => + tables + |> map(fn: (r) => ({ + r with + _value: r._value * x + }) + ) + +data + |> multiplyByX(x: 10) +``` + +## Examples + +### Convert bytes to gigabytes +To convert active memory from bytes to gigabytes (GB), divide the `active` field +in the `mem` measurement by 1,073,741,824. + +The `map()` function iterates over each row in the piped-forward data and defines +a new `_value` by dividing the original `_value` by 1073741824. + +```js +from(bucket: "db/rp") + |> range(start: -10m) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field == "active" + ) + |> map(fn: (r) => ({ + r with + _value: r._value / 1073741824 + }) + ) +``` + +You could turn that same calculation into a function: + +```js +bytesToGB = (tables=<-) => + tables + |> map(fn: (r) => ({ + r with + _value: r._value / 1073741824 + }) + ) + +data + |> bytesToGB() +``` + +#### Include partial gigabytes +Because the original metric (bytes) is an integer, the output of the operation is an integer and does not include partial GBs. +To calculate partial GBs, convert the `_value` column and its values to floats using the +[`float()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/type-conversions/float) +and format the denominator in the division operation as a float. + +```js +bytesToGB = (tables=<-) => + tables + |> map(fn: (r) => ({ + r with + _value: float(v: r._value) / 1073741824.0 + }) + ) +``` + +### Calculate a percentage +To calculate a percentage, use simple division, then multiply the result by 100. + +```js +> 1.0 / 4.0 * 100.0 +25.0 +``` + +_For an in-depth look at calculating percentages, see [Calculate percentates](/influxdb/v1.8/flux/guides/calculate-percentages)._ + +## Pivot vs join +To query and use values in mathematical operations in Flux, operand values must +exists in a single row. +Both `pivot()` and `join()` will do this, but there are important differences between the two: + +#### Pivot is more performant +`pivot()` reads and operates on a single stream of data. +`join()` requires two streams of data and the overhead of reading and combining +both streams can be significant, especially for larger data sets. + +#### Use join for multiple data sources +Use `join()` when querying data from different buckets or data sources. + +##### Pivot fields into columns for mathematic calculations +```js +data + |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value") + |> map(fn: (r) => ({ r with + _value: (r.field1 + r.field2) / r.field3 * 100.0 + })) +``` + +##### Join multiple data sources for mathematic calculations +```js +import "sql" +import "influxdata/influxdb/secrets" + +pgUser = secrets.get(key: "POSTGRES_USER") +pgPass = secrets.get(key: "POSTGRES_PASSWORD") +pgHost = secrets.get(key: "POSTGRES_HOST") + +t1 = sql.from( + driverName: "postgres", + dataSourceName: "postgresql://${pgUser}:${pgPass}@${pgHost}", + query:"SELECT id, name, available FROM exampleTable" +) + +t2 = from(bucket: "db/rp") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "example-measurement" and + r._field == "example-field" + ) + +join(tables: {t1: t1, t2: t2}, on: ["id"]) + |> map(fn: (r) => ({ r with _value: r._value_t2 / r.available_t1 * 100.0 })) +``` diff --git a/content/influxdb/v1.8/flux/guides/median.md b/content/influxdb/v1.8/flux/guides/median.md new file mode 100644 index 000000000..d3c0a5b1f --- /dev/null +++ b/content/influxdb/v1.8/flux/guides/median.md @@ -0,0 +1,144 @@ +--- +title: Find median values +seotitle: Find median values in Flux +list_title: Median +description: > + Use the `median()` function to return a value representing the `0.5` quantile (50th percentile) or median of input data. +weight: 10 +menu: + influxdb_1_8: + parent: Query with Flux + name: Median +list_query_example: median +--- + +Use the [`median()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/median/) +to return a value representing the `0.5` quantile (50th percentile) or median of input data. + +## Select a method for calculating the median +Select one of the following methods to calculate the median: + +- [estimate_tdigest](#estimate-tdigest) +- [exact_mean](#exact-mean) +- [exact_selector](#exact-selector) + +### estimate_tdigest +**(Default)** An aggregate method that uses a [t-digest data structure](https://github.com/tdunning/t-digest) +to compute an accurate `0.5` quantile estimate on large data sources. +Output tables consist of a single row containing the calculated median. + +{{< flex >}} +{{% flex-content %}} +**Given the following input table:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.0 | +| 2020-01-01T00:03:00Z | 2.0 | +| 2020-01-01T00:04:00Z | 3.0 | +{{% /flex-content %}} +{{% flex-content %}} +**`estimate_tdigest` returns:** + +| _value | +|:------:| +| 1.5 | +{{% /flex-content %}} +{{< /flex >}} + +### exact_mean +An aggregate method that takes the average of the two points closest to the `0.5` quantile value. +Output tables consist of a single row containing the calculated median. + +{{< flex >}} +{{% flex-content %}} +**Given the following input table:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.0 | +| 2020-01-01T00:03:00Z | 2.0 | +| 2020-01-01T00:04:00Z | 3.0 | +{{% /flex-content %}} +{{% flex-content %}} +**`exact_mean` returns:** + +| _value | +|:------:| +| 1.5 | +{{% /flex-content %}} +{{< /flex >}} + +### exact_selector +A selector method that returns the data point for which at least 50% of points are less than. +Output tables consist of a single row containing the calculated median. + +{{< flex >}} +{{% flex-content %}} +**Given the following input table:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.0 | +| 2020-01-01T00:03:00Z | 2.0 | +| 2020-01-01T00:04:00Z | 3.0 | +{{% /flex-content %}} +{{% flex-content %}} +**`exact_selector` returns:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:02:00Z | 1.0 | +{{% /flex-content %}} +{{< /flex >}} + +{{% note %}} +The examples below use the [example data variable](/influxdb/v1.8/flux/guides/#example-data-variable). +{{% /note %}} + +## Find the value that represents the median +Use the default method, `"estimate_tdigest"`, to return all rows in a table that +contain values in the 50th percentile of data in the table. + +```js +data + |> median() +``` + +## Find the average of values closest to the median +Use the `exact_mean` method to return a single row per input table containing the +average of the two values closest to the mathematical median of data in the table. + +```js +data + |> median(method: "exact_mean") +``` + +## Find the point with the median value +Use the `exact_selector` method to return a single row per input table containing the +value that 50% of values in the table are less than. + +```js +data + |> median(method: "exact_selector") +``` + +## Use median() with aggregateWindow() +[`aggregateWindow()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/aggregatewindow/) +segments data into windows of time, aggregates data in each window into a single +point, and then removes the time-based segmentation. +It is primarily used to downsample data. + +To specify the [median calculation method](#select-a-method-for-calculating-the-median) in `aggregateWindow()`, use the +[full function syntax](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/aggregatewindow/#specify-parameters-of-the-aggregate-function): + +```js +data + |> aggregateWindow( + every: 5m, + fn: (tables=<-, column) => tables |> median(method: "exact_selector") + ) +``` diff --git a/content/influxdb/v1.8/flux/guides/monitor-states.md b/content/influxdb/v1.8/flux/guides/monitor-states.md new file mode 100644 index 000000000..c3a0ba990 --- /dev/null +++ b/content/influxdb/v1.8/flux/guides/monitor-states.md @@ -0,0 +1,149 @@ +--- +title: Monitor states +seotitle: Monitor states and state changes in your events and metrics with Flux. +description: Flux provides several functions to help monitor states and state changes in your data. +menu: + influxdb_1_8: + name: Monitor states + parent: Query with Flux +weight: 20 +--- + +Flux helps you monitor states in your metrics and events: + +- [Find how long a state persists](#find-how-long-a-state-persists) +- [Count the number of consecutive states](#count-the-number-of-consecutive-states) +- [Detect state changes](#example-query-to-count-machine-state) + +If you're just getting started with Flux queries, check out the following: + +- [Get started with Flux](/influxdb/v1.8/flux/get-started/) for a conceptual overview of Flux. +- [Execute queries](/influxdb/v1.8/flux/guides/executing-queries/) to discover a variety of ways to run your queries. + +## Find how long a state persists + +1. Use the [`stateDuration()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/stateduration/) function to calculate how long a column value has remained the same value (or state). Include the following information: + + - **Column to search:** any tag key, tag value, field key, field value, or measurement. + - **Value:** the value (or state) to search for in the specified column. + - **State duration column:** a new column to store the state duration─the length of time that the specified value persists. + - **Unit:** the unit of time (`1s` (by default), `1m`, `1h`) used to increment the state duration. + + + ```js + |> stateDuration( + fn: (r) => + r._column_to_search == "value_to_search_for", + column: "state_duration", + unit: 1s + ) + ``` + +2. Use `stateDuration()` to search each point for the specified value: + + - For the first point that evaluates `true`, the state duration is set to `0`. For each consecutive point that evaluates `true`, the state duration increases by the time interval between each consecutive point (in specified units). + - If the state is `false`, the state duration is reset to `-1`. + +### Example query with stateDuration() + +The following query searches the `doors` bucket over the past 5 minutes to find how many seconds a door has been `closed`. + +```js +from(bucket: "doors") + |> range(start: -5m) + |> stateDuration( + fn: (r) => + r._value == "closed", + column: "door_closed", + unit: 1s + ) +``` + +In this example, `door_closed` is the **State duration** column. If you write data to the `doors` bucket every minute, the state duration increases by `60s` for each consecutive point where `_value` is `closed`. If `_value` is not `closed`, the state duration is reset to `0`. + +#### Query results + +Results for the example query above may look like this (for simplicity, we've omitted the measurement, tag, and field columns): + +```bash +_time _value door_closed +2019-10-26T17:39:16Z closed 0 +2019-10-26T17:40:16Z closed 60 +2019-10-26T17:41:16Z closed 120 +2019-10-26T17:42:16Z open -1 +2019-10-26T17:43:16Z closed 0 +2019-10-26T17:44:27Z closed 60 +``` + +## Count the number of consecutive states + +1. Use the `stateCount()` function and include the following information: + + - **Column to search:** any tag key, tag value, field key, field value, or measurement. + - **Value:** to search for in the specified column. + - **State count column:** a new column to store the state count─the number of consecutive records in which the specified value exists. + + + ```js + |> stateCount + (fn: (r) => + r._column_to_search == "value_to_search_for", + column: "state_count" + ) + ``` + +2. Use `stateCount()` to search each point for the specified value: + + - For the first point that evaluates `true`, the state count is set to `1`. For each consecutive point that evaluates `true`, the state count increases by 1. + - If the state is `false`, the state count is reset to `-1`. + +### Example query with stateCount() + +The following query searches the `doors` bucket over the past 5 minutes and +calculates how many points have `closed` as their `_value`. + +```js +from(bucket: "doors") + |> range(start: -5m) + |> stateDuration( + fn: (r) => + r._value == "closed", + column: "door_closed") +``` + +This example stores the **state count** in the `door_closed` column. +If you write data to the `doors` bucket every minute, the state count increases +by `1` for each consecutive point where `_value` is `closed`. +If `_value` is not `closed`, the state count is reset to `-1`. + +#### Query results + +Results for the example query above may look like this (for simplicity, we've omitted the measurement, tag, and field columns): + +```bash +_time _value door_closed +2019-10-26T17:39:16Z closed 1 +2019-10-26T17:40:16Z closed 2 +2019-10-26T17:41:16Z closed 3 +2019-10-26T17:42:16Z open -1 +2019-10-26T17:43:16Z closed 1 +2019-10-26T17:44:27Z closed 2 +``` + +#### Example query to count machine state + +The following query checks the machine state every minute (idle, assigned, or busy). +InfluxDB searches the `servers` bucket over the past hour and counts records with a machine state of `idle`, `assigned` or `busy`. + +```js +from(bucket: "servers") + |> range(start: -1h) + |> filter(fn: (r) => + r.machine_state == "idle" or + r.machine_state == "assigned" or + r.machine_state == "busy" + ) + |> stateCount(fn: (r) => r.machine_state == "busy", column: "_count") + |> stateCount(fn: (r) => r.machine_state == "assigned", column: "_count") + |> stateCount(fn: (r) => r.machine_state == "idle", column: "_count") +``` diff --git a/content/influxdb/v1.8/flux/guides/moving-average.md b/content/influxdb/v1.8/flux/guides/moving-average.md new file mode 100644 index 000000000..92dbb05e2 --- /dev/null +++ b/content/influxdb/v1.8/flux/guides/moving-average.md @@ -0,0 +1,113 @@ +--- +title: Calculate the moving average +seotitle: Calculate the moving average in Flux +list_title: Moving Average +description: > + Use the `movingAverage()` or `timedMovingAverage()` functions to return the moving average of data. +weight: 10 +menu: + influxdb_1_8: + parent: Query with Flux + name: Moving Average +list_query_example: moving_average +--- + +Use the [`movingAverage()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/movingaverage/) +or [`timedMovingAverage()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/timedmovingaverage/) +functions to return the moving average of data. + +```js +data + |> movingAverage(n: 5) + +// OR + +data + |> timedMovingAverage(every: 5m, period: 10m) +``` + +### movingAverage() +For each row in a table, `movingAverage()` returns the average of the current value and +**previous** values where `n` is the total number of values used to calculate the average. + +If `n = 3`: + +| Row # | Calculation | +|:-----:|:----------- | +| 1 | _Insufficient number of rows_ | +| 2 | _Insufficient number of rows_ | +| 3 | (Row1 + Row2 + Row3) / 3 | +| 4 | (Row2 + Row3 + Row4) / 3 | +| 5 | (Row3 + Row4 + Row5) / 3 | + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.2 | +| 2020-01-01T00:03:00Z | 1.8 | +| 2020-01-01T00:04:00Z | 0.9 | +| 2020-01-01T00:05:00Z | 1.4 | +| 2020-01-01T00:06:00Z | 2.0 | +{{% /flex-content %}} +{{% flex-content %}} +**The following would return:** + +```js +|> movingAverage(n: 3) +``` + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:03:00Z | 1.33 | +| 2020-01-01T00:04:00Z | 1.30 | +| 2020-01-01T00:05:00Z | 1.36 | +| 2020-01-01T00:06:00Z | 1.43 | +{{% /flex-content %}} +{{< /flex >}} + +### timedMovingAverage() +For each row in a table, `timedMovingAverage()` returns the average of the +current value and all row values in the **previous** `period` (duration). +It returns moving averages at a frequency defined by the `every` parameter. + +Each color in the diagram below represents a period of time used to calculate an +average and the time a point representing the average is returned. +If `every = 30m` and `period = 1h`: + +{{< svg "/static/svgs/timed-moving-avg.svg" >}} + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.2 | +| 2020-01-01T00:03:00Z | 1.8 | +| 2020-01-01T00:04:00Z | 0.9 | +| 2020-01-01T00:05:00Z | 1.4 | +| 2020-01-01T00:06:00Z | 2.0 | +{{% /flex-content %}} +{{% flex-content %}} +**The following would return:** + +```js +|> timedMovingAverage( + every: 2m, + period: 4m +) +``` + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:02:00Z | 1.000 | +| 2020-01-01T00:04:00Z | 1.333 | +| 2020-01-01T00:06:00Z | 1.325 | +| 2020-01-01T00:06:00Z | 1.150 | +{{% /flex-content %}} +{{< /flex >}} diff --git a/content/influxdb/v1.8/flux/guides/optimize-queries.md b/content/influxdb/v1.8/flux/guides/optimize-queries.md new file mode 100644 index 000000000..2cec9154f --- /dev/null +++ b/content/influxdb/v1.8/flux/guides/optimize-queries.md @@ -0,0 +1,74 @@ +--- +title: Optimize Flux queries +description: > + Optimize your Flux queries to reduce their memory and compute (CPU) requirements. +weight: 30 +menu: + influxdb_1_8: + name: Optimize queries + parent: Query with Flux +--- + +Optimize your Flux queries to reduce their memory and compute (CPU) requirements. + +- [Start queries with pushdown functions](#start-queries-with-pushdown-functions) +- [Avoid short window durations](#avoid-short-window-durations) +- [Use "heavy" functions sparingly](#use-heavy-functions-sparingly) +- [Balance time range and data precision](#balance-time-range-and-data-precision) + +## Start queries with pushdown functions +Some Flux functions can push their data manipulation down to the underlying +data source rather than storing and manipulating data in memory. +These are known as "pushdown" functions and using them correctly can greatly +reduce the amount of memory necessary to run a query. + +#### Pushdown functions +- [range()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/range/) +- [filter()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/filter/) +- [group()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/group/) + +Use pushdown functions at the beginning of your query. +Once a non-pushdown function runs, Flux pulls data into memory and runs all +subsequent operations there. + +##### Pushdown functions in use +```js +from(bucket: "db/rp") + |> range(start: -1h) // + |> filter(fn: (r) => r.sensor == "abc123") // Pushed to the data source + |> group(columns: ["_field", "host"]) // + + |> aggregateWindow(every: 5m, fn: max) // + |> filter(fn: (r) => r._value >= 90.0) // Run in memory + |> top(n: 10) // +``` + +## Avoid short window durations +Windowing (grouping data based on time intervals) is commonly used to aggregate and downsample data. +Increase performance by avoiding short window durations. +More windows require more compute power to evaluate which window each row should be assigned to. +Reasonable window durations depend on the total time range queried. + +## Use "heavy" functions sparingly +The following functions use more memory or CPU than others. +Consider their necessity in your data processing before using them: + +- [map()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/map/) +- [reduce()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/reduce/) +- [window()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/window/) +- [join()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/join/) +- [union()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/union/) +- [pivot()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/pivot/) + +{{% note %}} +We're continually optimizing Flux and this list may not represent its current state. +{{% /note %}} + +## Balance time range and data precision +To ensure queries are performant, balance the time range and the precision of your data. +For example, if you query data stored every second and request six months worth of data, +results will include a minimum of ≈15.5 million points. +Flux must store these points in memory to generate a response. + +To query data over large periods of time, create a [continuous query](/influxdb/v1.8/query_language/continuous_queries/) +to downsample data, and then query the downsampled data instead. diff --git a/content/influxdb/v1.8/flux/guides/percentile-quantile.md b/content/influxdb/v1.8/flux/guides/percentile-quantile.md new file mode 100644 index 000000000..9c4c13cbf --- /dev/null +++ b/content/influxdb/v1.8/flux/guides/percentile-quantile.md @@ -0,0 +1,161 @@ +--- +title: Find percentile and quantile values +seotitle: Query percentile and quantile values in Flux +list_title: Percentile & quantile +description: > + Use the `quantile()` function to return all values within the `q` quantile or + percentile of input data. +weight: 10 +menu: + influxdb_1_8: + parent: Query with Flux + name: Percentile & quantile +list_query_example: quantile +--- + +Use the [`quantile()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/quantile/) +to return a value representing the `q` quantile or percentile of input data. + +## Percentile versus quantile +Percentiles and quantiles are very similar, differing only in the number used to calculate return values. +A percentile is calculated using numbers between `0` and `100`. +A quantile is calculated using numbers between `0.0` and `1.0`. +For example, the **`0.5` quantile** is the same as the **50th percentile**. + +## Select a method for calculating the quantile +Select one of the following methods to calculate the quantile: + +- [estimate_tdigest](#estimate-tdigest) +- [exact_mean](#exact-mean) +- [exact_selector](#exact-selector) + +### estimate_tdigest +**(Default)** An aggregate method that uses a [t-digest data structure](https://github.com/tdunning/t-digest) +to compute a quantile estimate on large data sources. +Output tables consist of a single row containing the calculated quantile. + +If calculating the `0.5` quantile or 50th percentile: + +{{< flex >}} +{{% flex-content %}} +**Given the following input table:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.0 | +| 2020-01-01T00:03:00Z | 2.0 | +| 2020-01-01T00:04:00Z | 3.0 | +{{% /flex-content %}} +{{% flex-content %}} +**`estimate_tdigest` returns:** + +| _value | +|:------:| +| 1.5 | +{{% /flex-content %}} +{{< /flex >}} + +### exact_mean +An aggregate method that takes the average of the two points closest to the quantile value. +Output tables consist of a single row containing the calculated quantile. + +If calculating the `0.5` quantile or 50th percentile: + +{{< flex >}} +{{% flex-content %}} +**Given the following input table:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.0 | +| 2020-01-01T00:03:00Z | 2.0 | +| 2020-01-01T00:04:00Z | 3.0 | +{{% /flex-content %}} +{{% flex-content %}} +**`exact_mean` returns:** + +| _value | +|:------:| +| 1.5 | +{{% /flex-content %}} +{{< /flex >}} + +### exact_selector +A selector method that returns the data point for which at least `q` points are less than. +Output tables consist of a single row containing the calculated quantile. + +If calculating the `0.5` quantile or 50th percentile: + +{{< flex >}} +{{% flex-content %}} +**Given the following input table:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.0 | +| 2020-01-01T00:03:00Z | 2.0 | +| 2020-01-01T00:04:00Z | 3.0 | +{{% /flex-content %}} +{{% flex-content %}} +**`exact_selector` returns:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:02:00Z | 1.0 | +{{% /flex-content %}} +{{< /flex >}} + +{{% note %}} +The examples below use the [example data variable](/influxdb/v1.8/flux/guides/#example-data-variable). +{{% /note %}} + +## Find the value representing the 99th percentile +Use the default method, `"estimate_tdigest"`, to return all rows in a table that +contain values in the 99th percentile of data in the table. + +```js +data + |> quantile(q: 0.99) +``` + +## Find the average of values closest to the quantile +Use the `exact_mean` method to return a single row per input table containing the +average of the two values closest to the mathematical quantile of data in the table. +For example, to calculate the `0.99` quantile: + +```js +data + |> quantile(q: 0.99, method: "exact_mean") +``` + +## Find the point with the quantile value +Use the `exact_selector` method to return a single row per input table containing the +value that `q * 100`% of values in the table are less than. +For example, to calculate the `0.99` quantile: + +```js +data + |> quantile(q: 0.99, method: "exact_selector") +``` + +## Use quantile() with aggregateWindow() +[`aggregateWindow()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/aggregatewindow/) +segments data into windows of time, aggregates data in each window into a single +point, and then removes the time-based segmentation. +It is primarily used to downsample data. + +To specify the [quantile calculation method](#select-a-method-for-calculating-the-quantile) in +`aggregateWindow()`, use the [full function syntax](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/aggregatewindow/#specify-parameters-of-the-aggregate-function): + +```js +data + |> aggregateWindow( + every: 5m, + fn: (tables=<-, column) => + tables + |> quantile(q: 0.99, method: "exact_selector") + ) +``` diff --git a/content/influxdb/v1.8/flux/guides/query-fields.md b/content/influxdb/v1.8/flux/guides/query-fields.md new file mode 100644 index 000000000..573c97577 --- /dev/null +++ b/content/influxdb/v1.8/flux/guides/query-fields.md @@ -0,0 +1,76 @@ +--- +title: Query fields and tags +seotitle: Query fields and tags in InfluxDB using Flux +description: > + Use the `filter()` function to query data based on fields, tags, or any other column value. + `filter()` performs operations similar to the `SELECT` statement and the `WHERE` + clause in InfluxQL and other SQL-like query languages. +weight: 1 +menu: + influxdb_1_8: + parent: Query with Flux +list_code_example: | + ```js + from(bucket: "db/rp") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "example-measurement" and + r._field == "example-field" and + r.tag == "example-tag" + ) + ``` +--- + +Use the [`filter()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/filter/) +to query data based on fields, tags, or any other column value. +`filter()` performs operations similar to the `SELECT` statement and the `WHERE` +clause in InfluxQL and other SQL-like query languages. + +## The filter() function +`filter()` has an `fn` parameter that expects a **predicate function**, +an anonymous function comprised of one or more **predicate expressions**. +The predicate function evaluates each input row. +Rows that evaluate to `true` are **included** in the output data. +Rows that evaluate to `false` are **excluded** from the output data. + +```js +// ... + |> filter(fn: (r) => r._measurement == "example-measurement" ) +``` + +The `fn` predicate function requires an `r` argument, which represents each row +as `filter()` iterates over input data. +Key-value pairs in the row record represent columns and their values. +Use **dot notation** or **bracket notation** to reference specific column values in the predicate function. +Use [logical operators](/{{< latest "influxdb" "v2" >}}/reference/flux/language/operators/#logical-operators) +to chain multiple predicate expressions together. + +```js +// Row record +r = {foo: "bar", baz: "quz"} + +// Example predicate function +(r) => r.foo == "bar" and r["baz"] == "quz" + +// Evaluation results +(r) => true and true +``` + +## Filter by fields and tags +The combination of [`from()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/inputs/from), +[`range()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/range), +and `filter()` represent the most basic Flux query: + +1. Use `from()` to define your [bucket](/influxdb/v1.8/flux/get-started/#buckets). +2. Use `range()` to limit query results by time. +3. Use `filter()` to identify what rows of data to output. + +```js +from(bucket: "db/rp") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "example-measurement" and + r._field == "example-field" and + r.tag == "example-tag" + ) +``` diff --git a/content/influxdb/v1.8/flux/guides/rate.md b/content/influxdb/v1.8/flux/guides/rate.md new file mode 100644 index 000000000..5a95f4050 --- /dev/null +++ b/content/influxdb/v1.8/flux/guides/rate.md @@ -0,0 +1,173 @@ +--- +title: Calculate the rate of change +seotitle: Calculate the rate of change in Flux +list_title: Rate +description: > + Use the `derivative()` function to calculate the rate of change between subsequent values or the + `aggregate.rate()` function to calculate the average rate of change per window of time. + If time between points varies, these functions normalize points to a common time interval + making values easily comparable. +weight: 10 +menu: + influxdb_1_8: + parent: Query with Flux + name: Rate +list_query_example: rate_of_change +--- + + +Use the [`derivative()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/derivative/) +to calculate the rate of change between subsequent values or the +[`aggregate.rate()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/aggregate/rate/) +to calculate the average rate of change per window of time. +If time between points varies, these functions normalize points to a common time interval +making values easily comparable. + +- [Rate of change between subsequent values](#rate-of-change-between-subsequent-values) +- [Average rate of change per window of time](#average-rate-of-change-per-window-of-time) + +## Rate of change between subsequent values +Use the [`derivative()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/derivative/) +to calculate the rate of change per unit of time between subsequent _non-null_ values. + +```js +data + |> derivative(unit: 1s) +``` + +By default, `derivative()` returns only positive derivative values and replaces negative values with _null_. +Cacluated values are returned as [floats](/{{< latest "influxdb" "v2" >}}/reference/flux/language/types/#numeric-types). + + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:00:00Z | 250 | +| 2020-01-01T00:04:00Z | 160 | +| 2020-01-01T00:12:00Z | 150 | +| 2020-01-01T00:19:00Z | 220 | +| 2020-01-01T00:32:00Z | 200 | +| 2020-01-01T00:51:00Z | 290 | +| 2020-01-01T01:00:00Z | 340 | +{{% /flex-content %}} +{{% flex-content %}} +**`derivative(unit: 1m)` returns:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:04:00Z | | +| 2020-01-01T00:12:00Z | | +| 2020-01-01T00:19:00Z | 10.0 | +| 2020-01-01T00:32:00Z | | +| 2020-01-01T00:51:00Z | 4.74 | +| 2020-01-01T01:00:00Z | 5.56 | +{{% /flex-content %}} +{{< /flex >}} + +Results represent the rate of change **per minute** between subsequent values with +negative values set to _null_. + +### Return negative derivative values +To return negative derivative values, set the `nonNegative` parameter to `false`, + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:00:00Z | 250 | +| 2020-01-01T00:04:00Z | 160 | +| 2020-01-01T00:12:00Z | 150 | +| 2020-01-01T00:19:00Z | 220 | +| 2020-01-01T00:32:00Z | 200 | +| 2020-01-01T00:51:00Z | 290 | +| 2020-01-01T01:00:00Z | 340 | +{{% /flex-content %}} +{{% flex-content %}} +**The following returns:** + +```js +|> derivative( + unit: 1m, + nonNegative: false +) +``` + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:04:00Z | -22.5 | +| 2020-01-01T00:12:00Z | -1.25 | +| 2020-01-01T00:19:00Z | 10.0 | +| 2020-01-01T00:32:00Z | -1.54 | +| 2020-01-01T00:51:00Z | 4.74 | +| 2020-01-01T01:00:00Z | 5.56 | +{{% /flex-content %}} +{{< /flex >}} + +Results represent the rate of change **per minute** between subsequent values and +include negative values. + +## Average rate of change per window of time + +Use the [`aggregate.rate()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/aggregate/rate/) +to calculate the average rate of change per window of time. + +```js +import "experimental/aggregate" + +data + |> aggregate.rate( + every: 1m, + unit: 1s, + groupColumns: ["tag1", "tag2"] + ) +``` + +`aggregate.rate()` returns the average rate of change (as a [float](/{{< latest "influxdb" "v2" >}}/reference/flux/language/types/#numeric-types)) +per `unit` for time intervals defined by `every`. +Negative values are replaced with _null_. + +{{% note %}} +`aggregate.rate()` does not support `nonNegative: false`. +{{% /note %}} + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:00:00Z | 250 | +| 2020-01-01T00:04:00Z | 160 | +| 2020-01-01T00:12:00Z | 150 | +| 2020-01-01T00:19:00Z | 220 | +| 2020-01-01T00:32:00Z | 200 | +| 2020-01-01T00:51:00Z | 290 | +| 2020-01-01T01:00:00Z | 340 | +{{% /flex-content %}} +{{% flex-content %}} +**The following returns:** + +```js +|> aggregate.rate( + every: 20m, + unit: 1m +) +``` + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:20:00Z | | +| 2020-01-01T00:40:00Z | 10.0 | +| 2020-01-01T01:00:00Z | 4.74 | +| 2020-01-01T01:20:00Z | 5.56 | +{{% /flex-content %}} +{{< /flex >}} + +Results represent the **average change rate per minute** of every **20 minute interval** +with negative values set to _null_. +Timestamps represent the right bound of the time window used to average values. diff --git a/content/influxdb/v1.8/flux/guides/regular-expressions.md b/content/influxdb/v1.8/flux/guides/regular-expressions.md new file mode 100644 index 000000000..9259df2d7 --- /dev/null +++ b/content/influxdb/v1.8/flux/guides/regular-expressions.md @@ -0,0 +1,91 @@ +--- +title: Use regular expressions in Flux +list_title: Regular expressions +description: This guide walks through using regular expressions in evaluation logic in Flux functions. +menu: + influxdb_1_8: + name: Regular expressions + parent: Query with Flux +weight: 20 +list_query_example: regular_expressions +--- + +Regular expressions (regexes) are incredibly powerful when matching patterns in large collections of data. +With Flux, regular expressions are primarily used for evaluation logic in predicate functions for things +such as filtering rows, dropping and keeping columns, state detection, etc. +This guide shows how to use regular expressions in your Flux scripts. + +If you're just getting started with Flux queries, check out the following: + +- [Get started with Flux](/influxdb/v1.8/flux/get-started/) for a conceptual overview of Flux and parts of a Flux query. +- [Execute queries](/influxdb/v1.8/flux/guides/execute-queries/) to discover a variety of ways to run your queries. + +## Go regular expression syntax +Flux uses Go's [regexp package](https://golang.org/pkg/regexp/) for regular expression search. +The links [below](#helpful-links) provide information about Go's regular expression syntax. + +## Regular expression operators +Flux provides two comparison operators for use with regular expressions. + +#### `=~` +When the expression on the left **MATCHES** the regular expression on the right, this evaluates to `true`. + +#### `!~` +When the expression on the left **DOES NOT MATCH** the regular expression on the right, this evaluates to `true`. + +## Regular expressions in Flux +When using regex matching in your Flux scripts, enclose your regular expressions with `/`. +The following is the basic regex comparison syntax: + +###### Basic regex comparison syntax +```js +expression =~ /regex/ +expression !~ /regex/ +``` +## Examples + +### Use a regex to filter by tag value +The following example filters records by the `cpu` tag. +It only keeps records for which the `cpu` is either `cpu0`, `cpu1`, or `cpu2`. + +```js +from(bucket: "db/rp") + |> range(start: -15m) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_user" and + r.cpu =~ /cpu[0-2]/ + ) +``` + +### Use a regex to filter by field key +The following example excludes records that do not have `_percent` in a field key. + +```js +from(bucket: "db/rp") + |> range(start: -15m) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field =~ /_percent/ + ) +``` + +### Drop columns matching a regex +The following example drops columns whose names do not being with `_`. + +```js +from(bucket: "db/rp") + |> range(start: -15m) + |> filter(fn: (r) => r._measurement == "mem") + |> drop(fn: (column) => column !~ /_.*/) +``` + +## Helpful links + +##### Syntax documentation +[regexp Syntax GoDoc](https://godoc.org/regexp/syntax) +[RE2 Syntax Overview](https://github.com/google/re2/wiki/Syntax) + +##### Go regex testers +[Regex Tester - Golang](https://regex-golang.appspot.com/assets/html/index.html) +[Regex101](https://regex101.com/) diff --git a/content/influxdb/v1.8/flux/guides/scalar-values.md b/content/influxdb/v1.8/flux/guides/scalar-values.md new file mode 100644 index 000000000..d0092c813 --- /dev/null +++ b/content/influxdb/v1.8/flux/guides/scalar-values.md @@ -0,0 +1,260 @@ +--- +title: Extract scalar values in Flux +list_title: Extract scalar values +description: > + Use Flux stream and table functions to extract scalar values from Flux query output. + This lets you, for example, dynamically set variables using query results. +menu: + influxdb_1_8: + name: Extract scalar values + parent: Query with Flux +weight: 20 +list_code_example: | + ```js + scalarValue = { + _record = + data + |> tableFind(fn: key => true) + |> getRecord(idx: 0) + return _record._value + } + ``` +--- + +Use Flux [stream and table functions](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/stream-table/) +to extract scalar values from Flux query output. +This lets you, for example, dynamically set variables using query results. + +**To extract scalar values from output:** + +1. [Extract a table](#extract-a-table). +2. [Extract a column from the table](#extract-a-column-from-the-table) + _**or**_ [extract a row from the table](#extract-a-row-from-the-table). + +_The samples on this page use the [sample data provided below](#sample-data)._ + +{{% warn %}} +#### Current limitations +- The InfluxDB user interface (UI) does not currently support raw scalar output. + Use [`map()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/map/) to add + scalar values to output data. +- The [Flux REPL](/influxdb/v1.8/flux/guides/execute-queries/#influx-cli) does not currently support + Flux stream and table functions (also known as "dynamic queries"). + See [#15321](https://github.com/influxdata/influxdb/issues/15231). +{{% /warn %}} + +## Extract a table +Flux formats query results as a stream of tables. +To extract a scalar value from a stream of tables, you must first extract a single table. + +to extract a single table from the stream of tables. + +{{% note %}} +If query results include only one table, it is still formatted as a stream of tables. +You still must extract that table from the stream. +{{% /note %}} + +Use [`tableFind()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/stream-table/tablefind/) +to extract the **first** table whose [group key](/influxdb/v1.8/flux/get-started/#group-keys) +values match the `fn` **predicate function**. +The predicate function requires a `key` record, which represents the group key of +each table. + +```js +sampleData + |> tableFind(fn: (key) => + key._field == "temp" and + key.location == "sfo" + ) +``` + +The example above returns a single table: + +| _time | location | _field | _value | +|:----- |:--------:|:------:| ------:| +| 2019-11-01T12:00:00Z | sfo | temp | 65.1 | +| 2019-11-01T13:00:00Z | sfo | temp | 66.2 | +| 2019-11-01T14:00:00Z | sfo | temp | 66.3 | +| 2019-11-01T15:00:00Z | sfo | temp | 66.8 | + +{{% note %}} +#### Extract the correct table +Flux functions do not guarantee table order and `tableFind()` returns only the +**first** table that matches the `fn` predicate. +To extract the table that includes the data you actually want, be very specific in +your predicate function or filter and transform your data to minimize the number +of tables piped-forward into `tableFind()`. +{{% /note %}} + +## Extract a column from the table +Use the [`getColumn()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/stream-table/getcolumn/) +to output an array of values from a specific column in the extracted table. + + +```js +sampleData + |> tableFind(fn: (key) => + key._field == "temp" and + key.location == "sfo" + ) + |> getColumn(column: "_value") + +// Returns [65.1, 66.2, 66.3, 66.8] +``` + +### Use extracted column values +Use a variable to store the array of values. +In the example below, `SFOTemps` represents the array of values. +Reference a specific index (integer starting from `0`) in the array to return the +value at that index. + +```js +SFOTemps = sampleData + |> tableFind(fn: (key) => + key._field == "temp" and + key.location == "sfo" + ) + |> getColumn(column: "_value") + +SFOTemps +// Returns [65.1, 66.2, 66.3, 66.8] + +SFOTemps[0] +// Returns 65.1 + +SFOTemps[2] +// Returns 66.3 +``` + +## Extract a row from the table +Use the [`getRecord()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/stream-table/getrecord/) +to output data from a single row in the extracted table. +Specify the index of the row to output using the `idx` parameter. +The function outputs a record with key-value pairs for each column. + +```js +sampleData + |> tableFind(fn: (key) => + key._field == "temp" and + key.location == "sfo" + ) + |> getRecord(idx: 0) + +// Returns { +// _time:2019-11-11T12:00:00Z, +// _field:"temp", +// location:"sfo", +// _value: 65.1 +// } +``` + +### Use an extracted row record +Use a variable to store the extracted row record. +In the example below, `tempInfo` represents the extracted row. +Use [dot notation](/influxdb/v1.8/flux/get-started/syntax-basics/#records) to reference +keys in the record. + +```js +tempInfo = sampleData + |> tableFind(fn: (key) => + key._field == "temp" and + key.location == "sfo" + ) + |> getRecord(idx: 0) + +tempInfo +// Returns { +// _time:2019-11-11T12:00:00Z, +// _field:"temp", +// location:"sfo", +// _value: 65.1 +// } + +tempInfo._time +// Returns 2019-11-11T12:00:00Z + +tempInfo.location +// Returns sfo +``` + +## Example helper functions +Create custom helper functions to extract scalar values from query output. + +##### Extract a scalar field value +```js +// Define a helper function to extract field values +getFieldValue = (tables=<-, field) => { + extract = tables + |> tableFind(fn: (key) => key._field == field) + |> getColumn(column: "_value") + return extract[0] +} + +// Use the helper function to define a variable +lastJFKTemp = sampleData + |> filter(fn: (r) => r.location == "kjfk") + |> last() + |> getFieldValue(field: "temp") + +lastJFKTemp +// Returns 71.2 +``` + +##### Extract scalar row data +```js +// Define a helper function to extract a row as a record +getRow = (tables=<-, field, idx=0) => { + extract = tables + |> tableFind(fn: (key) => true) + |> getRecord(idx: idx) + return extract +} + +// Use the helper function to define a variable +lastReported = sampleData + |> last() + |> getRow(idx: 0) + +"The last location to report was ${lastReported.location}. +The temperature was ${string(v: lastReported._value)}°F." + +// Returns: +// The last location to report was kord. +// The temperature was 38.9°F. +``` + +--- + +## Sample data + +The following sample data set represents fictional temperature metrics collected +from three locations. +It's formatted in [annotated CSV](https://v2.docs.influxdata.com/v2.0/reference/syntax/annotated-csv/) and imported +into the Flux query using the [`csv.from()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/csv/from/). + +Place the following at the beginning of your query to use the sample data: + +{{% truncate %}} +```js +import "csv" + +sampleData = csv.from(csv: " +#datatype,string,long,dateTime:RFC3339,string,string,double +#group,false,true,false,true,true,false +#default,,,,,, +,result,table,_time,location,_field,_value +,,0,2019-11-01T12:00:00Z,sfo,temp,65.1 +,,0,2019-11-01T13:00:00Z,sfo,temp,66.2 +,,0,2019-11-01T14:00:00Z,sfo,temp,66.3 +,,0,2019-11-01T15:00:00Z,sfo,temp,66.8 +,,1,2019-11-01T12:00:00Z,kjfk,temp,69.4 +,,1,2019-11-01T13:00:00Z,kjfk,temp,69.9 +,,1,2019-11-01T14:00:00Z,kjfk,temp,71.0 +,,1,2019-11-01T15:00:00Z,kjfk,temp,71.2 +,,2,2019-11-01T12:00:00Z,kord,temp,46.4 +,,2,2019-11-01T13:00:00Z,kord,temp,46.3 +,,2,2019-11-01T14:00:00Z,kord,temp,42.7 +,,2,2019-11-01T15:00:00Z,kord,temp,38.9 +") +``` +{{% /truncate %}} diff --git a/content/influxdb/v1.8/flux/guides/sort-limit.md b/content/influxdb/v1.8/flux/guides/sort-limit.md new file mode 100644 index 000000000..e8092d04b --- /dev/null +++ b/content/influxdb/v1.8/flux/guides/sort-limit.md @@ -0,0 +1,68 @@ +--- +title: Sort and limit data with Flux +seotitle: Sort and limit data in InfluxDB with Flux +list_title: Sort and limit +description: > + Use the `sort()`function to order records within each table by specific columns and the + `limit()` function to limit the number of records in output tables to a fixed number, `n`. +menu: + influxdb_1_8: + name: Sort and limit + parent: Query with Flux +weight: 3 +list_query_example: sort_limit +--- + +Use the [`sort()`function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/sort) +to order records within each table by specific columns and the +[`limit()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/limit) +to limit the number of records in output tables to a fixed number, `n`. + +If you're just getting started with Flux queries, check out the following: + +- [Get started with Flux](/influxdb/v1.8/flux/get-started/) for a conceptual overview of Flux and parts of a Flux query. +- [Execute queries](/influxdb/v1.8/flux/guides/execute-queries/) to discover a variety of ways to run your queries. + +##### Example sorting system uptime + +The following example orders system uptime first by region, then host, then value. + +```js +from(bucket:"db/rp") + |> range(start:-12h) + |> filter(fn: (r) => + r._measurement == "system" and + r._field == "uptime" + ) + |> sort(columns:["region", "host", "_value"]) +``` + +The [`limit()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/limit) +limits the number of records in output tables to a fixed number, `n`. +The following example shows up to 10 records from the past hour. + +```js +from(bucket:"db/rp") + |> range(start:-1h) + |> limit(n:10) +``` + +You can use `sort()` and `limit()` together to show the top N records. +The example below returns the 10 top system uptime values sorted first by +region, then host, then value. + +```js +from(bucket:"db/rp") + |> range(start:-12h) + |> filter(fn: (r) => + r._measurement == "system" and + r._field == "uptime" + ) + |> sort(columns:["region", "host", "_value"]) + |> limit(n:10) +``` + +You now have created a Flux query that sorts and limits data. +Flux also provides the [`top()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/top) +and [`bottom()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/bottom) +functions to perform both of these functions at the same time. diff --git a/content/influxdb/v1.8/flux/guides/sql.md b/content/influxdb/v1.8/flux/guides/sql.md new file mode 100644 index 000000000..a6a731f56 --- /dev/null +++ b/content/influxdb/v1.8/flux/guides/sql.md @@ -0,0 +1,213 @@ +--- +title: Query SQL data sources +seotitle: Query SQL data sources with InfluxDB +list_title: Query SQL data +description: > + The Flux `sql` package provides functions for working with SQL data sources. + Use `sql.from()` to query SQL databases like PostgreSQL and MySQL +menu: + influxdb_1_8: + parent: Query with Flux + list_title: SQL data +weight: 20 +list_code_example: | + ```js + import "sql" + + sql.from( + driverName: "postgres", + dataSourceName: "postgresql://user:password@localhost", + query: "SELECT * FROM example_table" + ) + ``` +--- + +The Flux `sql` package provides functions for working with SQL data sources. +[`sql.from()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/sql/from/) lets you query SQL data sources +like [PostgreSQL](https://www.postgresql.org/), [MySQL](https://www.mysql.com/), +and [SQLite](https://www.sqlite.org/index.html), and use the results with InfluxDB +dashboards, tasks, and other operations. + +- [Query a SQL data source](#query-a-sql-data-source) +- [Join SQL data with data in InfluxDB](#join-sql-data-with-data-in-influxdb) +- [Sample sensor data](#sample-sensor-data) + +## Query a SQL data source +To query a SQL data source: + +1. Import the `sql` package in your Flux query +2. Use the `sql.from()` function to specify the driver, data source name (DSN), + and query used to query data from your SQL data source: + +{{< code-tabs-wrapper >}} +{{% code-tabs %}} +[PostgreSQL](#) +[MySQL](#) +[SQLite](#) +{{% /code-tabs %}} + +{{% code-tab-content %}} +```js +import "sql" + +sql.from( + driverName: "postgres", + dataSourceName: "postgresql://user:password@localhost", + query: "SELECT * FROM example_table" +) +``` +{{% /code-tab-content %}} + +{{% code-tab-content %}} +```js +import "sql" + +sql.from( + driverName: "mysql", + dataSourceName: "user:password@tcp(localhost:3306)/db", + query: "SELECT * FROM example_table" +) +``` +{{% /code-tab-content %}} + +{{% code-tab-content %}} +```js +// NOTE: InfluxDB OSS and InfluxDB Cloud do not have access to +// the local filesystem and cannot query SQLite data sources. +// Use the Flux REPL to query an SQLite data source. + +import "sql" +sql.from( + driverName: "sqlite3", + dataSourceName: "file:/path/to/test.db?cache=shared&mode=ro", + query: "SELECT * FROM example_table" +) +``` +{{% /code-tab-content %}} +{{< /code-tabs-wrapper >}} + +_See the [`sql.from()` documentation](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/sql/from/) for +information about required function parameters._ + +## Join SQL data with data in InfluxDB +One of the primary benefits of querying SQL data sources from InfluxDB +is the ability to enrich query results with data stored outside of InfluxDB. + +Using the [air sensor sample data](#sample-sensor-data) below, the following query +joins air sensor metrics stored in InfluxDB with sensor information stored in PostgreSQL. +The joined data lets you query and filter results based on sensor information +that isn't stored in InfluxDB. + +```js +// Import the "sql" package +import "sql" + +// Query data from PostgreSQL +sensorInfo = sql.from( + driverName: "postgres", + dataSourceName: "postgresql://localhost?sslmode=disable", + query: "SELECT * FROM sensors" +) + +// Query data from InfluxDB +sensorMetrics = from(bucket: "telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "airSensors") + +// Join InfluxDB query results with PostgreSQL query results +join(tables: {metric: sensorMetrics, info: sensorInfo}, on: ["sensor_id"]) +``` + +--- + +## Sample sensor data +The [sample data generator](#download-and-run-the-sample-data-generator) and +[sample sensor information](#import-the-sample-sensor-information) simulate a +group of sensors that measure temperature, humidity, and carbon monoxide +in rooms throughout a building. +Each collected data point is stored in InfluxDB with a `sensor_id` tag that identifies +the specific sensor it came from. +Sample sensor information is stored in PostgreSQL. + +**Sample data includes:** + +- Simulated data collected from each sensor and stored in the `airSensors` measurement in **InfluxDB**: + - temperature + - humidity + - co + +- Information about each sensor stored in the `sensors` table in **PostgreSQL**: + - sensor_id + - location + - model_number + - last_inspected + +### Import and generate sample sensor data + +#### Download and run the sample data generator +`air-sensor-data.rb` is a script that generates air sensor data and stores the data in InfluxDB. +To use `air-sensor-data.rb`: + +1. [Create a database](/influxdb/v1.8/introduction/getting-started/#creating-a-database) to store the data. +2. Download the sample data generator. _This tool requires [Ruby](https://www.ruby-lang.org/en/)._ + + Download Air Sensor Generator + +3. Give `air-sensor-data.rb` executable permissions: + + ``` + chmod +x air-sensor-data.rb + ``` + +4. Start the generator. Specify your database. + + ``` + ./air-sensor-data.rb -d database-name + ``` + + The generator begins to write data to InfluxDB and will continue until stopped. + Use `ctrl-c` to stop the generator. + + _**Note:** Use the `--help` flag to view other configuration options._ + + +5. Query your target database to ensure the generated data is writing successfully. + The generator doesn't catch errors from write requests, so it will continue running + even if data is not writing to InfluxDB successfully. + + ``` + from(bucket: "database-name/autogen") + |> range(start: -1m) + |> filter(fn: (r) => r._measurement == "airSensors") + ``` + +#### Import the sample sensor information +1. [Download and install PostgreSQL](https://www.postgresql.org/download/). +2. Download the sample sensor information CSV. + + Download Sample Data + +3. Use a PostgreSQL client (`psql` or a GUI) to create the `sensors` table: + + ``` + CREATE TABLE sensors ( + sensor_id character varying(50), + location character varying(50), + model_number character varying(50), + last_inspected date + ); + ``` + +4. Import the downloaded CSV sample data. + _Update the `FROM` file path to the path of the downloaded CSV sample data._ + + ``` + COPY sensors(sensor_id,location,model_number,last_inspected) + FROM '/path/to/sample-sensor-info.csv' DELIMITER ',' CSV HEADER; + ``` + +5. Query the table to ensure the data was imported correctly: + + ``` + SELECT * FROM sensors; + ``` diff --git a/content/influxdb/v1.8/flux/guides/window-aggregate.md b/content/influxdb/v1.8/flux/guides/window-aggregate.md new file mode 100644 index 000000000..40af0e5b5 --- /dev/null +++ b/content/influxdb/v1.8/flux/guides/window-aggregate.md @@ -0,0 +1,352 @@ +--- +title: Window and aggregate data with Flux +seotitle: Window and aggregate data in InfluxDB with Flux +list_title: Window & aggregate +description: > + This guide walks through windowing and aggregating data with Flux and outlines + how it shapes your data in the process. +menu: + influxdb_1_8: + name: Window & aggregate + parent: Query with Flux +weight: 4 +list_query_example: aggregate_window +--- + +A common operation performed with time series data is grouping data into windows of time, +or "windowing" data, then aggregating windowed values into a new value. +This guide walks through windowing and aggregating data with Flux and demonstrates +how data is shaped in the process. + +If you're just getting started with Flux queries, check out the following: + +- [Get started with Flux](/influxdb/v1.8/flux/get-started/) for a conceptual overview of Flux and parts of a Flux query. +- [Execute queries](/influxdb/v1.8/flux/guides/execute-queries/) to discover a variety of ways to run your queries. + +{{% note %}} +The following example is an in-depth walk-through of the steps required to window and aggregate data. +The [`aggregateWindow()` function](#summing-up) performs these operations for you, but understanding +how data is shaped in the process helps to successfully create your desired output. +{{% /note %}} + +## Data set +For the purposes of this guide, define a variable that represents your base data set. +The following example queries the memory usage of the host machine. + +```js +dataSet = from(bucket: "db/rp") + |> range(start: -5m) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field == "used_percent" + ) + |> drop(columns: ["host"]) +``` + +{{% note %}} +This example drops the `host` column from the returned data since the memory data +is only tracked for a single host and it simplifies the output tables. +Dropping the `host` column is optional and not recommended if monitoring memory +on multiple hosts. +{{% /note %}} + +`dataSet` can now be used to represent your base data, which will look similar to the following: + +{{% truncate %}} +``` +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:00.000000000Z 71.11611366271973 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:10.000000000Z 67.39630699157715 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:20.000000000Z 64.16666507720947 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:30.000000000Z 64.19951915740967 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:40.000000000Z 64.2122745513916 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:50.000000000Z 64.22209739685059 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:00.000000000Z 64.6336555480957 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:10.000000000Z 64.16516304016113 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:20.000000000Z 64.18349742889404 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:30.000000000Z 64.20474052429199 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:40.000000000Z 68.65062713623047 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:50.000000000Z 67.20139980316162 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:00.000000000Z 70.9143877029419 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:10.000000000Z 64.14549350738525 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:20.000000000Z 64.15379047393799 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:30.000000000Z 64.1592264175415 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:40.000000000Z 64.18190002441406 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:50.000000000Z 64.28837776184082 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:00.000000000Z 64.29731845855713 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:10.000000000Z 64.36963081359863 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:20.000000000Z 64.37397003173828 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:30.000000000Z 64.44413661956787 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:40.000000000Z 64.42906856536865 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:50.000000000Z 64.44573402404785 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:00.000000000Z 64.48912620544434 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:10.000000000Z 64.49522972106934 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:20.000000000Z 64.48652744293213 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:30.000000000Z 64.49949741363525 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:40.000000000Z 64.4949197769165 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:50.000000000Z 64.49787616729736 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49816226959229 +``` +{{% /truncate %}} + +## Windowing data +Use the [`window()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/window) +to group your data based on time bounds. +The most common parameter passed with the `window()` is `every` which +defines the duration of time between windows. +Other parameters are available, but for this example, window the base data +set into one minute windows. + +```js +dataSet + |> window(every: 1m) +``` + +{{% note %}} +The `every` parameter supports all [valid duration units](/{{< latest "influxdb" "v2" >}}/reference/flux/language/types/#duration-types), +including **calendar months (`1mo`)** and **years (`1y`)**. +{{% /note %}} + +Each window of time is output in its own table containing all records that fall within the window. + +{{% truncate %}} +###### window() output tables +``` +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:00.000000000Z 71.11611366271973 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:10.000000000Z 67.39630699157715 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:20.000000000Z 64.16666507720947 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:30.000000000Z 64.19951915740967 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:40.000000000Z 64.2122745513916 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:50.000000000Z 64.22209739685059 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:00.000000000Z 64.6336555480957 +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:10.000000000Z 64.16516304016113 +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:20.000000000Z 64.18349742889404 +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:30.000000000Z 64.20474052429199 +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:40.000000000Z 68.65062713623047 +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:50.000000000Z 67.20139980316162 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:00.000000000Z 70.9143877029419 +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:10.000000000Z 64.14549350738525 +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:20.000000000Z 64.15379047393799 +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:30.000000000Z 64.1592264175415 +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:40.000000000Z 64.18190002441406 +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:50.000000000Z 64.28837776184082 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:00.000000000Z 64.29731845855713 +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:10.000000000Z 64.36963081359863 +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:20.000000000Z 64.37397003173828 +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:30.000000000Z 64.44413661956787 +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:40.000000000Z 64.42906856536865 +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:50.000000000Z 64.44573402404785 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:00.000000000Z 64.48912620544434 +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:10.000000000Z 64.49522972106934 +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:20.000000000Z 64.48652744293213 +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:30.000000000Z 64.49949741363525 +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:40.000000000Z 64.4949197769165 +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:50.000000000Z 64.49787616729736 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:55:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49816226959229 +``` +{{% /truncate %}} + +When visualized in the InfluxDB UI, each window table is displayed in a different color. + + + +## Aggregate data +[Aggregate functions](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates) take the values +of all rows in a table and use them to perform an aggregate operation. +The result is output as a new value in a single-row table. + +Since windowed data is split into separate tables, aggregate operations run against +each table separately and output new tables containing only the aggregated value. + +For this example, use the [`mean()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/mean) +to output the average of each window: + +```js +dataSet + |> window(every: 1m) + |> mean() +``` + +{{% truncate %}} +###### mean() output tables +``` +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 65.88549613952637 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 65.50651391347249 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 65.30719598134358 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 64.39330975214641 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 64.49386278788249 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:55:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 64.49816226959229 +``` +{{% /truncate %}} + +Because each data point is contained in its own table, when visualized, +they appear as single, unconnected points. + + + +### Recreate the time column +**Notice the `_time` column is not in the [aggregated output tables](#mean-output-tables).** +Because records in each table are aggregated together, their timestamps no longer +apply and the column is removed from the group key and table. + +Also notice the `_start` and `_stop` columns still exist. +These represent the lower and upper bounds of the time window. + +Many Flux functions rely on the `_time` column. +To further process your data after an aggregate function, you need to re-add `_time`. +Use the [`duplicate()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/duplicate) to +duplicate either the `_start` or `_stop` column as a new `_time` column. + +```js +dataSet + |> window(every: 1m) + |> mean() + |> duplicate(column: "_stop", as: "_time") +``` + +{{% truncate %}} +###### duplicate() output tables +``` +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:51:00.000000000Z 65.88549613952637 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:52:00.000000000Z 65.50651391347249 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:53:00.000000000Z 65.30719598134358 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:54:00.000000000Z 64.39330975214641 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49386278788249 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:55:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49816226959229 +``` +{{% /truncate %}} + +## "Unwindow" aggregate tables +Keeping aggregate values in separate tables generally isn't the format in which you want your data. +Use the `window()` function to "unwindow" your data into a single infinite (`inf`) window. + +```js +dataSet + |> window(every: 1m) + |> mean() + |> duplicate(column: "_stop", as: "_time") + |> window(every: inf) +``` + +{{% note %}} +Windowing requires a `_time` column which is why it's necessary to +[recreate the `_time` column](#recreate-the-time-column) after an aggregation. +{{% /note %}} + +###### Unwindowed output table +``` +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:00.000000000Z 65.88549613952637 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:00.000000000Z 65.50651391347249 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:00.000000000Z 65.30719598134358 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:00.000000000Z 64.39330975214641 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49386278788249 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49816226959229 +``` + +With the aggregate values in a single table, data points in the visualization are connected. + + + +## Summing up +You have now created a Flux query that windows and aggregates data. +The data transformation process outlined in this guide should be used for all aggregation operations. + +Flux also provides the [`aggregateWindow()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/aggregatewindow) +which performs all these separate functions for you. + +The following Flux query will return the same results: + +###### aggregateWindow function +```js +dataSet + |> aggregateWindow(every: 1m, fn: mean) +``` diff --git a/content/influxdb/v1.8/flux/installation.md b/content/influxdb/v1.8/flux/installation.md new file mode 100644 index 000000000..dfd7176b5 --- /dev/null +++ b/content/influxdb/v1.8/flux/installation.md @@ -0,0 +1,33 @@ +--- +title: Enable Flux +description: Instructions for enabling Flux in your InfluxDB configuration. +menu: + influxdb_1_8: + name: Enable Flux + parent: Flux + weight: 1 +--- + +Flux is packaged with **InfluxDB v1.8+** and does not require any additional installation, +however it is **disabled by default and needs to be enabled**. + +## Enable Flux +Enable Flux by setting the `flux-enabled` option to `true` under the `[http]` section of your `influxdb.conf`: + +###### influxdb.conf +```toml +# ... + +[http] + + # ... + + flux-enabled = true + + # ... +``` + +> The default location of your `influxdb.conf` depends on your operating system. +> More information is available in the [Configuring InfluxDB](/influxdb/v1.8/administration/config/#using-the-configuration-file) guide. + +When InfluxDB starts, the Flux daemon starts as well and data can be queried using Flux. diff --git a/content/influxdb/v1.8/query_language/functions.md b/content/influxdb/v1.8/query_language/functions.md index 2e561e0c5..385f2cf61 100644 --- a/content/influxdb/v1.8/query_language/functions.md +++ b/content/influxdb/v1.8/query_language/functions.md @@ -4374,7 +4374,7 @@ InfluxDB then rounds those averages down to the nearest integer. _InfluxQL does not currently support histogram generation. For information about creating histograms with data stored in InfluxDB, see -[Flux's `histogram()` function](/flux/v0.7/functions/transformations/histogram)._ +[Flux's `histogram()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/transformations/histogram)._ ### LN() diff --git a/content/influxdb/v1.8/troubleshooting/query_management.md b/content/influxdb/v1.8/troubleshooting/query_management.md index d9c49020c..698635f74 100644 --- a/content/influxdb/v1.8/troubleshooting/query_management.md +++ b/content/influxdb/v1.8/troubleshooting/query_management.md @@ -14,7 +14,7 @@ Manage your InfluxQL queries using the following: - [KILL QUERIES](#stop-currently-running-queries-with-kill-query) to stop queries overloading your system - [Configuration settings](#configuration-settings-for-query-management) to prevent and halt the execution of inefficient queries -> The commands and configurations provided on this page are for **Influx Query Language (InfluxQL) only** -- **no equivalent set of Flux commands and configurations currently exists**. For the most current Flux documentation, see [Get started with Flux](/flux/v0.50/introduction/getting-started/). +> The commands and configurations provided on this page are for **Influx Query Language (InfluxQL) only** -- **no equivalent set of Flux commands and configurations currently exists**. For the most current Flux documentation, see [Get started with Flux](/influxdb/v1.8/flux/get-started/). ## List currently-running queries with `SHOW QUERIES` diff --git a/content/influxdb/v2.0/monitor-alert/templates/infrastructure/aws.md b/content/influxdb/v2.0/monitor-alert/templates/infrastructure/aws.md index 2c79ca100..5ae8cc299 100644 --- a/content/influxdb/v2.0/monitor-alert/templates/infrastructure/aws.md +++ b/content/influxdb/v2.0/monitor-alert/templates/infrastructure/aws.md @@ -29,7 +29,7 @@ The AWS CloudWatch Monitoring template includes the following: influx apply -f https://raw.githubusercontent.com/influxdata/community-templates/master/aws_cloudwatch/aws_cloudwatch.yml ``` For more information, see [influx apply](/v2.0/reference/cli/influx/apply/). -2. [Install Telegraf](/telegraf/latest/introduction/installation/) on a server with network access to both the CloudWatch API and [InfluxDB v2 API](/v2.0/reference/api/). +2. [Install Telegraf](/{{< latest "telegraf" >}}/introduction/installation/) on a server with network access to both the CloudWatch API and [InfluxDB v2 API](/v2.0/reference/api/). 3. In your Telegraf configuration file (`telegraf.conf`), find the following example `influxdb_v2` output plugins, and then **replace** the `urls` to specify the servers to monitor: ```sh diff --git a/content/influxdb/v2.0/monitor-alert/templates/infrastructure/docker.md b/content/influxdb/v2.0/monitor-alert/templates/infrastructure/docker.md index 8bb882870..6a1dd63a1 100644 --- a/content/influxdb/v2.0/monitor-alert/templates/infrastructure/docker.md +++ b/content/influxdb/v2.0/monitor-alert/templates/infrastructure/docker.md @@ -34,9 +34,9 @@ For more information about how checks, notification endpoints, and notifications ``` For more information, see [influx apply](/v2.0/reference/cli/influx/apply/). - > **Note:** Ensure your `influx` CLI is configured with your account credentials and that configuration is active. For more information, see [influx config](https://v2.docs.influxdata.com/v2.0/reference/cli/influx/config/). + > **Note:** Ensure your `influx` CLI is configured with your account credentials and that configuration is active. For more information, see [influx config](/influxdb/v2.0/reference/cli/influx/config/). -2. [Install Telegraf](/telegraf/latest/introduction/installation/) on a server with network access to both the Docker containers and [InfluxDB v2 API](/v2.0/reference/api/). +2. [Install Telegraf](/{{< latest "telegraf" >}}/introduction/installation/) on a server with network access to both the Docker containers and [InfluxDB v2 API](/v2.0/reference/api/). 3. In your [Telegraf configuration file (`telegraf.conf`)](/v2.0/write-data/no-code/use-telegraf/auto-config/view-telegraf-config/), do the following: - Depending on how you run Docker, you may need to customize the [Docker input plugin](/v2.0/reference/telegraf-plugins/#docker) configuration, for example, you may need to specify the `endpoint` value. - Set the following environment variables: diff --git a/content/influxdb/v2.0/monitor-alert/templates/infrastructure/vshpere.md b/content/influxdb/v2.0/monitor-alert/templates/infrastructure/vshpere.md index 564e953be..da815446c 100644 --- a/content/influxdb/v2.0/monitor-alert/templates/infrastructure/vshpere.md +++ b/content/influxdb/v2.0/monitor-alert/templates/infrastructure/vshpere.md @@ -29,9 +29,9 @@ The Docker Monitoring template includes the following: ``` For more information, see [influx apply](/v2.0/reference/cli/influx/apply/). - > **Note:** Ensure your `influx` CLI is configured with your account credentials and that configuration is active. For more information, see [influx config](https://v2.docs.influxdata.com/v2.0/reference/cli/influx/config/). + > **Note:** Ensure your `influx` CLI is configured with your account credentials and that configuration is active. For more information, see [influx config](/influxdb/v2.0/reference/cli/influx/config/). -2. [Install Telegraf](/telegraf/latest/introduction/installation/) on a server with network access to both the vSphere host and [InfluxDB v2 API](/v2.0/reference/api/). +2. [Install Telegraf](/{{< latest "telegraf" >}}/introduction/installation/) on a server with network access to both the vSphere host and [InfluxDB v2 API](/v2.0/reference/api/). 3. In your [Telegraf configuration file (`telegraf.conf`)](/v2.0/write-data/no-code/use-telegraf/auto-config/view-telegraf-config/), do the following: - Set the following environment variables: - INFLUX_TOKEN: Token must have permissions to read Telegraf configurations and write data to the `telegraf` bucket. See how to [view tokens](/v2.0/security/tokens/view-tokens/). diff --git a/content/influxdb/v2.0/monitor-alert/templates/infrastructure/windows.md b/content/influxdb/v2.0/monitor-alert/templates/infrastructure/windows.md index d1b461125..d39febcf4 100644 --- a/content/influxdb/v2.0/monitor-alert/templates/infrastructure/windows.md +++ b/content/influxdb/v2.0/monitor-alert/templates/infrastructure/windows.md @@ -28,9 +28,9 @@ The Windows System Monitoring template includes the following: ``` For more information, see [influx apply](/v2.0/reference/cli/influx/apply/). - > **Note:** Ensure your `influx` CLI is configured with your account credentials and that configuration is active. For more information, see [influx config](https://v2.docs.influxdata.com/v2.0/reference/cli/influx/config/). + > **Note:** Ensure your `influx` CLI is configured with your account credentials and that configuration is active. For more information, see [influx config](/influxdb/v2.0/reference/cli/influx/config/). -2. [Install Telegraf](/telegraf/latest/introduction/installation/) on a server with network access to both the Windows system and [InfluxDB v2 API](/v2.0/reference/api/). +2. [Install Telegraf](/{{< latest "telegraf" >}}/introduction/installation/) on a server with network access to both the Windows system and [InfluxDB v2 API](/v2.0/reference/api/). 3. In your [Telegraf configuration file (`telegraf.conf`)](/v2.0/write-data/no-code/use-telegraf/auto-config/view-telegraf-config/), do the following: - Set the following environment variables: - INFLUX_TOKEN: Token must have permissions to read Telegraf configurations and write data to the `telegraf` bucket. See how to [view tokens](/v2.0/security/tokens/view-tokens/). diff --git a/content/influxdb/v2.0/monitor-alert/templates/networks/haproxy.md b/content/influxdb/v2.0/monitor-alert/templates/networks/haproxy.md index 1d4ac217f..47008324c 100644 --- a/content/influxdb/v2.0/monitor-alert/templates/networks/haproxy.md +++ b/content/influxdb/v2.0/monitor-alert/templates/networks/haproxy.md @@ -29,9 +29,9 @@ The HAProxy for InfluxDB v2 template includes the following: ``` For more information, see [influx apply](/v2.0/reference/cli/influx/apply/). - > **Note:** Ensure your `influx` CLI is configured with your account credentials and that configuration is active. For more information, see [influx config](https://v2.docs.influxdata.com/v2.0/reference/cli/influx/config/). + > **Note:** Ensure your `influx` CLI is configured with your account credentials and that configuration is active. For more information, see [influx config](/influxdb/v2.0/reference/cli/influx/config/). -2. [Install Telegraf](/telegraf/latest/introduction/installation/) on a server with network access to both the HAProxy instances and [InfluxDB v2 API](/v2.0/reference/api/). +2. [Install Telegraf](/{{< latest "telegraf" >}}/introduction/installation/) on a server with network access to both the HAProxy instances and [InfluxDB v2 API](/v2.0/reference/api/). 3. In your [Telegraf configuration file (`telegraf.conf`)](/v2.0/write-data/no-code/use-telegraf/auto-config/view-telegraf-config/), do the following: - Set the following environment variables: - INFLUX_TOKEN: Token must have permissions to read Telegraf configurations and write data to the `haproxy` bucket. See how to [view tokens](/v2.0/security/tokens/view-tokens/). diff --git a/content/influxdb/v2.0/write-data/_index.md b/content/influxdb/v2.0/write-data/_index.md index 50d4c4f5c..967eff250 100644 --- a/content/influxdb/v2.0/write-data/_index.md +++ b/content/influxdb/v2.0/write-data/_index.md @@ -40,7 +40,7 @@ To write data into InfluxDB, you need the following: instructions on viewing your bucket ID._ - **authentication token** – _See [View tokens](/v2.0/security/tokens/view-tokens/) for instructions on viewing your authentication token._ -- **InfluxDB URL** – _See [InfluxDB URLs](https://v2.docs.influxdata.com/v2.0/reference/urls/)_. +- **InfluxDB URL** – _See [InfluxDB URLs](/influxdb/v2.0/reference/urls/)_. The [InfluxDB setup process](/v2.0/get-started/#set-up-influxdb) creates each of these.