From ec38ef074d2d0077f1761b31dae156cd2ac10877 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Tue, 28 Apr 2020 11:19:55 -0600 Subject: [PATCH 1/7] guide for calculating percentages in flux, resolves #981 --- .../query-data/flux/calculate-percentages.md | 247 ++++++++++++++++++ .../query-data/flux/mathematic-operations.md | 1 + data/query_examples.yml | 32 +++ 3 files changed, 280 insertions(+) create mode 100644 content/v2.0/query-data/flux/calculate-percentages.md diff --git a/content/v2.0/query-data/flux/calculate-percentages.md b/content/v2.0/query-data/flux/calculate-percentages.md new file mode 100644 index 000000000..b04b0905b --- /dev/null +++ b/content/v2.0/query-data/flux/calculate-percentages.md @@ -0,0 +1,247 @@ +--- +title: Calculate percentages with Flux +list_title: Calculate percentages +description: > + Use [`pivot()` or `join()`](/v2.0/query-data/flux/calculate-percentages/#pivot-vs-join) + and the [`map()` function](/v2.0/reference/flux/stdlib/built-in/transformations/map/) + to align operand values into row-wise sets and calculate a percentage. +menu: + v2_0: + name: Calculate percentages + parent: Query with Flux +weight: 220 +aliases: + - /v2.0/query-data/guides/manipulate-timestamps/ +related: + - /v2.0/query-data/flux/mathematic-operations + - /v2.0/reference/flux/stdlib/built-in/transformations/map + - /v2.0/reference/flux/stdlib/built-in/transformations/pivot + - /v2.0/reference/flux/stdlib/built-in/transformations/join +list_query_example: percentages +--- + + +Calculating percentages using multiple values in a queried data set is a common use case for time series data. +With Flux, all operand values need to exists in a single row to use them in a mathematic calculation. +Once operands are aligned in rows, use `map()` to re-map values in the row and calculate a percentage. + +**To calculate percentages:** + +1. Query the necessary operand values. +2. Use [`pivot()` or `join()`](#pivot-vs-join) to align operand values into row-wise sets. +3. Use [`map()`](/v2.0/reference/flux/stdlib/built-in/transformations/map/) + to divide the numerator operand value by the denominator operand value and multiply by 100. + +{{< code-tabs-wrapper >}} +{{% code-tabs %}} +[pivot()](#) +[join()](#) +{{% /code-tabs %}} +{{% code-tab-content %}} +```js +from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "m1" and r._field =~ /field[1-2]/ ) + |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value") + |> map(fn: (r) => ({ r with _value: r.field1 / r.field2 * 100.0 })) +``` +{{% /code-tab-content %}} +{{% code-tab-content %}} +```js +t1 = from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "m1" and r._field == "field1") + +t2 = from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "m2" and r._field == "field2") + +join(tables: {t1: t1, t2: t2}, on: ["_time"]) + |> map(fn: (r) => ({ r with _value: r._value_t1 / r._value_t2 * 100.0 })) +``` +{{% /code-tab-content %}} +{{< /code-tabs-wrapper >}} + +{{% note %}} +This guide uses `pivot()` to align operand values into row-wise sets because +`pivot()` works for the majority of use cases and is more performant than `join()`. +_See [Pivot vs join](#pivot-vs-join) below._ +{{% /note %}} + +## GPU monitoring example +The following example queries data collected from a GPU monitoring solution and +calculates the percentage of GPU memory used over time. +Data includes the following: + +- **`gpu` measurement** +- **`mem_used` field**: used GPU memory in bytes +- **`mem_total` field**: total GPU memory in bytes + +### Query mem_used and mem_total fields +```js +from(bucket: "gpu-monitor") + |> range(start: 2020-01-01T00:00:00Z) + |> filter(fn: (r) => r._measurement == "gpu" and r._field =~ /mem_/) +``` + +###### Returns the following stream of tables: + +| _time | _measurement | _field | _value | +|:----- |:------------:|:------: | ------: | +| 2020-01-01T00:00:00Z | gpu | mem_used | 2517924577 | +| 2020-01-01T00:00:10Z | gpu | mem_used | 2695091978 | +| 2020-01-01T00:00:20Z | gpu | mem_used | 2576980377 | +| 2020-01-01T00:00:30Z | gpu | mem_used | 3006477107 | +| 2020-01-01T00:00:40Z | gpu | mem_used | 3543348019 | +| 2020-01-01T00:00:50Z | gpu | mem_used | 4402341478 | + +

+ +| _time | _measurement | _field | _value | +|:----- |:------------:|:------: | ------: | +| 2020-01-01T00:00:00Z | gpu | mem_total | 8589934592 | +| 2020-01-01T00:00:10Z | gpu | mem_total | 8589934592 | +| 2020-01-01T00:00:20Z | gpu | mem_total | 8589934592 | +| 2020-01-01T00:00:30Z | gpu | mem_total | 8589934592 | +| 2020-01-01T00:00:40Z | gpu | mem_total | 8589934592 | +| 2020-01-01T00:00:50Z | gpu | mem_total | 8589934592 | + +### Pivot fields into columns +Use `pivot()` to pivot the `mem_used` and `mem_total` fields into columns. +Output includes `mem_used` and `mem_total` columns with values for each corresponding `_time`. + +```js +// ... + |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value") +``` + +###### Returns the following: + +| _time | _measurement | mem_used | mem_total | +|:----- |:------------:| --------: | ---------: | +| 2020-01-01T00:00:00Z | gpu | 2517924577 | 8589934592 | +| 2020-01-01T00:00:10Z | gpu | 2695091978 | 8589934592 | +| 2020-01-01T00:00:20Z | gpu | 2576980377 | 8589934592 | +| 2020-01-01T00:00:30Z | gpu | 3006477107 | 8589934592 | +| 2020-01-01T00:00:40Z | gpu | 3543348019 | 8589934592 | +| 2020-01-01T00:00:50Z | gpu | 4402341478 | 8589934592 | + +### Map new values +With fields pivoted into columns, each row contains the values necessary to calculate a percentage. +Use `map()` to re-map values in each row. +Divide `mem_used` by `mem_total` and multiply by 100 to return the percentage. + +{{% note %}} +To return a precise float percentage value that includes decimal points the example +below casts integer field values to floats and multiplies by a float value (`100.0`). +{{% /note %}} + +```js +// ... + |> map(fn: (r) => ({ + _time: r._time, + _measurement: r._measurement, + _field: "mem_used_percent", + _value: float(v: r.mem_used) / float(v: r.mem_total) * 100.0 + })) +``` + +| _time | _measurement | _field | _value | +|:----- |:------------:|:------: | ------: | +| 2020-01-01T00:00:00Z | gpu | mem_used_percent | 29.31 | +| 2020-01-01T00:00:10Z | gpu | mem_used_percent | 31.37 | +| 2020-01-01T00:00:20Z | gpu | mem_used_percent | 30.00 | +| 2020-01-01T00:00:30Z | gpu | mem_used_percent | 35.00 | +| 2020-01-01T00:00:40Z | gpu | mem_used_percent | 41.25 | +| 2020-01-01T00:00:50Z | gpu | mem_used_percent | 51.25 | + +### Full query +```js +from(bucket: "gpu-monitor") + |> range(start: 2020-01-01T00:00:00Z) + |> filter(fn: (r) => r._measurement == "gpu" and r._field =~ /mem_/ ) + |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value") + |> map(fn: (r) => ({ + _time: r._time, + _measurement: r._measurement, + _field: "mem_used_percent", + _value: float(v: r.mem_used) / float(v: r.mem_total) * 100.0 + })) +``` + +--- + +## Pivot vs join +To use values in mathematical operations in Flux, operand values must exists in a single row. +Both `pivot()` and `join()` will do this, but there are important differences between the two: + +#### Pivot is more performant +`pivot()` only has to read and operate on a single stream of data. +`join()` requires two streams of data and the overhead of reading and combing both +streams can be high, especially for larger data sets. + +#### Use join for multiple data sources +`join()` is really only necessary when querying data from multiple buckets or +different data sources. + +--- + +## Examples + +#### Use pivot() to align multiple fields +```js +from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "example-measurement") + |> filter(fn: (r) => + r._field == "used_system" or + r._field == "used_user" or + r._field == "total" + ) + |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value") + |> map(fn: (r) => ({ r with + _value: float(v: r.used_system + r.used_user) / float(v: r.total) * 100.0 + })) +``` + +#### Use pivot() for math across measurements + +1. Ensure measurements are in the same [bucket](/v2.0/reference/glossary/#bucket). +2. Use `filter()` to include data from both measurements. +3. Use `group()` to ungroup data to return a single table. +4. Use `pivot()` to pivot fields into row-wise sets. +5. Use `map()` to re-map rows and perform the percentage calculation. + + +```js +from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => + (r._measurement == "m1" or r._measurement == "m2") and + (r._field == "field1" or r._field == "field2") + ) + |> group() + |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value") + |> map(fn: (r) => ({ r with _value: r.field1 / r.field2 * 100.0 })) +``` + +#### Use join for multiple data sources +```js +import "sql" + +t1 = sql.from( + driverName: "postgres", + dataSourceName: "postgresql://user:password@localhost", + query:"SELECT * FROM exampleTable" +) + +t2 = from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "example-measurement" and + r._field == "example-field" + ) + +join(tables: {t1: t1, t2: t2}, on: ["id"]) + |> map(fn: (r) => ({ r with _value: r._value_t2 / r.available_t1 * 100.0 })) +``` diff --git a/content/v2.0/query-data/flux/mathematic-operations.md b/content/v2.0/query-data/flux/mathematic-operations.md index c297ef31a..a4b94810b 100644 --- a/content/v2.0/query-data/flux/mathematic-operations.md +++ b/content/v2.0/query-data/flux/mathematic-operations.md @@ -18,6 +18,7 @@ related: - /v2.0/reference/flux/stdlib/built-in/transformations/aggregates/reduce/ - /v2.0/reference/flux/language/operators/ - /v2.0/reference/flux/stdlib/built-in/transformations/type-conversions/ + - /v2.0/query-data/flux/calculate-percentages/ list_query_example: map_math --- diff --git a/data/query_examples.yml b/data/query_examples.yml index 17587f9e0..b9f89c6e6 100644 --- a/data/query_examples.yml +++ b/data/query_examples.yml @@ -332,6 +332,38 @@ moving_average: | 2020-01-01T00:06:00Z | 1.325 | | 2020-01-01T00:06:00Z | 1.150 | +percentages: + - + code: | + ```js + data + |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value") + |> map(fn: (r) => ({ + _time: r._time, + _field: "used_percent", + _value: float(v: r.used) / float(v: r.total) * 100.0 + })) + ``` + input: | + | _time | _field | _value | + |:----- |:------: | ------:| + | 2020-01-01T00:00:00Z | used | 2.5 | + | 2020-01-01T00:00:10Z | used | 3.1 | + | 2020-01-01T00:00:20Z | used | 4.2 | + + | _time | _field | _value | + |:----- |:------: | ------:| + | 2020-01-01T00:00:00Z | total | 8.0 | + | 2020-01-01T00:00:10Z | total | 8.0 | + | 2020-01-01T00:00:20Z | total | 8.0 | + output: | + | _time | _field | _value | + |:----- |:------: | ------:| + | 2020-01-01T00:00:00Z | used_percent | 31.25 | + | 2020-01-01T00:00:10Z | used_percent | 38.75 | + | 2020-01-01T00:00:20Z | used_percent | 52.50 | + + quantile: - code: | From e21188ab080161f4c58f622311cb5bca77879f77 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Tue, 28 Apr 2020 13:45:01 -0600 Subject: [PATCH 2/7] updated percentage sql query with secrets and limited columns --- content/v2.0/query-data/flux/calculate-percentages.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/content/v2.0/query-data/flux/calculate-percentages.md b/content/v2.0/query-data/flux/calculate-percentages.md index b04b0905b..0380fda46 100644 --- a/content/v2.0/query-data/flux/calculate-percentages.md +++ b/content/v2.0/query-data/flux/calculate-percentages.md @@ -228,11 +228,16 @@ from(bucket: "example-bucket") #### Use join for multiple data sources ```js import "sql" +import "influxdata/influxdb/secrets" + +pgUser = secrets.get(key: "POSTGRES_USER") +pgPass = secrets.get(key: "POSTGRES_PASSWORD") +pgHost = secrets.get(key: "POSTGRES_HOST") t1 = sql.from( driverName: "postgres", - dataSourceName: "postgresql://user:password@localhost", - query:"SELECT * FROM exampleTable" + dataSourceName: "postgresql://${pgUser}:${pgPass}@${pgHost}", + query:"SELECT id, name, available FROM exampleTable" ) t2 = from(bucket: "example-bucket") From 6e6bf273b2b1600b0d2f425296eb4c36796138eb Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Tue, 28 Apr 2020 13:52:18 -0600 Subject: [PATCH 3/7] typo fixes in percentage guide --- content/v2.0/query-data/flux/calculate-percentages.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/content/v2.0/query-data/flux/calculate-percentages.md b/content/v2.0/query-data/flux/calculate-percentages.md index 0380fda46..12feb43c0 100644 --- a/content/v2.0/query-data/flux/calculate-percentages.md +++ b/content/v2.0/query-data/flux/calculate-percentages.md @@ -132,7 +132,7 @@ Use `map()` to re-map values in each row. Divide `mem_used` by `mem_total` and multiply by 100 to return the percentage. {{% note %}} -To return a precise float percentage value that includes decimal points the example +To return a precise float percentage value that includes decimal points, the example below casts integer field values to floats and multiplies by a float value (`100.0`). {{% /note %}} @@ -177,8 +177,8 @@ Both `pivot()` and `join()` will do this, but there are important differences be #### Pivot is more performant `pivot()` only has to read and operate on a single stream of data. -`join()` requires two streams of data and the overhead of reading and combing both -streams can be high, especially for larger data sets. +`join()` requires two streams of data and the overhead of reading and combining +both streams can be significant, especially for larger data sets. #### Use join for multiple data sources `join()` is really only necessary when querying data from multiple buckets or @@ -208,8 +208,8 @@ from(bucket: "example-bucket") 1. Ensure measurements are in the same [bucket](/v2.0/reference/glossary/#bucket). 2. Use `filter()` to include data from both measurements. -3. Use `group()` to ungroup data to return a single table. -4. Use `pivot()` to pivot fields into row-wise sets. +3. Use `group()` to ungroup data and return a single table. +4. Use `pivot()` to pivot fields into columns. 5. Use `map()` to re-map rows and perform the percentage calculation. From df4702d797469cbf07ed14163ad2caa16b3abd0d Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Wed, 29 Apr 2020 08:43:15 -0600 Subject: [PATCH 4/7] updated percentages guide to address @kelseiv PR feedback --- assets/styles/layouts/article/_lists.scss | 41 ++++++---- .../query-data/flux/calculate-percentages.md | 79 +++++-------------- .../query-data/flux/mathematic-operations.md | 49 ++++++++++++ 3 files changed, 96 insertions(+), 73 deletions(-) diff --git a/assets/styles/layouts/article/_lists.scss b/assets/styles/layouts/article/_lists.scss index db5b67e83..c9e88b9d8 100644 --- a/assets/styles/layouts/article/_lists.scss +++ b/assets/styles/layouts/article/_lists.scss @@ -13,21 +13,32 @@ ul { } ol { -list-style: none; -counter-reset: item; - li { - position: relative; - counter-increment: item; - &:before { - content: counter(item) ". "; - position: absolute; - left: -1.6em; - color: $article-bold; - font-weight: bold; - } - ul { - counter-reset: item; - } + list-style-type: none; + counter-reset: item; + margin: 0; + padding: 0; +} + +ol > li { + display: table; + counter-increment: item; + margin-bottom: 0.6em; + + &:before { + content: counters(item, ".") ". "; + display: table-cell; + padding-right: 0.6em; + letter-spacing: .05rem; + color: $article-bold; + font-weight: bold; + } +} + +li ol > li { + margin: .5rem 0; + + &:before { + content: counters(item, ".") "."; } } diff --git a/content/v2.0/query-data/flux/calculate-percentages.md b/content/v2.0/query-data/flux/calculate-percentages.md index 12feb43c0..2b2233a43 100644 --- a/content/v2.0/query-data/flux/calculate-percentages.md +++ b/content/v2.0/query-data/flux/calculate-percentages.md @@ -4,7 +4,7 @@ list_title: Calculate percentages description: > Use [`pivot()` or `join()`](/v2.0/query-data/flux/calculate-percentages/#pivot-vs-join) and the [`map()` function](/v2.0/reference/flux/stdlib/built-in/transformations/map/) - to align operand values into row-wise sets and calculate a percentage. + to align operand values into rows and calculate a percentage. menu: v2_0: name: Calculate percentages @@ -21,23 +21,25 @@ list_query_example: percentages --- -Calculating percentages using multiple values in a queried data set is a common use case for time series data. -With Flux, all operand values need to exists in a single row to use them in a mathematic calculation. -Once operands are aligned in rows, use `map()` to re-map values in the row and calculate a percentage. +Calculating percentages from queried data is a common use case for time series data. +To calculate a percentage in Flux, all operands must be in one row. +Then, use `map()` to re-map values in the row and calculate a percentage. -**To calculate percentages:** +**To calculate percentages** -1. Query the necessary operand values. -2. Use [`pivot()` or `join()`](#pivot-vs-join) to align operand values into row-wise sets. +1. Use [`from()`](/v2.0/reference/flux/stdlib/built-in/inputs/from/), + [`range()`](/v2.0/reference/flux/stdlib/built-in/transformations/range/) and + [`filter()`](/v2.0/reference/flux/stdlib/built-in/transformations/filter/) to query operands. +2. Use [`pivot()` or `join()`](#pivot-vs-join) to align operand values into rows. 3. Use [`map()`](/v2.0/reference/flux/stdlib/built-in/transformations/map/) to divide the numerator operand value by the denominator operand value and multiply by 100. -{{< code-tabs-wrapper >}} -{{% code-tabs %}} -[pivot()](#) -[join()](#) -{{% /code-tabs %}} -{{% code-tab-content %}} +{{% note %}} +In the following examples use `pivot()` to align operands into rows because +`pivot()` works in most cases and is more performant than `join()`. +_See [Pivot vs join](/v2.0/query-data/flux/mathematic-operations/#pivot-vs-join)._ +{{% /note %}} + ```js from(bucket: "example-bucket") |> range(start: -1h) @@ -45,32 +47,10 @@ from(bucket: "example-bucket") |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value") |> map(fn: (r) => ({ r with _value: r.field1 / r.field2 * 100.0 })) ``` -{{% /code-tab-content %}} -{{% code-tab-content %}} -```js -t1 = from(bucket: "example-bucket") - |> range(start: -1h) - |> filter(fn: (r) => r._measurement == "m1" and r._field == "field1") - -t2 = from(bucket: "example-bucket") - |> range(start: -1h) - |> filter(fn: (r) => r._measurement == "m2" and r._field == "field2") - -join(tables: {t1: t1, t2: t2}, on: ["_time"]) - |> map(fn: (r) => ({ r with _value: r._value_t1 / r._value_t2 * 100.0 })) -``` -{{% /code-tab-content %}} -{{< /code-tabs-wrapper >}} - -{{% note %}} -This guide uses `pivot()` to align operand values into row-wise sets because -`pivot()` works for the majority of use cases and is more performant than `join()`. -_See [Pivot vs join](#pivot-vs-join) below._ -{{% /note %}} ## GPU monitoring example -The following example queries data collected from a GPU monitoring solution and -calculates the percentage of GPU memory used over time. +The following example queries data from the gpu-monitor bucket and calculates the +percentage of GPU memory used over time. Data includes the following: - **`gpu` measurement** @@ -127,7 +107,7 @@ Output includes `mem_used` and `mem_total` columns with values for each correspo | 2020-01-01T00:00:50Z | gpu | 4402341478 | 8589934592 | ### Map new values -With fields pivoted into columns, each row contains the values necessary to calculate a percentage. +Each row now contains the values necessary to calculate a percentage. Use `map()` to re-map values in each row. Divide `mem_used` by `mem_total` and multiply by 100 to return the percentage. @@ -169,26 +149,9 @@ from(bucket: "gpu-monitor") })) ``` ---- - -## Pivot vs join -To use values in mathematical operations in Flux, operand values must exists in a single row. -Both `pivot()` and `join()` will do this, but there are important differences between the two: - -#### Pivot is more performant -`pivot()` only has to read and operate on a single stream of data. -`join()` requires two streams of data and the overhead of reading and combining -both streams can be significant, especially for larger data sets. - -#### Use join for multiple data sources -`join()` is really only necessary when querying data from multiple buckets or -different data sources. - ---- - ## Examples -#### Use pivot() to align multiple fields +#### Calculate percentages using multiple fields ```js from(bucket: "example-bucket") |> range(start: -1h) @@ -204,7 +167,7 @@ from(bucket: "example-bucket") })) ``` -#### Use pivot() for math across measurements +#### Calculate percentages using multiple measurements 1. Ensure measurements are in the same [bucket](/v2.0/reference/glossary/#bucket). 2. Use `filter()` to include data from both measurements. @@ -225,7 +188,7 @@ from(bucket: "example-bucket") |> map(fn: (r) => ({ r with _value: r.field1 / r.field2 * 100.0 })) ``` -#### Use join for multiple data sources +#### Calculate percentages using multiple data sources ```js import "sql" import "influxdata/influxdb/secrets" diff --git a/content/v2.0/query-data/flux/mathematic-operations.md b/content/v2.0/query-data/flux/mathematic-operations.md index a4b94810b..65c2bb36a 100644 --- a/content/v2.0/query-data/flux/mathematic-operations.md +++ b/content/v2.0/query-data/flux/mathematic-operations.md @@ -99,6 +99,7 @@ percent(sample: 20.0, total: 80.0) To transform multiple values in an input stream, your function needs to: - [Handle piped-forward data](/v2.0/query-data/flux/custom-functions/#functions-that-manipulate-piped-forward-data). +- Each operand necessary for the calculation exists in each row _(see [Pivot vs join](#pivot-vs-join) below)_. - Use the [`map()` function](/v2.0/reference/flux/stdlib/built-in/transformations/map) to iterate over each row. The example `multiplyByX()` function below includes: @@ -269,3 +270,51 @@ from(bucket: "example-bucket") ``` {{% /code-tab-content %}} {{< /code-tabs-wrapper >}} + +## Pivot vs join +To query and use values in mathematical operations in Flux, operand values must +exists in a single row. +Both `pivot()` and `join()` will do this, but there are important differences between the two: + +#### Pivot is more performant +`pivot()` reads and operates on a single stream of data. +`join()` requires two streams of data and the overhead of reading and combining +both streams can be significant, especially for larger data sets. + +#### Use join for multiple data sources +Use `join()` when querying data from different buckets or data sources. + +##### Pivot fields into columns for mathematic calculations +```js +data + |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value") + |> map(fn: (r) => ({ r with + _value: (r.field1 + r.field2) / r.field3 * 100.0 + })) +``` + +##### Join multiple data sources for mathematic calculations +```js +import "sql" +import "influxdata/influxdb/secrets" + +pgUser = secrets.get(key: "POSTGRES_USER") +pgPass = secrets.get(key: "POSTGRES_PASSWORD") +pgHost = secrets.get(key: "POSTGRES_HOST") + +t1 = sql.from( + driverName: "postgres", + dataSourceName: "postgresql://${pgUser}:${pgPass}@${pgHost}", + query:"SELECT id, name, available FROM exampleTable" +) + +t2 = from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "example-measurement" and + r._field == "example-field" + ) + +join(tables: {t1: t1, t2: t2}, on: ["id"]) + |> map(fn: (r) => ({ r with _value: r._value_t2 / r.available_t1 * 100.0 })) +``` From 7fcb8e099bc08768b7ada66ef4a0f4f2955f2bda Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Wed, 29 Apr 2020 08:45:04 -0600 Subject: [PATCH 5/7] minor typo fix in percentage guide --- content/v2.0/query-data/flux/calculate-percentages.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/v2.0/query-data/flux/calculate-percentages.md b/content/v2.0/query-data/flux/calculate-percentages.md index 2b2233a43..bcb00552e 100644 --- a/content/v2.0/query-data/flux/calculate-percentages.md +++ b/content/v2.0/query-data/flux/calculate-percentages.md @@ -35,7 +35,7 @@ Then, use `map()` to re-map values in the row and calculate a percentage. to divide the numerator operand value by the denominator operand value and multiply by 100. {{% note %}} -In the following examples use `pivot()` to align operands into rows because +The following examples use `pivot()` to align operands into rows because `pivot()` works in most cases and is more performant than `join()`. _See [Pivot vs join](/v2.0/query-data/flux/mathematic-operations/#pivot-vs-join)._ {{% /note %}} From 9795ec2ddbb5a60faf1df76c201e8da5e386ddd1 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Wed, 29 Apr 2020 08:52:04 -0600 Subject: [PATCH 6/7] fixed links and adjusted sort order of percentages guide --- content/v2.0/query-data/flux/calculate-percentages.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/v2.0/query-data/flux/calculate-percentages.md b/content/v2.0/query-data/flux/calculate-percentages.md index bcb00552e..4f0af0e5a 100644 --- a/content/v2.0/query-data/flux/calculate-percentages.md +++ b/content/v2.0/query-data/flux/calculate-percentages.md @@ -2,14 +2,14 @@ title: Calculate percentages with Flux list_title: Calculate percentages description: > - Use [`pivot()` or `join()`](/v2.0/query-data/flux/calculate-percentages/#pivot-vs-join) + Use [`pivot()` or `join()`](/v2.0/query-data/flux/mathematic-operations/#pivot-vs-join) and the [`map()` function](/v2.0/reference/flux/stdlib/built-in/transformations/map/) to align operand values into rows and calculate a percentage. menu: v2_0: name: Calculate percentages parent: Query with Flux -weight: 220 +weight: 206 aliases: - /v2.0/query-data/guides/manipulate-timestamps/ related: @@ -20,7 +20,6 @@ related: list_query_example: percentages --- - Calculating percentages from queried data is a common use case for time series data. To calculate a percentage in Flux, all operands must be in one row. Then, use `map()` to re-map values in the row and calculate a percentage. @@ -30,7 +29,8 @@ Then, use `map()` to re-map values in the row and calculate a percentage. 1. Use [`from()`](/v2.0/reference/flux/stdlib/built-in/inputs/from/), [`range()`](/v2.0/reference/flux/stdlib/built-in/transformations/range/) and [`filter()`](/v2.0/reference/flux/stdlib/built-in/transformations/filter/) to query operands. -2. Use [`pivot()` or `join()`](#pivot-vs-join) to align operand values into rows. +2. Use [`pivot()` or `join()`](/v2.0/query-data/flux/mathematic-operations/#pivot-vs-join) + to align operand values into rows. 3. Use [`map()`](/v2.0/reference/flux/stdlib/built-in/transformations/map/) to divide the numerator operand value by the denominator operand value and multiply by 100. From 4d1a817a497e2849e14f42db3dd849030e036f94 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Wed, 29 Apr 2020 11:29:07 -0600 Subject: [PATCH 7/7] minor changes to math and percentage query guides --- .../query-data/flux/calculate-percentages.md | 5 +- .../query-data/flux/mathematic-operations.md | 86 +------------------ 2 files changed, 4 insertions(+), 87 deletions(-) diff --git a/content/v2.0/query-data/flux/calculate-percentages.md b/content/v2.0/query-data/flux/calculate-percentages.md index 4f0af0e5a..9b594ea0e 100644 --- a/content/v2.0/query-data/flux/calculate-percentages.md +++ b/content/v2.0/query-data/flux/calculate-percentages.md @@ -21,8 +21,8 @@ list_query_example: percentages --- Calculating percentages from queried data is a common use case for time series data. -To calculate a percentage in Flux, all operands must be in one row. -Then, use `map()` to re-map values in the row and calculate a percentage. +To calculate a percentage in Flux, operands must be in each row. +Use `map()` to re-map values in the row and calculate a percentage. **To calculate percentages** @@ -125,6 +125,7 @@ below casts integer field values to floats and multiplies by a float value (`100 _value: float(v: r.mem_used) / float(v: r.mem_total) * 100.0 })) ``` +##### Query results: | _time | _measurement | _field | _value | |:----- |:------------:|:------: | ------: | diff --git a/content/v2.0/query-data/flux/mathematic-operations.md b/content/v2.0/query-data/flux/mathematic-operations.md index 65c2bb36a..284657756 100644 --- a/content/v2.0/query-data/flux/mathematic-operations.md +++ b/content/v2.0/query-data/flux/mathematic-operations.md @@ -180,96 +180,12 @@ bytesToGB = (tables=<-) => ### Calculate a percentage To calculate a percentage, use simple division, then multiply the result by 100. -{{% note %}} -Operands in percentage calculations should always be floats. -{{% /note %}} - ```js > 1.0 / 4.0 * 100.0 25.0 ``` -#### User vs system CPU usage -The example below calculates the percentage of total CPU used by the `user` vs the `system`. - -{{< code-tabs-wrapper >}} -{{% code-tabs %}} -[Comments](#) -[No Comments](#) -{{% /code-tabs %}} - -{{% code-tab-content %}} -```js -// Custom function that converts usage_user and -// usage_system columns to floats -usageToFloat = (tables=<-) => - tables - |> map(fn: (r) => ({ - _time: r._time, - usage_user: float(v: r.usage_user), - usage_system: float(v: r.usage_system) - }) - ) - -// Define the data source and filter user and system CPU usage -// from 'cpu-total' in the 'cpu' measurement -from(bucket: "example-bucket") - |> range(start: -1h) - |> filter(fn: (r) => - r._measurement == "cpu" and - r._field == "usage_user" or - r._field == "usage_system" and - r.cpu == "cpu-total" - ) - - // Pivot the output tables so usage_user and usage_system are in each row - |> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value") - - // Convert usage_user and usage_system to floats - |> usageToFloat() - - // Map over each row and calculate the percentage of - // CPU used by the user vs the system - |> map(fn: (r) => ({ - // Preserve existing columns in each row - r with - usage_user: r.usage_user / (r.usage_user + r.usage_system) * 100.0, - usage_system: r.usage_system / (r.usage_user + r.usage_system) * 100.0 - }) - ) -``` -{{% /code-tab-content %}} - -{{% code-tab-content %}} -```js -usageToFloat = (tables=<-) => - tables - |> map(fn: (r) => ({ - _time: r._time, - usage_user: float(v: r.usage_user), - usage_system: float(v: r.usage_system) - }) - ) - -from(bucket: "example-bucket") - |> range(start: timeRangeStart, stop: timeRangeStop) - |> filter(fn: (r) => - r._measurement == "cpu" and - r._field == "usage_user" or - r._field == "usage_system" and - r.cpu == "cpu-total" - ) - |> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value") - |> usageToFloat() - |> map(fn: (r) => ({ - r with - usage_user: r.usage_user / (r.usage_user + r.usage_system) * 100.0, - usage_system: r.usage_system / (r.usage_user + r.usage_system) * 100.0 - }) - ) -``` -{{% /code-tab-content %}} -{{< /code-tabs-wrapper >}} +_For an in-depth look at calculating percentages, see [Calculate percentates](/v2.0/query-data/flux/calculate-percentages)._ ## Pivot vs join To query and use values in mathematical operations in Flux, operand values must