diff --git a/content/v2.0/query-data/_index.md b/content/v2.0/query-data/_index.md new file mode 100644 index 000000000..6a0c716dd --- /dev/null +++ b/content/v2.0/query-data/_index.md @@ -0,0 +1,11 @@ +--- +title: Query data in InfluxDB +seotitle: Query data stored in InfluxDB +description: placeholder +menu: + v2_0: + name: Query data + weight: 2 +--- + +_Placeholder content for the query data page._ diff --git a/content/v2.0/query-data/flux/_index.md b/content/v2.0/query-data/flux/_index.md new file mode 100644 index 000000000..ad0ba78f0 --- /dev/null +++ b/content/v2.0/query-data/flux/_index.md @@ -0,0 +1,37 @@ +--- +title: Introduction to Flux +description: > + Flux is InfluxData's functional data scripting language designed for querying, + analyzing, and acting on data. +menu: + v2_0: + parent: Query data + name: Flux + weight: 1 +--- + +Flux is InfluxData's functional data scripting language designed for querying, analyzing, and acting on data. + +## Flux design principles +Flux is designed to be usable, readable, flexible, composable, testable, contributable, and shareable. +Its syntax is largely inspired by [2018's most popular scripting language](https://insights.stackoverflow.com/survey/2018#technology), +Javascript, and takes a functional approach to data exploration and processing. + +The following example illustrates querying data stored from the last five minutes, +filtering by the `cpu` measurement and the `cpu=cpu-usage` tag, windowing the data in 1 minute intervals, +and calculating the average of each window: + +```js +from(bucket:"example-bucket") + |> range(start:-1h) + |> filter(fn:(r) => + r._measurement == "cpu" and + r.cpu == "cpu-total" + ) + |> aggregateWindow(every: 1m, fn: mean) +``` + +## Get started with Flux +The best way to familiarize yourself with Flux is to walk through creating a simple Flux query. + +[Get Started with Flux](/v2.0/query-data/flux/get-started) diff --git a/content/v2.0/query-data/flux/get-started/_index.md b/content/v2.0/query-data/flux/get-started/_index.md new file mode 100644 index 000000000..b292174e0 --- /dev/null +++ b/content/v2.0/query-data/flux/get-started/_index.md @@ -0,0 +1,87 @@ +--- +title: Get started with Flux +description: > + Get started with Flux, InfluxData's functional data scripting language. + This step-by-step guide through the basics of writing a Flux query. +menu: + v2_0: + name: Get started + identifier: get-started + parent: Flux + weight: 2 +--- + +Flux is InfluxData's new functional data scripting language designed for querying, +analyzing, and acting on data. + +This multi-part getting started guide walks through important concepts related to Flux, +how to query time series data from InfluxDB using Flux, and introduces Flux syntax and functions. + +## Key concepts +Flux introduces important new concepts you should understand as you get started. + +### Pipe-forward operator +Flux uses pipe-forward operators (`|>`) extensively to chain operations together. +After each function or operation, Flux returns a table or collection of tables containing data. +The pipe-forward operator pipes those tables into the next function or operation where +they are further processed or manipulated. + +### Tables +Flux structures all data in tables. +When data is streamed from data sources, Flux formats it as annotated +comma-separated values (CSV), representing tables. +Functions then manipulate or process them and output new tables. + +#### Group keys +Every table has a **group key** which describes the contents of the table. +It's a list of columns for which every row in the table will have the same value. +Columns with unique values in each row are **not** part of the group key. + +As functions process and transform data, each modifies the group keys of output tables. +Understanding how tables and group keys are modified by functions is key to properly +shaping your data for the desired output. + +###### Example group key +```js +[_start, _stop, _field, _measurement, host] +``` + +Note that `_time` and `_value` are excluded from the example group key because they +are unique to each row. + +## Tools for working with Flux + +You have multiple [options for writing and running Flux queries](/flux/v0.12/guides/executing-queries), +but as you're getting started, we recommend using the following: + +### 1. Data Explorer +The InfluxDB user interface's (UI) Data Explorer makes it easy to build or write +your first Flux script and visualize the results. + +![Flux in the Data Explorer](/img/flux-data-explorer,png) + +The Data Explorer provides multiple ways to create Flux queries. +Toggle between the two with the button to the left of **Submit** in the Data Explorer. + +![Flux Query Builder and Script Editor Toggle](/img/flux-ui-toggle.png) + +#### Query Builder _(default)_ +The Query Builder is a visual tool for building Flux Queries. +Select the organization and bucket from which you would like to query data. +Filter data by any columns available in the data. +Transform you data using using aggregate functions. + +#### Script Editor +The Script Editor is an in-browser code editor where you can write raw Flux scripts. + +### 2. influx CLI +The [`influx repl` command](/v2.0/reference/cli/influx/repl) opens an interactive +read-eval-print-loop (REPL) for querying data within an organization in InfluxDB with Flux. + +```bash +influx repl --org org-name +``` + +
diff --git a/content/v2.0/query-data/flux/get-started/query-influxdb.md b/content/v2.0/query-data/flux/get-started/query-influxdb.md new file mode 100644 index 000000000..b02e515b6 --- /dev/null +++ b/content/v2.0/query-data/flux/get-started/query-influxdb.md @@ -0,0 +1,128 @@ +--- +title: Query InfluxDB with Flux +description: Learn the basics of using Flux to query data from InfluxDB. +menu: + v2_0: + name: Query InfluxDB + parent: get-started + weight: 1 +--- + +This guide walks through the basics of using Flux to query data from InfluxDB. +Every Flux query needs the following: + +1. [A data source](#1-define-your-data-source) +2. [A time range](#2-specify-a-time-range) +3. [Data filters](#3-filter-your-data) + + +## 1. Define your data source +Flux's [`from()`](#) function defines an InfluxDB data source. +It requires a [`bucket`](#) parameter. +The following examples use `example-bucket` as the bucket name. + +```js +from(bucket:"example-bucket") +``` + +## 2. Specify a time range +Flux requires a time range when querying time series data. +"Unbounded" queries are very resource-intensive and as a protective measure, +Flux will not query the database without a specified range. + +Use the pipe-forward operator (`|>`) to pipe data from your data source into the [`range()`](/flux/v0.12/functions/transformations/range) +function, which specifies a time range for your query. +It accepts two properties: `start` and `stop`. +Ranges can be **relative** using negative [durations](/flux/v0.12/language/lexical-elements#duration-literals) +or **absolute** using [timestamps](/flux/v0.12/language/lexical-elements#date-and-time-literals). + +###### Example relative time ranges +```js +// Relative time range with start only. Stop defaults to now. +from(bucket:"example-bucket") + |> range(start: -1h) + +// Relative time range with start and stop +from(bucket:"example-bucket") + |> range(start: -1h, stop: -10m) +``` + +> Relative ranges are relative to "now." + +###### Example absolute time range +```js +from(bucket:"example-bucket") + |> range(start: 2018-11-05T23:30:00Z, stop: 2018-11-06T00:00:00Z) +``` + +#### Use the following: +For this guide, use the relative time range, `-15m`, to limit query results to data from the last 15 minutes: + +```js +from(bucket:"example-bucket") + |> range(start: -15m) +``` + +## 3. Filter your data +Pass your ranged data into the `filter()` function to narrow results based on data attributes or columns. +The `filter()` function has one parameter, `fn`, which expects an anonymous function +with logic that filters data based on columns or attributes. + +Flux's anonymous function syntax is similar to Javascript's. +Records or rows are passed into the `filter()` function as an object (`r`). +The anonymous function takes the object and evaluates it to see if it matches the defined filters. +Use the `and` relational operator to chain multiple filters. + +```js +// Pattern +(r) => (r.objectProperty comparisonOperator comparisonExpression) + +// Example with single filter +(r) => (r._measurement == "cpu") + +// Example with multiple filters +(r) => (r._measurement == "cpu") and (r._field != "usage_system" ) +``` + +#### Use the following: +For this example, filter by the `cpu` measurement, the `usage_system` field, and the `cpu-total` tag value: + +```js +from(bucket:"example-bucket") + |> range(start: -15m) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_system" and + r.cpu == "cpu-total" + ) +``` + +## 4. Yield your queried data +Use Flux's `yield()` function to output the filtered tables as the result of the query. + +```js +from(bucket:"example-bucket") + |> range(start: -15m) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_system" and + r.cpu == "cpu-total" + ) + |> yield() +``` + +{{% note %}} +Flux automatically assume a `yield()` function at +the end of each script in order to output and visualize the data. +`yield()` is only necessary when including multiple queries in the same Flux query. +Each set of returned data needs to be named using the `yield()` function. +{{% /note %}} + +## Congratulations! +You have now queried data from InfluxDB using Flux. +This is a barebones query that can be transformed in other ways. + + diff --git a/content/v2.0/query-data/flux/get-started/syntax-basics.md b/content/v2.0/query-data/flux/get-started/syntax-basics.md new file mode 100644 index 000000000..0fd91cfcc --- /dev/null +++ b/content/v2.0/query-data/flux/get-started/syntax-basics.md @@ -0,0 +1,217 @@ +--- +title: Flux syntax basics +description: An introduction to the basic elements of the Flux syntax with real-world application examples. +menu: + v2_0: + name: Syntax basics + parent: get-started + weight: 3 +--- + + +Flux, at its core, is a scripting language designed specifically for working with data. +This guide walks through a handful of simple expressions and how they are handled in Flux. + +## Use the influx CLI's REPL +Use the `influx repl` command to open the interactive read-eval-print-loop (REPL). +Run the commands provided in this guide in the REPL. + +##### Start in the influx CLI in Flux mode +```bash +influx repl --org org-name +``` + +## Basic Flux syntax +The code blocks below provide commands that illustrate the basic syntax of Flux. +Run these commands in the REPL. + +### Simple expressions +Flux is a scripting language that supports basic expressions. +For example, simple addition: + +```js +> 1 + 1 +2 +``` + +### Variables +Assign an expression to a variable using the assignment operator, `=`. + +```js +> s = "this is a string" +> i = 1 // an integer +> f = 2.0 // a floating point number +``` + +Type the name of a variable to print its value: + +```js +> s +this is a string +> i +1 +> f +2 +``` + +### Objects +Flux also supports objects. Each value in an object can be a different data type. + +```js +> o = {name:"Jim", age: 42} +``` + +Use dot notation to access a properties of an object: + +```js +> o.name +Jim +> o.age +42 +``` + +### Lists +Flux supports lists. List values must be the same type. + +```js +> n = 4 +> l = [1,2,3,n] +> l +[1, 2, 3, 4] +``` + +### Functions +Flux uses functions for most of its heavy lifting. +Below is a simple function that squares a number, `n`. + +```js +> square = (n) => n * n +> square(n:3) +9 +``` + +{{% note %}} +Flux does not support positional arguments or parameters. +Parameters must always be named when calling a function. +{{% /note %}} + +### Pipe-forward operator +Flux uses the pipe-forward operator (`|>`) extensively to chain operations together. +After each function or operation, Flux returns a table or collection of tables containing data. +The pipe-forward operator pipes those tables into the next function where they are further processed or manipulated. + +```js +data |> someFunction() |> anotherFunction() +``` + +## Real-world application of basic syntax +This likely seems familiar if you've already been through through the other +[getting started guides](/v2.0/query-data/flux/get-started). +Flux's syntax is inspired by Javascript and other functional scripting languages. +As you begin to apply these basic principles in real-world use cases such as creating data stream variables, +custom functions, etc., the power of Flux and its ability to query and process data will become apparent. + +The examples below provide both multi-line and single-line versions of each input command. +Carriage returns in Flux aren't necessary, but do help with readability. +Both single- and multi-line commands can be copied and pasted into the `influx` CLI running in Flux mode. + +### Define data stream variables +A common use case for variable assignments in Flux is creating variables for one +or more input data streams. + +{{< code-tabs-wrapper >}} + {{% code-tabs %}} + [Multi-line](#) + [Single-line](#) + {{% /code-tabs %}} +{{% code-tab-content %}} +```js +timeRange = -1h + +cpuUsageUser = + from(bucket:"example-bucket") + |> range(start: timeRange) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_user" and + r.cpu == "cpu-total" + ) + +memUsagePercent = + from(bucket:"example-bucket") + |> range(start: timeRange) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field == "used_percent" + ) +``` +{{% /code-tab-content %}} + +{{% code-tab-content %}} +```js +timeRange = -1h +cpuUsageUser = from(bucket:"example-bucket") |> range(start: timeRange) |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_user" and r.cpu == "cpu-total") +memUsagePercent = from(bucket:"example-bucket") |> range(start: timeRange) |> filter(fn: (r) => r._measurement == "mem" and r._field == "used_percent") +``` +{{% /code-tab-content %}} +{{< /code-tabs-wrapper>}} + +These variables can be used in other functions, such as `join()`, while keeping the syntax minimal and flexible. + +### Define custom functions +Create a function that returns the `N` number rows in the input stream with the highest `_value`s. +To do this, pass the input stream (`tables`) and the number of results to return (`n`) into a custom function. +Then using Flux's `sort()` and `limit()` functions to find the top `n` results in the data set. + +{{< code-tabs-wrapper >}} + {{% code-tabs %}} + [Multi-line](#) + [Single-line](#) + {{% /code-tabs %}} +{{% code-tab-content %}} +```js +topN = (tables=<-, n) => + tables + |> sort(desc: true) + |> limit(n: n) +``` +{{% /code-tab-content %}} +{{% code-tab-content %}} +```js +topN = (tables=<-, n) => tables |> sort(desc: true) |> limit(n: n) +``` +{{% /code-tab-content %}} +{{< /code-tabs-wrapper >}} + +_More information about creating custom functions is available in the [Custom functions](/flux/v0.12/functions/custom-functions) documentation._ + +Using the `cpuUsageUser` data stream variable defined above, find the top five data +points with the custom `topN` function and yield the results. + +{{< code-tabs-wrapper >}} +{{% code-tabs %}} +[Multi-line](#) +[Single-line](#) +{{% /code-tabs %}} + +{{% code-tab-content %}} +```js +cpuUsageUser + |> topN(n:5) + |> yield() +``` +{{% /code-tab-content %}} + +{{% code-tab-content %}} +```js +cpuUsageUser |> topN(n:5) |> yield() +``` +{{% /code-tab-content %}} + +{{< /code-tabs-wrapper>}} + +This query will return the five data points with the highest user CPU usage over the last hour. + + diff --git a/content/v2.0/query-data/flux/get-started/transform-data.md b/content/v2.0/query-data/flux/get-started/transform-data.md new file mode 100644 index 000000000..68111916f --- /dev/null +++ b/content/v2.0/query-data/flux/get-started/transform-data.md @@ -0,0 +1,176 @@ +--- +title: Transform data with Flux +description: Learn the basics of using Flux to transform data queried from InfluxDB. +menu: + v2_0: + name: Transform your data + parent: get-started + weight: 2 +--- + +When [querying data from InfluxDB](/v2.0/query-data/flux/get-started/query-influxdb), +you often need to transform that data in some way. +Common examples are aggregating data into averages, downsampling data, etc. + +This guide demonstrates using [Flux functions](/flux/v0.12/functions) to transform your data. +It walks through creating a Flux script that partitions data into windows of time, +averages the `_value`s in each window, and outputs the averages as a new table. + +It's important to understand how the "shape" of your data changes through each of these operations. + +## Query data +Use the query built in the previous [Query data from InfluxDB](/v2.0/query-data/flux/get-started/query-influxdb) +guide, but update the range to pull data from the last hour: + +```js +from(bucket:"example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_system" and + r.cpu == "cpu-total" + ) +``` + +## Flux functions +Flux provides a number of functions that perform specific operations, transformations, and tasks. +You can also [create custom functions](/flux/v0.12/functions/custom-functions) in your Flux queries. +_Functions are covered in detail in the [Flux functions](/flux/v0.12/functions) documentation._ + +A common type of function used when transforming data queried from InfluxDB is an aggregate function. +Aggregate functions take a set of `_value`s in a table, aggregate them, and transform +them into a new value. + +This example uses the [`mean()` function](/flux/v0.12/functions/transformations/aggregates/mean) +to average values within each time window. + +{{% note %}} +The following example walks through the steps required to window and aggregate data, +but there is a [`aggregateWindow()` helper function](#helper-functions) that does it for you. +It's just good to understand the steps in the process. +{{% /note %}} + +## Window your data +Flux's [`window()` function](/flux/v0.12/functions/transformations/window) partitions records based on a time value. +Use the `every` parameter to define a duration of each window. + +For this example, window data in five minute intervals (`5m`). + +```js +from(bucket:"example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_system" and + r.cpu == "cpu-total" + ) + |> window(every: 5m) +``` + +As data is gathered into windows of time, each window is output as its own table. +When visualized, each table is assigned a unique color. + +![Windowed data tables](/img/flux-windowed-data.png) + +## Aggregate windowed data +Flux aggregate functions take the `_value`s in each table and aggregate them in some way. +Use the [`mean()` function](/flux/v0.12/functions/transformations/aggregates/mean) to average the `_value`s of each table. + +```js +from(bucket:"example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_system" and + r.cpu == "cpu-total" + ) + |> window(every: 5m) + |> mean() +``` + +As rows in each window are aggregated, their output table contains only a single row with the aggregate value. +Windowed tables are all still separate and, when visualized, will appear as single, unconnected points. + +![Windowed aggregate data](/img/flux-windowed-aggregates.png) + +## Add times to your aggregates +As values are aggregated, the resulting tables do not have a `_time` column because +the records used for the aggregation all have different timestamps. +Aggregate functions don't infer what time should be used for the aggregate value. +Therefore the `_time` column is dropped. + +A `_time` column is required in the [next operation](#unwindow-aggregate-tables). +To add one, use the [`duplicate()` function](/flux/v0.12/functions/transformations/duplicate) +to duplicate the `_stop` column as the `_time` column for each windowed table. + +```js +from(bucket:"example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_system" and + r.cpu == "cpu-total" + ) + |> window(every: 5m) + |> mean() + |> duplicate(column: "_stop", as: "_time") +``` + +## Unwindow aggregate tables + +Use the `window()` function with the `every: inf` parameter to gather all points +into a single, infinite window. + +```js +from(bucket:"example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_system" and + r.cpu == "cpu-total" + ) + |> window(every: 5m) + |> mean() + |> duplicate(column: "_stop", as: "_time") + |> window(every: inf) +``` + +Once ungrouped and combined into a single table, the aggregate data points will appear connected in your visualization. + +![Unwindowed aggregate data](/img/flux-windowed-aggregates-ungrouped.png) + +## Helper functions +This may seem like a lot of coding just to build a query that aggregates data, however going through the +process helps to understand how data changes "shape" as it is passed through each function. + +Flux provides (and allows you to create) "helper" functions that abstract many of these steps. +The same operation performed in this guide can be accomplished using the +[`aggregateWindow()` function](/flux/v0.12/functions/transformations/aggregates/aggregatewindow). + +```js +from(bucket:"example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_system" and + r.cpu == "cpu-total" + ) + |> aggregateWindow(every: 5m, fn: mean) +``` + +## Congratulations! +You have now constructed a Flux query that uses Flux functions to transform your data. +There are many more ways to manipulate your data using both Flux's primitive functions +and your own custom functions, but this is a good introduction into the basic syntax and query structure. + +--- + +_For a deeper dive into windowing and aggregating data with example data output for each transformation, +view the [Windowing and aggregating data](/flux/v0.12/guides/windowing-aggregating) guide._ + +--- + +