From 404c7b97d6ababc622d63e13009623492daf6cd9 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Tue, 15 Jan 2019 11:30:22 -0700 Subject: [PATCH 01/11] created structure and outlines for task docs --- content/v2.0/process-data/_index.md | 20 +++++ .../v2.0/process-data/common-tasks/_index.md | 10 +++ .../common-tasks/downsample-data.md | 16 ++++ .../v2.0/process-data/manage-tasks/_index.md | 19 +++++ .../process-data/manage-tasks/create-task.md | 22 ++++++ .../process-data/manage-tasks/delete-task.md | 21 +++++ .../process-data/manage-tasks/update-task.md | 21 +++++ .../process-data/manage-tasks/view-tasks.md | 21 +++++ content/v2.0/process-data/write-a-task.md | 78 +++++++++++++++++++ 9 files changed, 228 insertions(+) create mode 100644 content/v2.0/process-data/_index.md create mode 100644 content/v2.0/process-data/common-tasks/_index.md create mode 100644 content/v2.0/process-data/common-tasks/downsample-data.md create mode 100644 content/v2.0/process-data/manage-tasks/_index.md create mode 100644 content/v2.0/process-data/manage-tasks/create-task.md create mode 100644 content/v2.0/process-data/manage-tasks/delete-task.md create mode 100644 content/v2.0/process-data/manage-tasks/update-task.md create mode 100644 content/v2.0/process-data/manage-tasks/view-tasks.md create mode 100644 content/v2.0/process-data/write-a-task.md diff --git a/content/v2.0/process-data/_index.md b/content/v2.0/process-data/_index.md new file mode 100644 index 000000000..4b4c94483 --- /dev/null +++ b/content/v2.0/process-data/_index.md @@ -0,0 +1,20 @@ +--- +title: Process Data with InfluxDB tasks +seotitle: Process Data with InfluxDB tasks +description: placeholder +menu: + v2_0: + name: Process data + weight: 3 +--- + +InfluxDB's _**task engine**_ is designed for processing and analyzing data. +Tasks are user defined Flux scripts that take a stream of input data, modify or +analyze it in some way, then perform an action all on a specified schedule. +Examples include data downsampling, anomaly detection _(Coming)_, alerting _(Coming)_, etc. + +The following articles explain how to configure and build tasks using the InfluxDB user interface (UI) +and via raw Flux scripts with the `influx` command line interface (CLI). +They also provide examples of commonly used tasks. + +_Links for nested docs._ diff --git a/content/v2.0/process-data/common-tasks/_index.md b/content/v2.0/process-data/common-tasks/_index.md new file mode 100644 index 000000000..7b0455dff --- /dev/null +++ b/content/v2.0/process-data/common-tasks/_index.md @@ -0,0 +1,10 @@ +--- +title: Common data processing tasks +seotitle: Common data processing tasks performed with with InfluxDB +description: placeholder +menu: + v2_0: + name: Common tasks + parent: Process data + weight: 4 +--- diff --git a/content/v2.0/process-data/common-tasks/downsample-data.md b/content/v2.0/process-data/common-tasks/downsample-data.md new file mode 100644 index 000000000..5eafaabc5 --- /dev/null +++ b/content/v2.0/process-data/common-tasks/downsample-data.md @@ -0,0 +1,16 @@ +--- +title: Downsample data +seotitle: Downsample data in an InfluxDB task +description: placeholder +menu: + v2_0: + name: Downsample data + parent: Common tasks + weight: 4 +--- + +**Requirements:** + +- Data source +- Some type of aggregation +- and a `to` statement diff --git a/content/v2.0/process-data/manage-tasks/_index.md b/content/v2.0/process-data/manage-tasks/_index.md new file mode 100644 index 000000000..197c1244a --- /dev/null +++ b/content/v2.0/process-data/manage-tasks/_index.md @@ -0,0 +1,19 @@ +--- +title: Manage tasks in InfluxDB +seotitle: Manage data processing tasks in InfluxDB +description: placeholder +menu: + v2_0: + name: Manage tasks + parent: Process data + weight: 2 +--- + +InfluxDB provides two options for managing the creation, reading, updating, and deletion (CRUD) of tasks - +through the InfluxDB user interface (UI) or using the `influx` command line interface (CLI). +Both tools can perform all task CRUD operations. + +[Create a task](/v2.0/process-data/manage-tasks/create-task) +[View tasks](/v2.0/process-data/manage-tasks/view-tasks) +[Update a task](/v2.0/process-data/manage-tasks/update-task) +[Delete a task](/v2.0/process-data/manage-tasks/delete-task) diff --git a/content/v2.0/process-data/manage-tasks/create-task.md b/content/v2.0/process-data/manage-tasks/create-task.md new file mode 100644 index 000000000..f578415e9 --- /dev/null +++ b/content/v2.0/process-data/manage-tasks/create-task.md @@ -0,0 +1,22 @@ +--- +title: Create a task +seotitle: Create a task for processing data in InfluxDB +description: > + How to create a task that processes data in InfluxDB using the InfluxDB user + interface or the 'influx' command line interface. +menu: + v2_0: + name: Create a task + parent: Manage tasks + weight: 1 +--- + + +## Create a task in the InfluxDB UI + + +## Create a task with the `influx` CLI + +```sh +influx task create /path/to/task-script.flux +``` diff --git a/content/v2.0/process-data/manage-tasks/delete-task.md b/content/v2.0/process-data/manage-tasks/delete-task.md new file mode 100644 index 000000000..843c10243 --- /dev/null +++ b/content/v2.0/process-data/manage-tasks/delete-task.md @@ -0,0 +1,21 @@ +--- +title: Delete a task +seotitle: Delete a task for processing data in InfluxDB +description: > + How to delete a task in InfluxDB using the InfluxDB user interface or using + the 'influx' command line interface. +menu: + v2_0: + name: Delete a task + parent: Manage tasks + weight: 4 +--- + +## Delete a task in the InfluxDB UI + + +## Delete a task with the `influx` CLI + +```sh +influx task delete -i task-id +``` diff --git a/content/v2.0/process-data/manage-tasks/update-task.md b/content/v2.0/process-data/manage-tasks/update-task.md new file mode 100644 index 000000000..1e47709dc --- /dev/null +++ b/content/v2.0/process-data/manage-tasks/update-task.md @@ -0,0 +1,21 @@ +--- +title: Update a task +seotitle: Update a task for processing data in InfluxDB +description: > + How to update a task that processes data in InfluxDB using the InfluxDB user + interface or the 'influx' command line interface. +menu: + v2_0: + name: Update a task + parent: Manage tasks + weight: 3 +--- + +## Update a task in the InfluxDB UI + + +## Update a task with the `influx` CLI + +```sh +influx task update -i task-id --status task-status +``` diff --git a/content/v2.0/process-data/manage-tasks/view-tasks.md b/content/v2.0/process-data/manage-tasks/view-tasks.md new file mode 100644 index 000000000..9f206f57d --- /dev/null +++ b/content/v2.0/process-data/manage-tasks/view-tasks.md @@ -0,0 +1,21 @@ +--- +title: View tasks in InfluxDB +seotitle: View created tasks that process data in InfluxDB +description: > + How to view all created data processing tasks using the InfluxDB user interface + or the 'influx' command line interface. +menu: + v2_0: + name: View tasks + parent: Manage tasks + weight: 2 +--- + +## View tasks in the InfluxDB UI + + +## View tasks with the `influx` CLI + +```sh +influx task find -i task-id -n user-id --limit=100 --org-id=organization-id +``` diff --git a/content/v2.0/process-data/write-a-task.md b/content/v2.0/process-data/write-a-task.md new file mode 100644 index 000000000..20f48aa11 --- /dev/null +++ b/content/v2.0/process-data/write-a-task.md @@ -0,0 +1,78 @@ +--- +title: Write a task +seotitle: Write a task that processes data in InfluxDB +description: placeholder +menu: + v2_0: + name: Write a task + parent: Process data + weight: 1 +--- + +Tasks are essentially Flux scripts with a "destination." +This destination could be + + +**Requirements:** + +- Task options / scheduler block +- Data source +- Some type of aggregation +- and a `to` statement + +```js +option task = { + name: "cqinterval15m", + every: 1h, + offset: 15m , + concurrency: 4, + retry: 5 +} +``` + +```js +option task = { + name: "cqinterval15m", + cron: "0 0 * * *", + offset: 15m , + concurrency: 4, + retry: 5 +} +``` + +## Task configuration options + +### name +I think it might even be optional +if you dont specify one i think we just put in some default name. + +_**Data type:** String_ + +### every +Defines the inteveral at which the task will run. + +_**Data type:** Duration_ + +_Cannot be used with `cron`_ + +### cron +- The cron schedule. +- Based on system time. +_**Data type:** String_ + +_Cannot be used with `every`_ + +### offset +is so you can allow for data to come in off scheduler. so if you want a task to run on the hour `cron: "0 * * * *"` but your data might come in 10 min late you could say `offset: 15m` + +_**Data type:** Duration_ + +### concurrency +how many concurrent runs of a task can happen at once.. say your schedule is `every: 1s` but it takes 10 sec to complete. you can set a concurrency that will allow that to happen and not just queue up. + +_**Data type:** Integer_ + +### retry +is a int, the number of times to retry before we assume failure. + +_**Data type:** Integer_ From 31e0b6a02f1a5e2d370b053883e3d3983b89bc3c Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Wed, 16 Jan 2019 10:41:37 -0700 Subject: [PATCH 02/11] finalized write a task content --- content/v2.0/process-data/_index.md | 2 +- .../common-tasks/downsample-data.md | 21 +++ content/v2.0/process-data/task-options.md | 45 ++++++ content/v2.0/process-data/write-a-task.md | 145 +++++++++++++----- 4 files changed, 171 insertions(+), 42 deletions(-) create mode 100644 content/v2.0/process-data/task-options.md diff --git a/content/v2.0/process-data/_index.md b/content/v2.0/process-data/_index.md index 4b4c94483..6aa8a7985 100644 --- a/content/v2.0/process-data/_index.md +++ b/content/v2.0/process-data/_index.md @@ -9,7 +9,7 @@ menu: --- InfluxDB's _**task engine**_ is designed for processing and analyzing data. -Tasks are user defined Flux scripts that take a stream of input data, modify or +Tasks are user-defined Flux scripts that take a stream of input data, modify or analyze it in some way, then perform an action all on a specified schedule. Examples include data downsampling, anomaly detection _(Coming)_, alerting _(Coming)_, etc. diff --git a/content/v2.0/process-data/common-tasks/downsample-data.md b/content/v2.0/process-data/common-tasks/downsample-data.md index 5eafaabc5..9a565fef4 100644 --- a/content/v2.0/process-data/common-tasks/downsample-data.md +++ b/content/v2.0/process-data/common-tasks/downsample-data.md @@ -14,3 +14,24 @@ menu: - Data source - Some type of aggregation - and a `to` statement + + +```js +option task = { + name: "cqinterval15m", + every: 1w, +} + +data = from(bucket: "telegraf") + |> range(start: -task.every * 2) + |> filter(fn: (r) => r._measurement == "cpu") + +downsampleHourly = (table=<-) => + table + |> aggregateWindow(fn: mean, every: 1h) + |> set(key: "_measurement", value: "cpu_1h" ) + |> to(bucket: "telegraf_downsampled") + +downsampleHourly(data) + +``` diff --git a/content/v2.0/process-data/task-options.md b/content/v2.0/process-data/task-options.md new file mode 100644 index 000000000..740a6ab20 --- /dev/null +++ b/content/v2.0/process-data/task-options.md @@ -0,0 +1,45 @@ +--- +title: Task configuration options +seotitle: InfluxDB task configuration options +description: placeholder +menu: + v2_0: + name: Task options + parent: Process data + weight: 5 +--- + +### name +I think it might even be optional +if you dont specify one i think we just put in some default name. + +_**Data type:** String_ + +### every +Defines the interval at which the task will run. + +_**Data type:** Duration_ + +_Cannot be used with `cron`_ + +### cron +- The cron schedule. +- Based on system time. +_**Data type:** String_ + +_Cannot be used with `every`_ + +### offset +is so you can allow for data to come in off scheduler. so if you want a task to run on the hour `cron: "0 * * * *"` but your data might come in 10 min late you could say `offset: 15m` + +_**Data type:** Duration_ + +### concurrency +how many concurrent runs of a task can happen at once.. say your schedule is `every: 1s` but it takes 10 sec to complete. you can set a concurrency that will allow that to happen and not just queue up. + +_**Data type:** Integer_ + +### retry +The number of times to retry before we assume failure. + +_**Data type:** Integer_ diff --git a/content/v2.0/process-data/write-a-task.md b/content/v2.0/process-data/write-a-task.md index 20f48aa11..2f04f8e03 100644 --- a/content/v2.0/process-data/write-a-task.md +++ b/content/v2.0/process-data/write-a-task.md @@ -1,6 +1,6 @@ --- -title: Write a task -seotitle: Write a task that processes data in InfluxDB +title: Write an InfluxDB task +seotitle: Write an InfluxDB task that processes data description: placeholder menu: v2_0: @@ -9,70 +9,133 @@ menu: weight: 1 --- -Tasks are essentially Flux scripts with a "destination." -This destination could be +InfluxDB tasks are user-defined Flux scripts that take a stream of input data, +modify or analyze it in some way, then perform an action all on a specified schedule. +In their simplest form, tasks are essentially Flux scripts with a "destination." +This destination could be another bucket, another measurement, an alert endpoint _(Coming)_, etc. +This article walks through writing a basic InfluxDB task that downsamples +data and stores it in a new bucket. -**Requirements:** +## Components of a Task +Every InfluxDB task needs the following four components. +Their form and order can vary, but the are all essential parts of a task. -- Task options / scheduler block -- Data source -- Some type of aggregation -- and a `to` statement +- [Task options](#define-task-options) +- [A data source](#define-a-data-source) +- [Data processing or transformation](#process-or-transform-your-data) +- [A destination](#define-a-destination) + +_[Skip to the full example task script](#full-example-task-script)_ + +## Define task options +Task options define specific information about the task. +The example below illustrates how task options are defined in your Flux script: ```js option task = { name: "cqinterval15m", every: 1h, - offset: 15m , - concurrency: 4, + offset: 0m, + concurrency: 1, retry: 5 } ``` +_See [Task configuration options](/v2.0/process-data/task-options) for detailed information +about each option._ + +{{% note %}} +**When creating a task in the InfluxDB user interface (UI)**, task options are not required +in your Flux script. They are defined in UI while creating the task. +{{% /note %}} + +## Define a data source +Define a data source using Flux's [`from()` function](#) or any other [Flux input functions](#). + +For convenience, consider creating a variable that includes the sourced data with +the required `range()` and any relevant filters. + ```js -option task = { - name: "cqinterval15m", - cron: "0 0 * * *", - offset: 15m , - concurrency: 4, - retry: 5 -} +data = from(bucket: "telegraf/default") + |> range(start: -task.every) + |> filter(fn: (r) => + r._measurement == "mem" AND + r.host == "myHost" + ) ``` -## Task configuration options +{{% note %}} +#### Using task options in your Flux script +Task options are passed as part of a `task` object and can be referenced in your Flux script. +In the example above, the time range is defined as `-task.every`. -### name -I think it might even be optional -if you dont specify one i think we just put in some default name. +`task.every` is dot notation that references the `every` property of the `task` object. +`every` is defined as `1h`, therefore `-task.every` equates to `-1h`. -_**Data type:** String_ +Using task options to define values in your Flux script can make reusing your task easier. +{{% /note %}} -### every -Defines the inteveral at which the task will run. +## Process or transform your data +The purpose of tasks is to process or transform data in some way. +What exactly happens and what form the output data takes is up to you and your +specific use case. -_**Data type:** Duration_ +The example below illustrates a task that downsamples data by calculating the average of set intervals. +It uses the `data` variable defined [above](#define-a-data-source) as the data source. +It then windows the data into 5 minute intervals and calculates the average of each +window using the [`aggregateWindow()` function](#). -_Cannot be used with `cron`_ +```js +data + |> aggregateWindow( + every: 5m, + fn: mean + ) +``` -### cron -- The cron schedule. -- Based on system time. -_**Data type:** String_ +_See [Common tasks](/v2.0/process-data/common-tasks) for examples of tasks commonly used with InfluxDB._ -_Cannot be used with `every`_ +## Define a destination +In the vast majority of task use cases, once data is transformed, it needs to sent and stored somewhere. +This could be a separate bucket with a different retention policy, another measurement, or even an alert endpoint _(Coming)_. -### offset -is so you can allow for data to come in off scheduler. so if you want a task to run on the hour `cron: "0 * * * *"` but your data might come in 10 min late you could say `offset: 15m` +The example below uses Flux's [`to()` function](#) to send the transformed data to another bucket: -_**Data type:** Duration_ +```js +// ... +|> to(bucket: "telegraf_downsampled") +``` -### concurrency -how many concurrent runs of a task can happen at once.. say your schedule is `every: 1s` but it takes 10 sec to complete. you can set a concurrency that will allow that to happen and not just queue up. +## Full example task script +Below is the full example task script that combines all of the components described above: -_**Data type:** Integer_ -### retry -is a int, the number of times to retry before we assume failure. +```js +// Task options +option task = { + name: "cqinterval15m", + every: 1h, + offset: 0m, + concurrency: 1, + retry: 5 +} -_**Data type:** Integer_ +// Data source +data = from(bucket: "telegraf/default") + |> range(start: -task.every) + |> filter(fn: (r) => + r._measurement == "mem" AND + r.host == "myHost" + ) + +data + // Data transformation + |> aggregateWindow( + every: 5m, + fn: mean + ) + // Data destination + |> to(bucket: "telegraf_downsampled") + +``` From 61851dfad35c2cbe3e0c4a9e8182fba255115b56 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Wed, 16 Jan 2019 15:26:44 -0700 Subject: [PATCH 03/11] completed the task options doc --- content/v2.0/process-data/task-options.md | 89 +++++++++++++++++++---- 1 file changed, 73 insertions(+), 16 deletions(-) diff --git a/content/v2.0/process-data/task-options.md b/content/v2.0/process-data/task-options.md index 740a6ab20..fc1ea7d5c 100644 --- a/content/v2.0/process-data/task-options.md +++ b/content/v2.0/process-data/task-options.md @@ -9,37 +9,94 @@ menu: weight: 5 --- -### name -I think it might even be optional -if you dont specify one i think we just put in some default name. +Task options define specific information about the task and are specified in your +Flux script or in the InfluxDB user interface (UI). +The following task options are available: + +- [name](#name) +- [every](#every) +- [cron](#cron) +- [offset](#offset) +- [concurrency](#concurrency) +- [retry](#retry) + +{{% note %}} +`every` and `cron` are mutually exclusive, but at least one is required. +{{% /note %}} + +## name +The name of the task. +If no name is specified, the generated task ID is used. _**Data type:** String_ -### every -Defines the interval at which the task will run. +```js +options task = { + // ... + name: "taskName", +} +``` + +## every +The interval at which the task runs. _**Data type:** Duration_ -_Cannot be used with `cron`_ +```js +options task = { + every: 1h, +} +``` + +## cron +The cron schedule on which the task runs. Cron execution is based on system time. -### cron -- The cron schedule. -- Based on system time. _**Data type:** String_ -_Cannot be used with `every`_ +```js +options task = { + cron: "0 * * * *", +} +``` -### offset -is so you can allow for data to come in off scheduler. so if you want a task to run on the hour `cron: "0 * * * *"` but your data might come in 10 min late you could say `offset: 15m` +## offset +Delays the execution of the task but preserves the original time range. +For example, if a task is to run on the hour, a `10m` offset will delay it to 10 +minutes after the hour, but all time ranges defined in the task are relative to +the specified execution time. +A common use case is to allow late data to come in before executing the task. _**Data type:** Duration_ -### concurrency -how many concurrent runs of a task can happen at once.. say your schedule is `every: 1s` but it takes 10 sec to complete. you can set a concurrency that will allow that to happen and not just queue up. +```js +options task = { + // ... + offset: "0 * * * *", +} +``` + +## concurrency +The number task executions that can run concurrently. +If the concurrency limit is reached, all subsequent executions of the task are be queued +until other running task executions complete. _**Data type:** Integer_ -### retry -The number of times to retry before we assume failure. +```js +options task = { + // ... + concurrency: 2, +} +``` + +## retry +The number of times to retry the task before it is considered as having failed. _**Data type:** Integer_ + +```js +options task = { + // ... + retry: 2, +} +``` From ac4b488fa4c33c2a07a741bc10992f7a174db327 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Thu, 17 Jan 2019 15:39:35 -0700 Subject: [PATCH 04/11] WIP task docs --- .../common-tasks/downsample-data.md | 7 ++++-- .../process-data/manage-tasks/create-task.md | 22 ++++++++++++++++++- content/v2.0/process-data/task-options.md | 3 +-- content/v2.0/process-data/write-a-task.md | 6 ++++- 4 files changed, 32 insertions(+), 6 deletions(-) diff --git a/content/v2.0/process-data/common-tasks/downsample-data.md b/content/v2.0/process-data/common-tasks/downsample-data.md index 9a565fef4..0a5149684 100644 --- a/content/v2.0/process-data/common-tasks/downsample-data.md +++ b/content/v2.0/process-data/common-tasks/downsample-data.md @@ -15,6 +15,10 @@ menu: - Some type of aggregation - and a `to` statement +- You can't write data into the same bucket you're reading from +- A two buckets +- `to()` requires a bucket AND org + ```js option task = { @@ -30,8 +34,7 @@ downsampleHourly = (table=<-) => table |> aggregateWindow(fn: mean, every: 1h) |> set(key: "_measurement", value: "cpu_1h" ) - |> to(bucket: "telegraf_downsampled") + |> to(bucket: "telegraf_downsampled", org: "my-org") downsampleHourly(data) - ``` diff --git a/content/v2.0/process-data/manage-tasks/create-task.md b/content/v2.0/process-data/manage-tasks/create-task.md index f578415e9..da9c08534 100644 --- a/content/v2.0/process-data/manage-tasks/create-task.md +++ b/content/v2.0/process-data/manage-tasks/create-task.md @@ -11,12 +11,32 @@ menu: weight: 1 --- +_This article assumes you have already [written a task](/v2.0/process-data/write-a-task)._ ## Create a task in the InfluxDB UI +- From the data Explorer +- From the task UI ## Create a task with the `influx` CLI +Use `influx task create` command to create a new task. +It accepts either a file path or raw Flux. +###### Create a task using a file ```sh -influx task create /path/to/task-script.flux +influx task create --org=org-name @/path/to/task-script.flux +``` + +###### Create a task using raw Flux +```sh +influx task create --org=org-name - # to open stdin pipe + +options task = { + name: "task-name", + every: 6h +} + +# ... Task script ... + +# to close the pipe and submit the command ``` diff --git a/content/v2.0/process-data/task-options.md b/content/v2.0/process-data/task-options.md index fc1ea7d5c..3c86f818a 100644 --- a/content/v2.0/process-data/task-options.md +++ b/content/v2.0/process-data/task-options.md @@ -25,8 +25,7 @@ The following task options are available: {{% /note %}} ## name -The name of the task. -If no name is specified, the generated task ID is used. +The name of the task. _**Required**_. _**Data type:** String_ diff --git a/content/v2.0/process-data/write-a-task.md b/content/v2.0/process-data/write-a-task.md index 2f04f8e03..243a68ae1 100644 --- a/content/v2.0/process-data/write-a-task.md +++ b/content/v2.0/process-data/write-a-task.md @@ -104,9 +104,13 @@ The example below uses Flux's [`to()` function](#) to send the transformed data ```js // ... -|> to(bucket: "telegraf_downsampled") +|> to(bucket: "telegraf_downsampled", org: "my-org") ``` +{{% note %}} +You cannot write to the same bucket you are reading from. +{{% /note %}} + ## Full example task script Below is the full example task script that combines all of the components described above: From c7a0863550e0b93d1900a7e490554c01236c40ad Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Thu, 17 Jan 2019 23:04:56 -0700 Subject: [PATCH 05/11] more WIP task content --- .../process-data/manage-tasks/create-task.md | 8 ++++++-- .../process-data/manage-tasks/delete-task.md | 8 +++++++- .../process-data/manage-tasks/update-task.md | 18 +++++++++++++++++- .../process-data/manage-tasks/view-tasks.md | 6 +++++- content/v2.0/process-data/task-options.md | 16 ++++++++++------ content/v2.0/process-data/write-a-task.md | 6 +++--- 6 files changed, 48 insertions(+), 14 deletions(-) diff --git a/content/v2.0/process-data/manage-tasks/create-task.md b/content/v2.0/process-data/manage-tasks/create-task.md index da9c08534..47f3306aa 100644 --- a/content/v2.0/process-data/manage-tasks/create-task.md +++ b/content/v2.0/process-data/manage-tasks/create-task.md @@ -24,12 +24,16 @@ It accepts either a file path or raw Flux. ###### Create a task using a file ```sh -influx task create --org=org-name @/path/to/task-script.flux +# Pattern +influx task create --org @ + +# Example +influx task create --org my-org @/tasks/cq-mean-1h.flux ``` ###### Create a task using raw Flux ```sh -influx task create --org=org-name - # to open stdin pipe +influx task create --org my-org - # to open stdin pipe options task = { name: "task-name", diff --git a/content/v2.0/process-data/manage-tasks/delete-task.md b/content/v2.0/process-data/manage-tasks/delete-task.md index 843c10243..530851876 100644 --- a/content/v2.0/process-data/manage-tasks/delete-task.md +++ b/content/v2.0/process-data/manage-tasks/delete-task.md @@ -15,7 +15,13 @@ menu: ## Delete a task with the `influx` CLI +Use the `influx task delete` command to delete a task. +It requires a task ID, which is available in the output of `influx task find`. ```sh -influx task delete -i task-id +# Pattern +influx task delete -i + +# Example +influx task delete -i 0343698431c35000 ``` diff --git a/content/v2.0/process-data/manage-tasks/update-task.md b/content/v2.0/process-data/manage-tasks/update-task.md index 1e47709dc..cfc4646a9 100644 --- a/content/v2.0/process-data/manage-tasks/update-task.md +++ b/content/v2.0/process-data/manage-tasks/update-task.md @@ -15,7 +15,23 @@ menu: ## Update a task with the `influx` CLI +Use the `influx task update` command to update or change the status of an existing tasks. +It requires a task ID which is available in the output of `influx task find`. +###### Update a task's Flux script ```sh -influx task update -i task-id --status task-status +# Pattern +influx task update -i @/path/to/updated-task-script + +# Example +influx task update -i 0343698431c35000 @/tasks/cq-mean-1h.flux +``` + +###### Update the status of a task +```sh +# Pattern +influx task update -i --status < active | inactive > + +# Example +influx task update -i 0343698431c35000 --status inactive ``` diff --git a/content/v2.0/process-data/manage-tasks/view-tasks.md b/content/v2.0/process-data/manage-tasks/view-tasks.md index 9f206f57d..d2c5fa356 100644 --- a/content/v2.0/process-data/manage-tasks/view-tasks.md +++ b/content/v2.0/process-data/manage-tasks/view-tasks.md @@ -15,7 +15,11 @@ menu: ## View tasks with the `influx` CLI +Use the `influx task find` command to return a list of created tasks. ```sh -influx task find -i task-id -n user-id --limit=100 --org-id=organization-id +influx task find ``` + +_Other filtering options such as filtering by organization or user, or limiting the number of tasks returned are available. +See the [`influx task find` documentation](#) for information about other available flags._ diff --git a/content/v2.0/process-data/task-options.md b/content/v2.0/process-data/task-options.md index 3c86f818a..f9fb70308 100644 --- a/content/v2.0/process-data/task-options.md +++ b/content/v2.0/process-data/task-options.md @@ -31,8 +31,8 @@ _**Data type:** String_ ```js options task = { - // ... name: "taskName", + // ... } ``` @@ -43,17 +43,21 @@ _**Data type:** Duration_ ```js options task = { + // ... every: 1h, } ``` ## cron -The cron schedule on which the task runs. Cron execution is based on system time. +The [cron expression](https://en.wikipedia.org/wiki/Cron#Overview) that +defines the schedule on which the task runs. +Cron scheduling is based on system time. _**Data type:** String_ ```js options task = { + // ... cron: "0 * * * *", } ``` @@ -63,7 +67,7 @@ Delays the execution of the task but preserves the original time range. For example, if a task is to run on the hour, a `10m` offset will delay it to 10 minutes after the hour, but all time ranges defined in the task are relative to the specified execution time. -A common use case is to allow late data to come in before executing the task. +A common use case is offsetting execution to account for data that may arrive late. _**Data type:** Duration_ @@ -75,9 +79,9 @@ options task = { ``` ## concurrency -The number task executions that can run concurrently. -If the concurrency limit is reached, all subsequent executions of the task are be queued -until other running task executions complete. +The number task of executions that can run concurrently. +If the concurrency limit is reached, all subsequent executions are queued until +other running task executions complete. _**Data type:** Integer_ diff --git a/content/v2.0/process-data/write-a-task.md b/content/v2.0/process-data/write-a-task.md index 243a68ae1..97f9ae993 100644 --- a/content/v2.0/process-data/write-a-task.md +++ b/content/v2.0/process-data/write-a-task.md @@ -46,15 +46,15 @@ _See [Task configuration options](/v2.0/process-data/task-options) for detailed about each option._ {{% note %}} -**When creating a task in the InfluxDB user interface (UI)**, task options are not required -in your Flux script. They are defined in UI while creating the task. +If creating a task in the InfluxDB user interface (UI), task options are defined +in form fields when creating the task. {{% /note %}} ## Define a data source Define a data source using Flux's [`from()` function](#) or any other [Flux input functions](#). For convenience, consider creating a variable that includes the sourced data with -the required `range()` and any relevant filters. +the required time range and any relevant filters. ```js data = from(bucket: "telegraf/default") From 34cc68eac25119c1beaeda1509a53cb47686f1cb Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Fri, 18 Jan 2019 12:12:28 -0700 Subject: [PATCH 06/11] more work on task docs --- content/v2.0/process-data/_index.md | 8 +++- .../v2.0/process-data/common-tasks/_index.md | 2 + .../process-data/manage-tasks/create-task.md | 45 +++++++++++++++++-- .../process-data/manage-tasks/delete-task.md | 14 +++++- .../process-data/manage-tasks/update-task.md | 36 ++++++++++++--- .../process-data/manage-tasks/view-tasks.md | 19 ++++++-- content/v2.0/process-data/task-options.md | 2 + content/v2.0/process-data/write-a-task.md | 6 +-- layouts/shortcodes/img-hd.html | 11 +++-- 9 files changed, 121 insertions(+), 22 deletions(-) diff --git a/content/v2.0/process-data/_index.md b/content/v2.0/process-data/_index.md index 6aa8a7985..d2d8f16de 100644 --- a/content/v2.0/process-data/_index.md +++ b/content/v2.0/process-data/_index.md @@ -9,10 +9,14 @@ menu: --- InfluxDB's _**task engine**_ is designed for processing and analyzing data. -Tasks are user-defined Flux scripts that take a stream of input data, modify or -analyze it in some way, then perform an action all on a specified schedule. +A task is a scheduled Flux query that take a stream of input data, modify or +analyze it in some way, then perform an action. Examples include data downsampling, anomaly detection _(Coming)_, alerting _(Coming)_, etc. +{{% note %}} +Tasks are a replacement for InfluxDB v1.x's continuous queries. +{{% /note %}} + The following articles explain how to configure and build tasks using the InfluxDB user interface (UI) and via raw Flux scripts with the `influx` command line interface (CLI). They also provide examples of commonly used tasks. diff --git a/content/v2.0/process-data/common-tasks/_index.md b/content/v2.0/process-data/common-tasks/_index.md index 7b0455dff..22922804e 100644 --- a/content/v2.0/process-data/common-tasks/_index.md +++ b/content/v2.0/process-data/common-tasks/_index.md @@ -8,3 +8,5 @@ menu: parent: Process data weight: 4 --- + +The following articles walk through common task use cases. diff --git a/content/v2.0/process-data/manage-tasks/create-task.md b/content/v2.0/process-data/manage-tasks/create-task.md index 47f3306aa..64616ea95 100644 --- a/content/v2.0/process-data/manage-tasks/create-task.md +++ b/content/v2.0/process-data/manage-tasks/create-task.md @@ -11,14 +11,53 @@ menu: weight: 1 --- +InfluxDB provides multiple ways to create tasks both in the InfluxDB user interface (UI) +and the `influx` command line interface (CLI). + _This article assumes you have already [written a task](/v2.0/process-data/write-a-task)._ ## Create a task in the InfluxDB UI +The InfluxDB UI provides multiple ways to create a task: -- From the data Explorer -- From the task UI +- [Create a task from the Data Explorer](#create-a-task-from-the-data-explorer) +- [Create a task in the Task UI](#create-a-task-in-the-task-ui) +- [Import a task](#import-a-task) -## Create a task with the `influx` CLI +### Create a task from the Data Explorer +1. Click on the **Data Explorer** icon in the left navigation menu. + + {{< img-hd src="/img/data-explorer-icon.png" alt="Data Explorer Icon" />}} + +2. Building a query and click **Save As** in the upper right. +3. Select the **Task** option. +4. Specify the task options. See [Task options](/v2.0/process-data/task-options) + for detailed information about each option. +5. Click **Save as Task**. + +{{< img-hd src="/img/data-explorer-save-as-task.png" alt="Add a task from the Data Explorer"/>}} + +### Create a task in the Task UI +1. Click on the **Tasks** icon in the left navigation menu. + + {{< img-hd src="/img/tasks-icon.png" alt="Tasks Icon" />}} + +2. Click **+ Create Task** in the upper right. +3. In the left panel, specify the task options. + See [Task options](/v2.0/process-data/task-options)for detailed information about each option. +4. In the right panel, enter your task script. +5. Click **Save** in the upper right. + +{{< img-hd src="/img/tasks-create-edit.png" alt="Create a task" />}} + +### Import a task +1. Click on the **Tasks** icon in the left navigation menu. +2. Click **Import** in the upper right. +3. Drag and drop or select a file to upload. +4. Click **Upload Task**. + +{{< img-hd src="/img/tasks-import-task.png" alt="Import a task" />}} + +## Create a task using the influx CLI Use `influx task create` command to create a new task. It accepts either a file path or raw Flux. diff --git a/content/v2.0/process-data/manage-tasks/delete-task.md b/content/v2.0/process-data/manage-tasks/delete-task.md index 530851876..064bd79f0 100644 --- a/content/v2.0/process-data/manage-tasks/delete-task.md +++ b/content/v2.0/process-data/manage-tasks/delete-task.md @@ -12,11 +12,21 @@ menu: --- ## Delete a task in the InfluxDB UI +1. Click the **Tasks** icon in the left navigation menu. + + {{< img-hd src="/img/tasks-icon.png" alt="Tasks Icon" />}} + +2. In the list of tasks, hover over the task you would like to delete. +3. Click **Delete** on the far right. +4. Click **Confirm**. + +{{< img-hd src="/img/tasks-delete-task.png" alt="Delete a task" />}} -## Delete a task with the `influx` CLI +## Delete a task with the influx CLI Use the `influx task delete` command to delete a task. -It requires a task ID, which is available in the output of `influx task find`. + +_This command requires a task ID, which is available in the output of `influx task find`._ ```sh # Pattern diff --git a/content/v2.0/process-data/manage-tasks/update-task.md b/content/v2.0/process-data/manage-tasks/update-task.md index cfc4646a9..88ca555a0 100644 --- a/content/v2.0/process-data/manage-tasks/update-task.md +++ b/content/v2.0/process-data/manage-tasks/update-task.md @@ -12,13 +12,34 @@ menu: --- ## Update a task in the InfluxDB UI +To view your tasks, click the **Tasks** icon in the left navigation menu. + +{{< img-hd src="/img/tasks-icon.png" alt="Tasks Icon" />}} + +#### Update a task's Flux script +1. In the list of tasks, click the **Name** of the task you would like to update. +2. In the left panel, modify the task options. +3. In the right panel, modify the task script. +4. Click **Save** in the upper right. + +{{< img-hd src="/img/tasks-create-edit.png" alt="Update a task" />}} + +#### Update the status of a task +In the list of tasks, click the toggle in the **Active** column of the task you +would like to activate or inactivate. -## Update a task with the `influx` CLI -Use the `influx task update` command to update or change the status of an existing tasks. -It requires a task ID which is available in the output of `influx task find`. +## Update a task with the influx CLI +Use the `influx task update` command to update or change the status of an existing task. + +_This command requires a task ID, which is available in the output of `influx task find`._ + +#### Update a task's Flux script +Pass the file path of your updated Flux script to the `influx task update` command +with the ID of the task you would like to update. +Modified [task options](/v2.0/process-data/task-options) defined in the Flux +script are also updated. -###### Update a task's Flux script ```sh # Pattern influx task update -i @/path/to/updated-task-script @@ -27,7 +48,12 @@ influx task update -i @/path/to/updated-task-script influx task update -i 0343698431c35000 @/tasks/cq-mean-1h.flux ``` -###### Update the status of a task +#### Update the status of a task +Pass the ID of the task you would like to update to the `influx task update` +command with the `--status` flag. + +_Possible arguments of the `--status` flag are `active` or `inactive`._ + ```sh # Pattern influx task update -i --status < active | inactive > diff --git a/content/v2.0/process-data/manage-tasks/view-tasks.md b/content/v2.0/process-data/manage-tasks/view-tasks.md index d2c5fa356..1f13aae67 100644 --- a/content/v2.0/process-data/manage-tasks/view-tasks.md +++ b/content/v2.0/process-data/manage-tasks/view-tasks.md @@ -12,14 +12,27 @@ menu: --- ## View tasks in the InfluxDB UI +Click the **Tasks** icon in the left navigation to view the lists of tasks. +{{< img-hd src="/img/tasks-icon.png" alt="Tasks Icon" />}} -## View tasks with the `influx` CLI +### Filter the list of tasks + +1. Enable the **Show Inactive** option to include inactive tasks in the list. +2. Enter text in the **Filter tasks by name** field to search for tasks by name. +3. Select an organization from the **All Organizations** dropdown to filter the list by organization. +4. Click on the heading of any column to sort by that field. + +{{< img-hd src="/img/tasks-list.png" alt="View and filter tasks" />}} + +## View tasks with the influx CLI Use the `influx task find` command to return a list of created tasks. ```sh influx task find ``` -_Other filtering options such as filtering by organization or user, or limiting the number of tasks returned are available. -See the [`influx task find` documentation](#) for information about other available flags._ +#### Filter tasks using the CLI +Other filtering options such as filtering by organization or user, +or limiting the number of tasks returned are available. +See the [`influx task find` documentation](#) for information about other available flags. diff --git a/content/v2.0/process-data/task-options.md b/content/v2.0/process-data/task-options.md index f9fb70308..373a06d6f 100644 --- a/content/v2.0/process-data/task-options.md +++ b/content/v2.0/process-data/task-options.md @@ -41,6 +41,8 @@ The interval at which the task runs. _**Data type:** Duration_ +_**Note:** In the InfluxDB UI, the **Interval** field sets this option_. + ```js options task = { // ... diff --git a/content/v2.0/process-data/write-a-task.md b/content/v2.0/process-data/write-a-task.md index 97f9ae993..de58c80d6 100644 --- a/content/v2.0/process-data/write-a-task.md +++ b/content/v2.0/process-data/write-a-task.md @@ -9,10 +9,8 @@ menu: weight: 1 --- -InfluxDB tasks are user-defined Flux scripts that take a stream of input data, -modify or analyze it in some way, then perform an action all on a specified schedule. -In their simplest form, tasks are essentially Flux scripts with a "destination." -This destination could be another bucket, another measurement, an alert endpoint _(Coming)_, etc. +InfluxDB tasks are scheduled Flux scripts that take a stream of input data, modify or analyze +it in some way, then store the modified data in a new bucket or perform other actions. This article walks through writing a basic InfluxDB task that downsamples data and stores it in a new bucket. diff --git a/layouts/shortcodes/img-hd.html b/layouts/shortcodes/img-hd.html index 9c08a761c..a49801b1b 100644 --- a/layouts/shortcodes/img-hd.html +++ b/layouts/shortcodes/img-hd.html @@ -1,7 +1,12 @@ {{ .Inner }} {{ $src := .Get "src" }} {{ $alt := .Get "alt" }} -{{ with (imageConfig ( print "/static" $src )) }} - {{ $imageWidth := div .Width 2 }} - {{ $alt }} + +{{ if (fileExists ( print "/static" $src )) }} + {{ with (imageConfig ( print "/static" $src )) }} + {{ $imageWidth := div .Width 2 }} + {{ $alt }} + {{ end }} +{{ else }} + {{ $alt }} {{ end }} From fb5d9f2684bddc4ab83396b6dfdccd4868caf6dc Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Fri, 18 Jan 2019 12:25:15 -0700 Subject: [PATCH 07/11] updates to common tasks docs --- content/v2.0/process-data/common-tasks/_index.md | 8 ++++++++ .../v2.0/process-data/common-tasks/downsample-data.md | 11 +++++++---- 2 files changed, 15 insertions(+), 4 deletions(-) diff --git a/content/v2.0/process-data/common-tasks/_index.md b/content/v2.0/process-data/common-tasks/_index.md index 22922804e..465eac997 100644 --- a/content/v2.0/process-data/common-tasks/_index.md +++ b/content/v2.0/process-data/common-tasks/_index.md @@ -10,3 +10,11 @@ menu: --- The following articles walk through common task use cases. + +{{% note %}} +This list will continue to grow. +If you have suggestions, please [create an issue](https://github.com/influxdata/docs-v2/issues/new) +on the InfluxData documentation repository on Github. +{{% /note %}} + +[Downsample Data with InfluxDB](/v2.0/process-data/common-tasks/downsample-data) diff --git a/content/v2.0/process-data/common-tasks/downsample-data.md b/content/v2.0/process-data/common-tasks/downsample-data.md index 0a5149684..56a320893 100644 --- a/content/v2.0/process-data/common-tasks/downsample-data.md +++ b/content/v2.0/process-data/common-tasks/downsample-data.md @@ -1,5 +1,5 @@ --- -title: Downsample data +title: Downsample data with InfluxDB seotitle: Downsample data in an InfluxDB task description: placeholder menu: @@ -9,9 +9,12 @@ menu: weight: 4 --- +- Talk about Continuous Queries + **Requirements:** -- Data source +- A "source" bucket +- A "destination" bucket - Some type of aggregation - and a `to` statement @@ -30,8 +33,8 @@ data = from(bucket: "telegraf") |> range(start: -task.every * 2) |> filter(fn: (r) => r._measurement == "cpu") -downsampleHourly = (table=<-) => - table +downsampleHourly = (tables=<-) => + tables |> aggregateWindow(fn: mean, every: 1h) |> set(key: "_measurement", value: "cpu_1h" ) |> to(bucket: "telegraf_downsampled", org: "my-org") From 6e13d5292f68e0ee7c44baa3d207739fdced4a20 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Fri, 18 Jan 2019 15:55:38 -0700 Subject: [PATCH 08/11] minor updates to process-data docs --- content/v2.0/UI/using-tasks.md | 37 ---------------------------------- 1 file changed, 37 deletions(-) delete mode 100644 content/v2.0/UI/using-tasks.md diff --git a/content/v2.0/UI/using-tasks.md b/content/v2.0/UI/using-tasks.md deleted file mode 100644 index c9848082a..000000000 --- a/content/v2.0/UI/using-tasks.md +++ /dev/null @@ -1,37 +0,0 @@ ---- -title: Using tasks -description: This is just an example post to show the format of new 2.0 posts -menu: - v2_0: - name: Using tasks - weight: 1 ---- -A task is a scheduled Flux query. Main use case is replacement for continuous queries, add info about CQs. - -**To filter the list of tasks**: - -1. Enable the **Show Inactive** option to include inactive tasks on the list. -2. Enter text in the **Filter tasks by name** field to search for tasks by name. -3. Select an organization from the **All Organizations** dropdown to filter the list by organization. -4. Click on the heading of any column to sort by that field. - -**To import a task**: - -1. Click the Tasks (calendar) icon in the left navigation menu. -2. Click **Import** in the upper right. -3. Drag and drop or select a file to upload. -4. !!! - -**To create a task**: - -1. Click **+ Create Task**. -2. In the left sidebar panel, enter the following details: - * **Name**: The name of your task. - * **Owner**: Select an organization from the drop-down menu. - * **Schedule Task**: Select **Interval** for !!!! or **Cron** to !!!. Also enter value below (interval window or Cron thing). - * **Offset**: Enter an offset time. If you schedule it to run at the hour but you have an offset of ten minutes, then it runs at an hour and ten minutes. -3. In the right panel, enter your task script. -4. Click **Save**. - - -**Disable tasks** From e901980d5df6e27a882549712ec9d312f5eca7e7 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Fri, 18 Jan 2019 17:29:28 -0700 Subject: [PATCH 09/11] finalized downsampling guide, added page descriptions to task docs --- content/v2.0/process-data/_index.md | 9 ++- .../v2.0/process-data/common-tasks/_index.md | 8 +- .../common-tasks/downsample-data.md | 81 ++++++++++++++----- .../v2.0/process-data/manage-tasks/_index.md | 4 +- content/v2.0/process-data/task-options.md | 4 +- content/v2.0/process-data/write-a-task.md | 4 +- 6 files changed, 82 insertions(+), 28 deletions(-) diff --git a/content/v2.0/process-data/_index.md b/content/v2.0/process-data/_index.md index d2d8f16de..0ba0c738c 100644 --- a/content/v2.0/process-data/_index.md +++ b/content/v2.0/process-data/_index.md @@ -1,7 +1,9 @@ --- title: Process Data with InfluxDB tasks seotitle: Process Data with InfluxDB tasks -description: placeholder +description: > + InfluxDB's task engine runs scheduled Flux tasks that process and analyze data. + This collection of articles provides information about creating and managing InfluxDB tasks. menu: v2_0: name: Process data @@ -21,4 +23,7 @@ The following articles explain how to configure and build tasks using the Influx and via raw Flux scripts with the `influx` command line interface (CLI). They also provide examples of commonly used tasks. -_Links for nested docs._ +[Write a task](/v2.0/process-data/write-a-task) +[Manage Tasks](/v2.0/process-data/manage-tasks) +[Common Tasks](/v2.0/process-data/common-tasks) +[Task Options](/v2.0/process-data/task-options) diff --git a/content/v2.0/process-data/common-tasks/_index.md b/content/v2.0/process-data/common-tasks/_index.md index 465eac997..818bcef33 100644 --- a/content/v2.0/process-data/common-tasks/_index.md +++ b/content/v2.0/process-data/common-tasks/_index.md @@ -1,7 +1,9 @@ --- title: Common data processing tasks seotitle: Common data processing tasks performed with with InfluxDB -description: placeholder +description: > + InfluxDB Tasks process data on specified schedules. + This collection of articles walks through common use cases for InfluxDB tasks. menu: v2_0: name: Common tasks @@ -11,10 +13,10 @@ menu: The following articles walk through common task use cases. +[Downsample Data with InfluxDB](/v2.0/process-data/common-tasks/downsample-data) + {{% note %}} This list will continue to grow. If you have suggestions, please [create an issue](https://github.com/influxdata/docs-v2/issues/new) on the InfluxData documentation repository on Github. {{% /note %}} - -[Downsample Data with InfluxDB](/v2.0/process-data/common-tasks/downsample-data) diff --git a/content/v2.0/process-data/common-tasks/downsample-data.md b/content/v2.0/process-data/common-tasks/downsample-data.md index 56a320893..c03c7aa1d 100644 --- a/content/v2.0/process-data/common-tasks/downsample-data.md +++ b/content/v2.0/process-data/common-tasks/downsample-data.md @@ -1,7 +1,9 @@ --- title: Downsample data with InfluxDB seotitle: Downsample data in an InfluxDB task -description: placeholder +description: > + How to create a task that downsamples data much like continuous queries + in previous versions of InfluxDB. menu: v2_0: name: Downsample data @@ -9,35 +11,74 @@ menu: weight: 4 --- -- Talk about Continuous Queries +One of the most common use cases for InfluxDB tasks is downsampling data to reduce +the overall disk usage as data collects over time. +In previous versions of InfluxDB, continuous queries filled this role. -**Requirements:** +This article walks through creating a continuous-query-like task that downsamples +data by aggregating data within windows of time, then storing the aggregate value in a new bucket. -- A "source" bucket -- A "destination" bucket -- Some type of aggregation -- and a `to` statement +### Requirements +To perform a downsampling task, you need to the following: -- You can't write data into the same bucket you're reading from -- A two buckets -- `to()` requires a bucket AND org +##### A "source" bucket +The bucket from which data is queried. +##### A "destination" bucket +A separate bucket where aggregated, downsampled data is stored. + +##### Some type of aggregation +To downsample data, it must be aggregated in some way. +What specific method of aggregation you use depends on your specific use case, +but examples include mean, median, top, bottom, etc. +View [Flux's aggregate functions](#) for more information and ideas. + +## Create a destination bucket +By design, tasks cannot write to the same bucket from which they are reading. +You need another bucket where the task can store the aggregated, downsampled data. + +_For information about creating buckets, see [Create a bucket](#)._ + +## Example downsampling task script +The example task script below is a very basic form of data downsampling that does the following: + +1. Defines a task named "cq-mem-data-1w" that runs once a week. +2. Defines a `data` variable that represents all data from the last 2 weeks in the + `mem` measurement of the `system-data` bucket. +3. Uses the [`aggregateWindow()` function](#) to window the data into 1 hour intervals + and calculate the average of each interval. +4. Stores the aggregated data in the `system-data-downsampled` bucket under the + `my-org` organization. ```js +// Task Options option task = { - name: "cqinterval15m", + name: "cq-mem-data-1w", every: 1w, } -data = from(bucket: "telegraf") +// Defines a data source +data = from(bucket: "system-data") |> range(start: -task.every * 2) - |> filter(fn: (r) => r._measurement == "cpu") + |> filter(fn: (r) => r._measurement == "mem") -downsampleHourly = (tables=<-) => - tables - |> aggregateWindow(fn: mean, every: 1h) - |> set(key: "_measurement", value: "cpu_1h" ) - |> to(bucket: "telegraf_downsampled", org: "my-org") - -downsampleHourly(data) +data + // Windows and aggregates the data in to 1h averages + |> aggregateWindow(fn: mean, every: 1h) + // Stores the aggregated data in a new bucket + |> to(bucket: "system-data-downsampled", org: "my-org") ``` + +Again, this is a very basic example, but it should provide you with a foundation +to build more complex downsampling tasks. + +## Add your task +Once your task is ready, see [Create a task](#) for information about adding it to InfluxDB. + +## Things to consider +- If there is a chance that data may arrive late, specify an `offset` in your + task options long enough to account for late-data. +- If running a task against a bucket with a finite retention policy, do not schedule + tasks to run too closely to the end of the retention policy. + Always provide a "cushion" for downsampling tasks to complete before the data + is dropped by the retention policy. diff --git a/content/v2.0/process-data/manage-tasks/_index.md b/content/v2.0/process-data/manage-tasks/_index.md index 197c1244a..b26aa7bb1 100644 --- a/content/v2.0/process-data/manage-tasks/_index.md +++ b/content/v2.0/process-data/manage-tasks/_index.md @@ -1,7 +1,9 @@ --- title: Manage tasks in InfluxDB seotitle: Manage data processing tasks in InfluxDB -description: placeholder +description: > + InfluxDB provides options for managing the creation, reading, updating, and deletion + of tasks using both the 'influx' CLI and the InfluxDB UI. menu: v2_0: name: Manage tasks diff --git a/content/v2.0/process-data/task-options.md b/content/v2.0/process-data/task-options.md index 373a06d6f..3600767e2 100644 --- a/content/v2.0/process-data/task-options.md +++ b/content/v2.0/process-data/task-options.md @@ -1,7 +1,9 @@ --- title: Task configuration options seotitle: InfluxDB task configuration options -description: placeholder +description: > + Task options define specific information about a task such as its name, + the schedule on which it runs, execution delays, and others. menu: v2_0: name: Task options diff --git a/content/v2.0/process-data/write-a-task.md b/content/v2.0/process-data/write-a-task.md index de58c80d6..53f73bd0d 100644 --- a/content/v2.0/process-data/write-a-task.md +++ b/content/v2.0/process-data/write-a-task.md @@ -1,7 +1,9 @@ --- title: Write an InfluxDB task seotitle: Write an InfluxDB task that processes data -description: placeholder +description: > + How to write an InfluxDB task that processes data in some way, then performs an action + such as storing the modified data in a new bucket or sending an alert. menu: v2_0: name: Write a task From ccde0199ca46cc7ca04ee948fc63ff649728e162 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Mon, 21 Jan 2019 15:21:03 -0700 Subject: [PATCH 10/11] updated placeholder link for task find cli command, other minor updates --- content/v2.0/process-data/common-tasks/downsample-data.md | 2 +- content/v2.0/process-data/manage-tasks/view-tasks.md | 3 ++- content/v2.0/process-data/write-a-task.md | 2 +- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/content/v2.0/process-data/common-tasks/downsample-data.md b/content/v2.0/process-data/common-tasks/downsample-data.md index c03c7aa1d..93472b688 100644 --- a/content/v2.0/process-data/common-tasks/downsample-data.md +++ b/content/v2.0/process-data/common-tasks/downsample-data.md @@ -73,7 +73,7 @@ Again, this is a very basic example, but it should provide you with a foundation to build more complex downsampling tasks. ## Add your task -Once your task is ready, see [Create a task](#) for information about adding it to InfluxDB. +Once your task is ready, see [Create a task](/v2.0/process-data/manage-tasks/create-task) for information about adding it to InfluxDB. ## Things to consider - If there is a chance that data may arrive late, specify an `offset` in your diff --git a/content/v2.0/process-data/manage-tasks/view-tasks.md b/content/v2.0/process-data/manage-tasks/view-tasks.md index 1f13aae67..40f019b92 100644 --- a/content/v2.0/process-data/manage-tasks/view-tasks.md +++ b/content/v2.0/process-data/manage-tasks/view-tasks.md @@ -35,4 +35,5 @@ influx task find #### Filter tasks using the CLI Other filtering options such as filtering by organization or user, or limiting the number of tasks returned are available. -See the [`influx task find` documentation](#) for information about other available flags. +See the [`influx task find` documentation](/v2.0/reference/cli/influx/task/find) +for information about other available flags. diff --git a/content/v2.0/process-data/write-a-task.md b/content/v2.0/process-data/write-a-task.md index 53f73bd0d..acfb31a13 100644 --- a/content/v2.0/process-data/write-a-task.md +++ b/content/v2.0/process-data/write-a-task.md @@ -17,7 +17,7 @@ it in some way, then store the modified data in a new bucket or perform other ac This article walks through writing a basic InfluxDB task that downsamples data and stores it in a new bucket. -## Components of a Task +## Components of a task Every InfluxDB task needs the following four components. Their form and order can vary, but the are all essential parts of a task. From 3f286f2783077c856523425aaea2594838bc9796 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Tue, 22 Jan 2019 09:10:25 -0700 Subject: [PATCH 11/11] updated flux-related placeholder content --- content/v2.0/process-data/common-tasks/downsample-data.md | 7 ++++--- content/v2.0/process-data/write-a-task.md | 8 +++++--- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/content/v2.0/process-data/common-tasks/downsample-data.md b/content/v2.0/process-data/common-tasks/downsample-data.md index 93472b688..c0ef87c45 100644 --- a/content/v2.0/process-data/common-tasks/downsample-data.md +++ b/content/v2.0/process-data/common-tasks/downsample-data.md @@ -31,7 +31,8 @@ A separate bucket where aggregated, downsampled data is stored. To downsample data, it must be aggregated in some way. What specific method of aggregation you use depends on your specific use case, but examples include mean, median, top, bottom, etc. -View [Flux's aggregate functions](#) for more information and ideas. +View [Flux's aggregate functions](/v2.0/reference/flux/functions/transformations/aggregates/) +for more information and ideas. ## Create a destination bucket By design, tasks cannot write to the same bucket from which they are reading. @@ -45,8 +46,8 @@ The example task script below is a very basic form of data downsampling that doe 1. Defines a task named "cq-mem-data-1w" that runs once a week. 2. Defines a `data` variable that represents all data from the last 2 weeks in the `mem` measurement of the `system-data` bucket. -3. Uses the [`aggregateWindow()` function](#) to window the data into 1 hour intervals - and calculate the average of each interval. +3. Uses the [`aggregateWindow()` function](/v2.0/reference/flux/functions/transformations/aggregates/aggregatewindow/) + to window the data into 1 hour intervals and calculate the average of each interval. 4. Stores the aggregated data in the `system-data-downsampled` bucket under the `my-org` organization. diff --git a/content/v2.0/process-data/write-a-task.md b/content/v2.0/process-data/write-a-task.md index acfb31a13..3c0315297 100644 --- a/content/v2.0/process-data/write-a-task.md +++ b/content/v2.0/process-data/write-a-task.md @@ -51,7 +51,8 @@ in form fields when creating the task. {{% /note %}} ## Define a data source -Define a data source using Flux's [`from()` function](#) or any other [Flux input functions](#). +Define a data source using Flux's [`from()` function](/v2.0/reference/flux/functions/inputs/from/) +or any other [Flux input functions](/v2.0/reference/flux/functions/inputs/). For convenience, consider creating a variable that includes the sourced data with the required time range and any relevant filters. @@ -84,7 +85,7 @@ specific use case. The example below illustrates a task that downsamples data by calculating the average of set intervals. It uses the `data` variable defined [above](#define-a-data-source) as the data source. It then windows the data into 5 minute intervals and calculates the average of each -window using the [`aggregateWindow()` function](#). +window using the [`aggregateWindow()` function](/v2.0/reference/flux/functions/transformations/aggregates/aggregatewindow/). ```js data @@ -100,7 +101,8 @@ _See [Common tasks](/v2.0/process-data/common-tasks) for examples of tasks commo In the vast majority of task use cases, once data is transformed, it needs to sent and stored somewhere. This could be a separate bucket with a different retention policy, another measurement, or even an alert endpoint _(Coming)_. -The example below uses Flux's [`to()` function](#) to send the transformed data to another bucket: +The example below uses Flux's [`to()` function](/v2.0/reference/flux/functions/outputs/to) +to send the transformed data to another bucket: ```js // ...