Influxdata
/
docs-v2
mirror of https://github.com/influxdata/docs-v2.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

chore(telegraf): Update input and output data formats from Telegraf docs (Closes #5054): (#5072) 

* Telegraf parser docs aren't consistent with the available parsers. Include parser docs from Telegraf repo, separately from data formats.
 #5054

* chore(telegraf): Update input and output data formats from Telegraf docs (Closes #5054):

- Update input data formats with latest from Telegraf docs.
- Revise output data format descriptions and frontmatter.
- Trying not to change too much yet; ideally, we'd settle on style for Telegraf docs and only pull from there.

* fix(telegraf): set list titles, fix frontmatter

* Delete accidentally committed file

* Update content/telegraf/v1.27/configure_plugins/template-patterns.md

Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>

* Update content/telegraf/v1.27/data_formats/input/xpath_json.md

Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>

* Update content/telegraf/v1.27/configure_plugins/template-patterns.md

Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>

* Update content/telegraf/v1.27/data_formats/input/xpath_json.md

Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>

* Update content/telegraf/v1.27/data_formats/input/xpath_json.md

Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>

* Update content/telegraf/v1.27/data_formats/input/xpath_json.md

Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>

* Update content/telegraf/v1.27/data_formats/input/xpath_json.md

Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>

* Update content/telegraf/v1.27/configure_plugins/template-patterns.md

Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>

* Update content/telegraf/v1.27/configure_plugins/template-patterns.md

Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>

* Update content/telegraf/v1.27/configure_plugins/template-patterns.md

Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>

* Update content/telegraf/v1.27/configure_plugins/template-patterns.md

Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>

* Update content/telegraf/v1.27/configure_plugins/template-patterns.md

Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>

* fix(telegraf): configuration cleanup and apply suggestions

---------

Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
pull/5081/head
Jason Stirnaman 2023-08-08 11:03:42 -05:00 committed by GitHub
parent 0a5c0b9bea
commit 4b6494969b
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
37 changed files with 3126 additions and 670 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
.ci/vale/styles/Vocab/InfluxData/accept.txt
View File

@ -4,6 +4,7 @@ APIs?
Arrow
authToken
AuthToken
[Bb]oolean
bundlers?
[Cc]hronograf
CLI
@ -38,6 +39,7 @@ IOx
Kapacitor
lat
locf
[Ll]ogstash
lon
namespace
noaa
@ -49,7 +51,9 @@ pyinflux
rearchitect(ed)?
retention_policy
rp
serializer
[Ss]erverless
Splunk
stdout
Superset
svg
@ -57,4 +61,6 @@ svg
[Tt]ombstoned
unescaped
venv
Webpack
Webpack
xpath
XPath

1
content/telegraf/v1.27/commands.md
View File

@ -3,7 +3,6 @@ title: Telegraf commands and flags
description: The `telegraf` command starts and runs all the processes necessary for Telegraf to function.
menu:
telegraf_1_27_ref:
name: Commands
weight: 25
---

162
content/telegraf/v1.27/configuration.md
View File

@ -5,14 +5,14 @@ aliases:
- /telegraf/v1.27/administration/configuration/
menu:
telegraf_1_27_ref:
name: Configuration options
weight: 40
---
The Telegraf configuration file (`telegraf.conf`) lists all available Telegraf plugins. See the current version here: [telegraf.conf](https://github.com/influxdata/telegraf/blob/master/etc/telegraf.conf).
The `telegraf.conf` Telegraf configuration file lists all available Telegraf plugins.
See the [current version](https://github.com/influxdata/telegraf/blob/master/etc/telegraf.conf).
> To quickly get started with Telegraf, see [Get started](/telegraf/v1.27/get_started/).
> See [Get started](/telegraf/v1.27/get_started/) to quickly get up and running with Telegraf.
## Generate a configuration file
@ -51,11 +51,11 @@ For numbers and Booleans, variables must be unquoted (for example, `${INT_VAR}`,
You can also set environment variables using the Linux `export` command: `export password=mypassword`
> **Note:** We recommend using environment variables for sensitive information.
> **Note:** Use a secret store or environment variables to store sensitive credentials.
### Example: Telegraf environment variables
In the Telegraf environment variables file (`/etc/default/telegraf`):
Set environment variables in the Telegraf environment variables file (`/etc/default/telegraf`)--for example:
```sh
USER="alice"
@ -64,7 +64,7 @@ INFLUX_SKIP_DATABASE_CREATION="true"
INFLUX_PASSWORD="monkey123"
```
In the Telegraf configuration file (`/etc/telegraf.conf`):
In the Telegraf configuration file (`/etc/telegraf.conf`), reference the variables--for example:
```sh
[global_tags]
@ -78,7 +78,7 @@ In the Telegraf configuration file (`/etc/telegraf.conf`):
password = "${INFLUX_PASSWORD}"
```
The environment variables above add the following configuration settings to Telegraf:
When Telegraf runs, the effective configuration is the following:
```sh
[global_tags]
@ -93,51 +93,54 @@ The environment variables above add the following configuration settings to Tele
## Global tags
Global tags can be specified in the `[global_tags]` section of the config file
in `key="value"` format. All metrics being gathered on this host will be tagged
with the tags specified here.
Global tags can be specified in the `[global_tags]` section of the configuration file
in `key="value"` format.
Telegraf applies the global tags to all metrics gathered on this host.
## Agent configuration
Telegraf has a few options you can configure under the `[agent]` section of the
config.
The `[agent]` section contains the following configuration options:
* **interval**: Default data collection interval for all inputs
* **round_interval**: Rounds collection interval to `interval`.
For example, if `interval` is set to 10s then always collect on :00, :10, :20, etc.
* **metric_batch_size**: Telegraf will send metrics to output in batch of at
For example, if `interval` is set to `10s`, then the agent collects on :00, :10, :20, etc.
* **metric_batch_size**: Sends metrics to the output in batches of at
most `metric_batch_size` metrics.
* **metric_buffer_limit**: Telegraf will cache `metric_buffer_limit` metrics
for each output, and will flush this buffer on a successful write.
* **metric_buffer_limit**: Caches `metric_buffer_limit` metrics
for each output, and flushes this buffer on a successful write.
This should be a multiple of `metric_batch_size` and could not be less
than 2 times `metric_batch_size`.
* **collection_jitter**: Collection jitter is used to jitter
* **collection_jitter**: Used to jitter
the collection by a random amount.
Each plugin will sleep for a random time within jitter before collecting.
This can be used to avoid many plugins querying things like sysfs at the
Each plugin sleeps for a random time within jitter before collecting.
This can be used to avoid many plugins querying things like **sysfs** at the
same time, which can have a measurable effect on the system.
* **flush_interval**: Default data flushing interval for all outputs.
You should not set this below `interval`.
Maximum `flush_interval` will be `flush_interval` + `flush_jitter`
Don't set this below `interval`.
Maximum `flush_interval` is `flush_interval` + `flush_jitter`
* **flush_jitter**: Jitter the flush interval by a random amount.
This is primarily to avoid
large write spikes for users running a large number of Telegraf instances.
For example, a `flush_jitter` of 5s and `flush_interval` of 10s means flushes will happen every 10-15s.
For example, a `flush_jitter` of `5s` and `flush_interval` of `10s` means flushes happen every 10-15s.
* **precision**: Collected metrics are rounded to the precision specified as an
`interval` (integer + unit, ex: `1ns`, `1us`, `1ms`, and `1s` . Precision will NOT
be used for service inputs, such as `logparser` and `statsd`.
`interval` (integer + unit, ex: `1ns`, `1us`, `1ms`, and `1s` . Precision isn't
used for service inputs, such as `logparser` and `statsd`.
* **debug**: Run Telegraf in debug mode.
* **quiet**: Run Telegraf in quiet mode (error messages only).
* **logtarget**: Controls the destination for logs and can be set to “file”, “stderr”, or, on Windows, “eventlog”. When set to “file”, the output file is determined by the logfile setting.
* **logfile**: If logtarget is set to “file” specify the logfile name. If set to the empty string then logs are written to stderr.
* **logfile_rotation_interval**: Rotates logfile after the time interval specified. When
set to 0 no time based rotation is performed.
* **logtarget**: Controls the destination for logs and can be set to `"file"`, `"stderr"`, or, on Windows, `"eventlog"`.
When set to `"file"`, the output file is determined by the logfile setting.
* **logfile**: If logtarget is set to `“file”`, specify the logfile name. If set to an empty string, then logs are written to stderr.
* **logfile_rotation_interval**: Rotates the logfile after the time interval specified.
When set to `0`, no time-based rotation is performed.
* **logfile_rotation_max_size**: Rotates logfile when it becomes larger than the specified
size. When set to 0 no size based rotation is performed.
size.
When set to `0`, no size-based rotation is performed.
* **logfile_rotation_max_archives**: Maximum number of rotated archives to keep, any
older logs are deleted. If set to -1, no archives are removed.
* **log_with_timezone**: Set a timezone to use when logging or type 'local' for local time. Example: 'America/Chicago'.
[See this page for options/formats.](https://socketloop.com/tutorials/golang-display-list-of-timezones-with-gmt)
older logs are deleted.
If set to `-1`, no archives are removed.
* **log_with_timezone**: Set a timezone to use when logging--for example, `"America/Chicago"`.
To use local time, set to `"local"`.
See [timezone options and formats](https://socketloop.com/tutorials/golang-display-list-of-timezones-with-gmt).
* **hostname**: Override default hostname, if empty use `os.Hostname()`.
* **omit_hostname**: If true, do no set the `host` tag in the Telegraf agent.
@ -184,18 +187,18 @@ The following config parameters are available for all aggregators:
* **alias**: Name an instance of a plugin.
* **period**: The period on which to flush & clear each aggregator. All metrics
that are sent with timestamps outside of this period will be ignored by the
that are sent with timestamps outside of this period are ignored by the
aggregator.
* **delay**: The delay before each aggregator is flushed. This is to control
how long for aggregators to wait before receiving metrics from input plugins,
in the case that aggregators are flushing and inputs are gathering on the
same interval.
* **grace**: The duration the metrics will still be aggregated by the plugin
even though they're outside of the aggregation period. This setting is needed
in a situation when the agent is expected to receive late metrics and can
be rolled into next aggregation period.
* **drop_original**: If true, the original metric will be dropped by the
aggregator and will not get sent to the output plugins.
* **grace**: The duration the metrics are aggregated by the plugin
even though they're outside the aggregation period.
This setting is needed when the agent is expected to receive late metrics and can
be rolled into the next aggregation period.
* **drop_original**: If true, the original metric is dropped by the
aggregator and not sent to the output plugins.
* **name_override**: Override the base name of the measurement.
(Default is the name of the input).
* **name_prefix**: Specifies a prefix to attach to the measurement name.
@ -211,65 +214,67 @@ For a demonstration of how to configure SNMP, MQTT, and PostGRE SQL plugins to g
The following config parameters are available for all processors:
* **alias**: Name an instance of a plugin.
* **order**: This is the order in which processors are executed. If this
is not specified, then processor execution order will be random.
* **order**: This is the order in which processors are executed.
If not specified, then order is random.
The [metric filtering][] parameters can be used to limit what metrics are
handled by the processor. Excluded metrics are passed downstream to the next
processor.
The [metric filtering](#metric-filtering) parameters can be used to limit what metrics are
handled by the processor.
Excluded metrics are passed downstream to the next processor.
## Metric filtering
Filters can be configured per input, output, processor, or aggregator,
see below for examples.
Filters can be configured per input, output, processor, or aggregator.
- [Filters](#filters)
- [Filtering examples](#filter-examples)
### Filters
* **namepass**:
An array of glob pattern strings. Only points whose measurement name matches
a pattern in this list are emitted.
An array of glob pattern strings.
Only emits points whose measurement name matches a pattern in this list.
* **namedrop**:
The inverse of `namepass`. If a match is found the point is discarded. This
is tested on points after they have passed the `namepass` test.
The inverse of `namepass`.
Discards points whose measurement name matches a pattern in this list.
This test applies to points _after_ they have passed the `namepass` test.
* **fieldpass**:
An array of glob pattern strings. Only fields whose field key matches a
pattern in this list are emitted.
An array of glob pattern strings.
Only emits fields whose field key matches a pattern in this list.
* **fielddrop**:
The inverse of `fieldpass`. Fields with a field key matching one of the
patterns will be discarded from the point.
The inverse of `fieldpass`.
Discards fields that have a field key matching one of the patterns.
* **tagpass**:
A table mapping tag keys to arrays of glob pattern strings. Only points
that contain a tag key in the table and a tag value matching one of its
patterns is emitted.
A table that maps tag keys to arrays of glob pattern strings.
Only emits points that contain a tag key in the table and a tag value that matches one of the associated patterns.
* **tagdrop**:
The inverse of `tagpass`. If a match is found the point is discarded. This
is tested on points after they have passed the `tagpass` test.
The inverse of `tagpass`.
Discards points that contain a tag key in the table and a tag value that matches one of the associated patterns.
This test applies to points _after_ they have passed the `tagpass` test.
* **taginclude**:
An array of glob pattern strings. Only tags with a tag key matching one of
the patterns are emitted. In contrast to `tagpass`, which will pass an entire
point based on its tag, `taginclude` removes all non matching tags from the
point. This filter can be used on both inputs & outputs, but it is
_recommended_ to be used on inputs, as it is more efficient to filter out tags
at the ingestion point.
An array of glob pattern strings.
Only tags with a tag key matching one of the patterns are emitted.
In contrast to `tagpass`, which emits an entire
point if a tag passes, `taginclude` removes all non-matching tags from the
point. This filter can be used on inputs and outputs, but is more efficient when used on inputs (filtering out tags is more efficient during ingestion).
* **tagexclude**:
The inverse of `taginclude`. Tags with a tag key matching one of the patterns
will be discarded from the point.
are discarded from the point.
**NOTE** Due to the way TOML is parsed, `tagpass` and `tagdrop` parameters
must be defined at the _end_ of the plugin definition, otherwise subsequent
plugin config options will be interpreted as part of the tagpass/tagdrop
plugin configuration options are interpreted as part of the tagpass and tagdrop
tables.
To learn more about metric filtering, watch the following video:
{{< youtube R3DnObs_OKA >}}
## Examples
## Filtering examples
#### Input configuration examples
This is a full working config that will output CPU data to an InfluxDB instance
at `192.168.59.103:8086`, tagging measurements with `dc="denver-1"`. It will output
measurements at a 10s interval and will collect per-cpu data, dropping any
fields which begin with `time_`.
The following example configuration collects per-cpu data, drops any
fields that begin with `time_`, tags measurements with `dc="denver-1"`, and then outputs measurements at a 10 s interval to an InfluxDB database named `telegraf` at the address `192.168.59.103:8086`.
```toml
[global_tags]
@ -295,8 +300,7 @@ fields which begin with `time_`.
#### Input Config: `tagpass` and `tagdrop`
**NOTE** `tagpass` and `tagdrop` parameters must be defined at the _end_ of
the plugin definition, otherwise subsequent plugin config options will be
interpreted as part of the tagpass/tagdrop map.
the plugin definition, otherwise subsequent plugin configuration options are interpreted as part of the tagpass and tagdrop tables.
```toml
[[inputs.cpu]]
@ -361,7 +365,7 @@ interpreted as part of the tagpass/tagdrop map.
#### Input config: `prefix`, `suffix`, and `override`
This plugin will emit measurements with the name `cpu_total`.
The following example emits measurements with the name `cpu_total`:
```toml
[[inputs.cpu]]
@ -370,7 +374,7 @@ This plugin will emit measurements with the name `cpu_total`.
totalcpu = true
```
This will emit measurements with the name `foobar`.
The following example emits measurements with the name `foobar`:
```toml
[[inputs.cpu]]
@ -381,10 +385,10 @@ This will emit measurements with the name `foobar`.
#### Input config: tags
This plugin will emit measurements with two additional tags: `tag1=foo` and
The following example emits measurements with two additional tags: `tag1=foo` and
`tag2=bar`.
NOTE: Order matters, the `[inputs.cpu.tags]` table must be at the _end_ of the
NOTE: Order matters; the `[inputs.cpu.tags]` table must be at the _end_ of the
plugin definition.
```toml
@ -398,7 +402,7 @@ plugin definition.
#### Multiple inputs of the same type
Additional inputs (or outputs) of the same type can be specified by defining these instances in the configuration file. To avoid measurement collisions, use the `name_override`, `name_prefix`, or `name_suffix` config options:
Additional inputs (or outputs) of the same type can be specified by defining these instances in the configuration file. To avoid measurement collisions, use the `name_override`, `name_prefix`, or `name_suffix` configuration options:
```toml
[[inputs.cpu]]

143
content/telegraf/v1.27/configure_plugins/template-patterns.md Normal file
View File

@ -0,0 +1,143 @@
---
title: Telegraf template patterns
description: |
Template patterns describe how a dot-delimited string should be mapped to
and from Telegraf metrics.
menu:
telegraf_1_27_ref:
weight: 10
name: Template patterns
---
Template patterns describe how a dot-delimited string should be mapped to
and from Telegraf [metrics](/telegraf/v1.27/metrics/).
A template has the form:
```text
"host.mytag.mytag.measurement.measurement.field*"
```
Where the following keywords can be set:
- `measurement`: specifies that this section of the graphite bucket corresponds
to the measurement name. This can be specified multiple times.
- `field`: specifies that this section of the graphite bucket corresponds
to the field name. This can be specified multiple times.
- `measurement*`: specifies that all remaining elements of the graphite bucket
correspond to the measurement name.
- `field*`: specifies that all remaining elements of the graphite bucket
correspond to the field name.
Any part of the template that is not a keyword is treated as a tag key. This
can also be specified multiple times.
**Note the following:**
- `measurement` must be specified in your template.
- `field*` cannot be used in conjunction with `measurement*`.
## Examples
### Measurement and tag templates
A basic template specifies a single transformation to apply to all
incoming metrics:
```toml
templates = [
"region.region.measurement*"
]
```
This results in the following Graphite to Telegraf metric transformation.
```text
us.west.cpu.load 100
=> cpu.load,region=us.west value=100
```
You can specify multiple templates and differentiate them using _[filters](#filter-templates)_.
```toml
templates = [
"*.*.* region.region.measurement", # All 3-part measurements will match this one.
"*.*.*.* region.region.host.measurement", # All 4-part measurements will match this one.
]
```
### Field templates
The `field` keyword tells Telegraf to give the metric that field name.
```toml
separator = "_"
templates = [
"measurement.measurement.field.field.region"
]
```
This results in the following Graphite to Telegraf metric transformation.
```text
cpu.usage.idle.percent.eu-east 100
=> cpu_usage,region=eu-east idle_percent=100
```
The field key can also be derived from all remaining elements of the graphite
bucket by specifying `field*`:
```toml
separator = "_"
templates = [
"measurement.measurement.region.field*"
]
```
This results in the following Graphite to Telegraf metric transformation.
```text
cpu.usage.eu-east.idle.percentage 100
=> cpu_usage,region=eu-east idle_percentage=100
```
### Filter templates
Use glob matching to filter templates to use based on the name of the bucket:
```toml
templates = [
"cpu.* measurement.measurement.region",
"mem.* measurement.measurement.host"
]
```
This results in the following transformation:
```text
cpu.load.eu-east 100
=> cpu_load,region=eu-east value=100
mem.cached.localhost 256
=> mem_cached,host=localhost value=256
```
### Add tags
To add additional tags to a metric, include them after the template pattern
using the InfluxDB line protocol tag format (comma-separated key-value pairs).
```toml
templates = [
"measurement.measurement.field.region datacenter=1a"
]
```
This results in the following Graphite to Telegraf metric transformation.
```text
cpu.usage.idle.eu-east 100
=> cpu_usage,region=eu-east,datacenter=1a idle=100
```
[metrics]: /docs/METRICS.md

1
content/telegraf/v1.27/data_formats/_index.md
View File

@ -1,5 +1,6 @@
---
title: Telegraf data formats
list_title: Data formats
description: Telegraf supports input data formats and output data formats for converting input and output data.
menu:
telegraf_1_27_ref:

13
content/telegraf/v1.27/data_formats/input/_index.md
View File

@ -1,16 +1,18 @@
---
title: Telegraf input data formats
list_title: Input data formats
description: Telegraf supports parsing input data formats into Telegraf metrics.
menu:
telegraf_1_27_ref:
name: Input data formats
weight: 1
parent: Data formats
---
Telegraf contains many general purpose plugins that use a configurable parser for parsing input data into metrics.
This allows input plugins such as the [`kafka_consumer` plugin](/telegraf/v1.27/plugins/#input-kafka_consumer)
Telegraf [input plugins](/telegraf/v1.27/plugins/inputs/) consume data in one or more data formats and
parse the data into Telegraf [metrics][/telegraf/v1.27/metrics/].
Many input plugins use configurable parsers for parsing data formats into metrics.
This allows input plugins such as [`kafka_consumer` input plugin](/telegraf/v1.27/plugins/#input-kafka_consumer)
to consume and process different data formats, such as InfluxDB line
protocol or JSON.
Telegraf supports the following input **data formats**:
@ -35,4 +37,9 @@ desired parser:
data_format = "json_v2"
```
## Input parser plugins
When you specify a `data_format` in an [input plugin](/telegraf/v1.27/plugins/inputs/) configuration that supports it, the input plugin uses the associated [parser plugin](https://github.com/influxdata/telegraf/tree/master/plugins/parsers) to convert data from its source format into Telegraf metrics.
Many parser plugins provide additional configuration options for specifying details about your data schema and how it should map to fields in Telegraf metrics.
[metrics]: /telegraf/v1.27/metrics/

6
content/telegraf/v1.27/data_formats/input/avro.md
View File

@ -1,14 +1,16 @@
---
title: Avro input data format
description: Use the `avro` input data format to parse metrics from a message serialized as Avro binary or JSON format.
list_title: Avro
description: Use the `avro` input data format to parse Avro binary or JSON data into Telegraf metrics.
menu:
telegraf_1_27_ref:
name: Avro
weight: 10
parent: Input data formats
metadata: [Avro Parser Plugin]
---
The Avro input data format parses messages serialized as [Avro](https://avro.apache.org/) format and encoded as binary or JSON.
Use the `avro` input data format to parse binary or JSON [Avro](https://avro.apache.org/) message data into Telegraf metrics.
## Wire format

355
content/telegraf/v1.27/data_formats/input/binary.md Normal file
View File

@ -0,0 +1,355 @@
---
title: Binary input data format
list_title: Binary
description:
Use the `binary` input data format with user-specified configurations to parse binary protocols into Telegraf metrics.
menu:
telegraf_1_27_ref:
name: Binary
weight: 10
parent: Input data formats
metadata: [Binary Parser Plugin]
---
Use the `binary` input data format with user-specified configurations to parse binary protocols into Telegraf metrics.
## Configuration
```toml
[[inputs.file]]
files = ["example.bin"]
## Data format to consume.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "binary"
## Do not error-out if none of the filter expressions below matches.
# allow_no_match = false
## Specify the endianness of the data.
## Available values are "be" (big-endian), "le" (little-endian) and "host",
## where "host" means the same endianness as the machine running Telegraf.
# endianess = "host"
## Interpret input as string containing hex-encoded data.
# hex_encoding = false
## Multiple parsing sections are allowed
[[inputs.file.binary]]
## Optional: Metric (measurement) name to use if not extracted from the data.
# metric_name = "my_name"
## Definition of the message format and the extracted data.
## Please note that you need to define all elements of the data in the
## correct order with the correct length as the data is parsed in the order
## given.
## An entry can have the following properties:
## name -- Name of the element (e.g. field or tag). Can be omitted
## for special assignments (i.e. time & measurement) or if
## entry is omitted.
## type -- Data-type of the entry. Can be "int8/16/32/64", "uint8/16/32/64",
## "float32/64", "bool" and "string".
## In case of time, this can be any of "unix" (default), "unix_ms", "unix_us",
## "unix_ns" or a valid Golang time format.
## bits -- Length in bits for this entry. If omitted, the length derived from
## the "type" property will be used. For "time" 64-bit will be used
## as default.
## assignment -- Assignment of the gathered data. Can be "measurement", "time",
## "field" or "tag". If omitted "field" is assumed.
## omit -- Omit the given data. If true, the data is skipped and not added
## to the metric. Omitted entries only need a length definition
## via "bits" or "type".
## terminator -- Terminator for dynamic-length strings. Only used for "string" type.
## Valid values are "fixed" (fixed length string given by "bits"),
## "null" (null-terminated string) or a character sequence specified
## as HEX values (e.g. "0x0D0A"). Defaults to "fixed" for strings.
## timezone -- Timezone of "time" entries. Only applies to "time" assignments.
## Can be "utc", "local" or any valid Golang timezone (e.g. "Europe/Berlin")
entries = [
{ type = "string", assignment = "measurement", terminator = "null" },
{ name = "address", type = "uint16", assignment = "tag" },
{ name = "value", type = "float64" },
{ type = "unix", assignment = "time" },
]
## Optional: Filter evaluated before applying the configuration.
## This option can be used to mange multiple configuration specific for
## a certain message type. If no filter is given, the configuration is applied.
# [inputs.file.binary.filter]
# ## Filter message by the exact length in bytes (default: N/A).
# # length = 0
# ## Filter the message by a minimum length in bytes.
# ## Messages longer of of equal length will pass.
# # length_min = 0
# ## List of data parts to match.
# ## Only if all selected parts match, the configuration will be
# ## applied. The "offset" is the start of the data to match in bits,
# ## "bits" is the length in bits and "match" is the value to match
# ## against. Non-byte boundaries are supported, data is always right-aligned.
# selection = [
# { offset = 0, bits = 8, match = "0x1F" },
# ]
#
#
```
In this configuration mode, you explicitly specify the field and tags
to parse from your data.
A configuration can contain multiple `binary` subsections.
For example, the `file` plugin can process binary data multiple times.
This can be useful (together with _filters_) to handle different message types.
**Note**: The `filter` section needs to be placed _after_ the `entries`
definitions, otherwise the entries will be assigned to the filter section.
### General options and remarks
#### `allow_no_match` (optional)
By specifying `allow_no_match` you allow the parser to silently ignore data
that does not match _any_ given configuration filter. This can be useful if
you only want to collect a subset of the available messages.
#### `endianness` (optional)
This specifies the endianness of the data. If not specified, the parser will
fall back to the "host" endianness, assuming that the message and Telegraf
machine share the same endianness.
Alternatively, you can explicitly specify big-endian format (`"be"`) or
little-endian format (`"le"`).
#### `hex_encoding` (optional)
If `true`, the input data is interpreted as a string containing hex-encoded
data like `C0 C7 21 A9`. The value is _case insensitive_ and can handle spaces,
however prefixes like `0x` or `x` are _not_ allowed.
### Non-byte aligned value extraction
In both, `filter` and `entries` definitions, values can be extracted at non-byte
boundaries. You can for example extract 3-bit starting at bit-offset 8. In those
cases, the result will be masked and shifted such that the resulting byte-value
is _right_ aligned. In case your 3-bit are `101` the resulting byte value is
`0x05`.
This is especially important when specifying the `match` value in the filter
section.
### Entries definitions
The `entries` array specifies how to parse the message into the measurement name,
timestamp, tags, and fields.
#### `measurement` specification
When setting the `assignment` to `"measurement"`, the extracted value
is used as the metric name, overriding other specifications.
The `type` setting is assumed to be `"string"` and can be omitted similar
to the `name` option. See [`string` type handling](#string-type-handling)
for details and further options.
#### `time` specification
When setting the `assignment` to `"time"`, the extracted value
is used as the timestamp of the metric. The default is the _current
time_ for all created metrics.
The `type` setting specifies the time-format of included timestamps.
Use one of the following:
- `unix` _(default)_
- `unix_ms`
- `unix_us`
- `unix_ns`
- [Go "reference time"][time const]. Consult the Go [time][time parse]
package for details and additional examples on how to set the time format.
For the `unix` format and derivatives, the underlying value is assumed
to be a 64-bit integer. The `bits` setting can be used to specify other
length settings. All other time-formats assume a fixed-length `string`
value to be extracted. The length of the string is automatically
determined using the format setting in `type`.
The `timezone` setting converts the extracted time to the
given value timezone. By default, the time will be interpreted as `utc`.
Other valid values are `local` (the local timezone configured for
the machine), or valid timezone-specification (for example,`Europe/Berlin`).
### `tag` specification
When setting the `assignment` to `"tag"`, the extracted value
is used as a tag. The `name` setting is the name of the tag
and the `type` defaults to `string`. When specifying other types,
the extracted value is first interpreted as the given type and
then converted to `string`.
The `bits` setting can be used to specify the length of the data to
extract and is required for fixed-length `string` types.
### `field` specification
When setting the `assignment` to `"field"` or omitting the `assignment`
setting, the extracted value is used as a field. The `name` setting
is used as the name of the field and the `type` as the type of the field value.
The `bits` setting can be used to specify the length of the data to
extract. By default the length corresponding to `type` is used.
Please see the [string](#string-type-handling) and [bool](#bool-type-handling)
specific sections when using those types.
### `string` type handling
Strings are assumed to be fixed-length strings by default. In this case, the
`bits` setting is mandatory to specify the length of the string in _bit_.
To handle dynamic strings, the `terminator` setting can be used to specify
characters to terminate the string. The two named options, `fixed` and `null`
specify fixed-length and null-terminated strings, respectively.
Any other setting is interpreted as a hexadecimal sequence of bytes
matching the end of the string. The termination-sequence is removed from
the result.
### `bool` type handling
By default, `bool` types are assumed to be _one_ bit in length. You can
specify any other length by using the `bits` setting.
When interpreting values as booleans, any zero value is `false` and
any non-zero value is `true`.
### omitting data
Parts of the data can be omitted by setting `omit = true`. In this case,
you only need to specify the length of the chunk to omit by either using
the `type` or `bits` setting. All other options can be skipped.
### Filter definitions
Filters can be used to match the length or the content of the data against
a specified reference. See the [examples section](#examples) for details.
You can also check multiple parts of the message by specifying multiple
`section` entries for a filter. Each `section` is then matched separately.
All have to match to apply the configuration.
#### `length` and `length_min` options
Using the `length` option, the filter checks if the parsed data has
exactly the given number of _bytes_. Otherwise, the configuration is not applied.
Similarly, for `length_min` the data has to have _at least_ the given number
of _bytes_ to generate a match.
#### `selection` list
Selections can be used with or without length constraints to match the content
of the data. Here, the `offset` and `bits` properties specify the start
and length of the data to check. Both values are in _bit_ allowing for non-byte
aligned value extraction. The extracted data is checked against the
given `match` value specified in HEX.
If multiple `selection` entries are specified _all_ of the selections must
match for the configuration to get applied.
## Examples
In the following example, we use a binary protocol with three different messages
in little-endian format
### Message A definition
```text
+--------+------+------+--------+--------+------------+--------------------+--------------------+
| ID | type | len | addr | count | failure | value | timestamp |
+--------+------+------+--------+--------+------------+--------------------+--------------------+
| 0x0201 | 0x0A | 0x18 | 0x7F01 | 0x2A00 | 0x00000000 | 0x6F1283C0CA210940 | 0x10D4DF6200000000 |
+--------+------+------+--------+--------+------------+--------------------+--------------------+
```
### Message B definition
```text
+--------+------+------+------------+
| ID | type | len | value |
+--------+------+------+------------+
| 0x0201 | 0x0B | 0x04 | 0xDEADC0DE |
+--------+------+------+------------+
```
### Message C definition
```text
+--------+------+------+------------+------------+--------------------+
| ID | type | len | value x | value y | timestamp |
+--------+------+------+------------+------------+--------------------+
| 0x0201 | 0x0C | 0x10 | 0x4DF82D40 | 0x5F305C08 | 0x10D4DF6200000000 |
+--------+------+------+------------+------------+--------------------+
```
All messages consists of a 4-byte header containing the _message type_
in the 3rd byte and a message specific body. To parse those messages
you can use the following configuration:
```toml
[[inputs.file]]
files = ["messageA.bin", "messageB.bin", "messageC.bin"]
data_format = "binary"
endianess = "le"
[[inputs.file.binary]]
metric_name = "messageA"
entries = [
{ bits = 32, omit = true },
{ name = "address", type = "uint16", assignment = "tag" },
{ name = "count", type = "int16" },
{ name = "failure", type = "bool", bits = 32, assignment = "tag" },
{ name = "value", type = "float64" },
{ type = "unix", assignment = "time" },
]
[inputs.file.binary.filter]
selection = [{ offset = 16, bits = 8, match = "0x0A" }]
[[inputs.file.binary]]
metric_name = "messageB"
entries = [
{ bits = 32, omit = true },
{ name = "value", type = "uint32" },
]
[inputs.file.binary.filter]
selection = [{ offset = 16, bits = 8, match = "0x0B" }]
[[inputs.file.binary]]
metric_name = "messageC"
entries = [
{ bits = 32, omit = true },
{ name = "x", type = "float32" },
{ name = "y", type = "float32" },
{ type = "unix", assignment = "time" },
]
[inputs.file.binary.filter]
selection = [{ offset = 16, bits = 8, match = "0x0C" }]
```
The above configuration has one `[[inputs.file.binary]]` section per
message type and uses a filter in each of those sections to apply
the correct configuration by comparing the 3rd byte (containing
the message type). This results in the following output:
```text
metricA,address=383,failure=false count=42i,value=3.1415 1658835984000000000
metricB value=3737169374i 1658847037000000000
metricC x=2.718280076980591,y=0.0000000000000000000000000000000006626070178575745 1658835984000000000
```
`metricB` uses the parsing time as timestamp due to missing
information in the data. The other two metrics use the timestamp
derived from the data.
[time const]: https://golang.org/pkg/time/#pkg-constants
[time parse]: https://golang.org/pkg/time/#Parse

42
content/telegraf/v1.27/data_formats/input/collectd.md
View File

@ -1,32 +1,39 @@
---
title: Collectd input data format
description: Use the `collectd` input data format to parse the collectd network binary protocol to create tags for host, instance, type, and type instance.
list_title: Collectd
description: Use the `collectd` input data format to parse collectd network binary protocol to create tags for host, instance, type, and type instance.
menu:
telegraf_1_27_ref:
name: collectd
weight: 10
parent: Input data formats
metadata: [Collectd Parser Plugin]
---
The collectd input data format parses the collectd network binary protocol to create tags for host, instance, type, and type instance. All collectd values are added as float64 fields.
Use the `collectd` input data format to parse [collectd binary network protocol](https://collectd.org/wiki/index.php/Binary_protocol) data into Telegraf metrics.
Tags are
created for host, instance, type, and type instance.
All collectd values are added as float64 fields.
For more information, see [binary protocol](https://collectd.org/wiki/index.php/Binary_protocol) in the collectd Wiki.
For more information about the binary network protocol see
[here](https://collectd.org/wiki/index.php/Binary_protocol).
You can control the cryptographic settings with parser options.
Create an authentication file and set `collectd_auth_file` to the path of the file, then set the desired security level in `collectd_security_level`.
You can control the cryptographic settings with parser options. Create an
authentication file and set `collectd_auth_file` to the path of the file, then
set the desired security level in `collectd_security_level`.
For more information, including client setup, see
[Cryptographic setup](https://collectd.org/wiki/index.php/Networking_introduction#Cryptographic_setup) in the collectd Wiki.
Additional information including client setup can be found [here][1].
You can also change the path to the typesdb or add additional typesdb using
`collectd_typesdb`.
[1]: https://collectd.org/wiki/index.php/Networking_introduction#Cryptographic_setup
## Configuration
```toml
[[inputs.file]]
files = ["example"]
[[inputs.socket_listener]]
service_address = "udp://:25826"
## Data format to consume.
## Each data format has its own unique set of configuration options, read
@ -44,6 +51,19 @@ You can also change the path to the typesdb or add additional typesdb using
## Multi-value plugins can be handled two ways.
## "split" will parse and store the multi-value plugin data into separate measurements
## "join" will parse and store the multi-value plugin as a single multi-value measurement.
## "split" is the default behavior for backward compatability with previous versions of influxdb.
## "split" is the default behavior for backward compatibility with previous versions of InfluxDB.
collectd_parse_multivalue = "split"
```
## Example Output
```text
memory,type=memory,type_instance=buffered value=2520051712 1560455990829955922
memory,type=memory,type_instance=used value=3710791680 1560455990829955922
memory,type=memory,type_instance=buffered value=2520047616 1560455980830417318
memory,type=memory,type_instance=cached value=9472626688 1560455980830417318
memory,type=memory,type_instance=slab_recl value=2088894464 1560455980830417318
memory,type=memory,type_instance=slab_unrecl value=146984960 1560455980830417318
memory,type=memory,type_instance=free value=2978258944 1560455980830417318
memory,type=memory,type_instance=used value=3707047936 1560455980830417318
```

116
content/telegraf/v1.27/data_formats/input/csv.md
View File

@ -1,15 +1,16 @@
---
title: CSV input data format
description: Use the `csv` input data format to parse a document containing comma-separated values into Telegraf metrics.
list_title: CSV
description: Use the `csv` input data format to parse comma-separated values into Telegraf metrics.
menu:
telegraf_1_27_ref:
name: CSV
weight: 20
weight: 10
parent: Input data formats
metadata: [CSV parser plugin]
---
The CSV input data format parses documents containing comma-separated values into Telegraf metrics.
Use the `csv` input data format to parse comma-separated values into Telegraf metrics.
## Configuration
@ -43,20 +44,20 @@ The CSV input data format parses documents containing comma-separated values int
## Indicates the number of rows to skip before looking for metadata and header information.
csv_skip_rows = 0
## Indicates the number of rows to parse as metadata before looking for header information.
## By default, the parser assumes there are no metadata rows to parse.
## Indicates the number of rows to parse as metadata before looking for header information.
## By default, the parser assumes there are no metadata rows to parse.
## If set, the parser would use the provided separators in the csv_metadata_separators to look for metadata.
## Please note that by default, the (key, value) pairs will be added as tags.
## Please note that by default, the (key, value) pairs will be added as tags.
## If fields are required, use the converter processor.
csv_metadata_rows = 0
## A list of metadata separators. If csv_metadata_rows is set,
## csv_metadata_separators must contain at least one separator.
## Please note that separators are case sensitive and the sequence of the seperators are respected.
csv_metadata_separators = [":", "="]
## A set of metadata trim characters.
## A set of metadata trim characters.
## If csv_metadata_trim_set is not set, no trimming is performed.
## Please note that the trim cutset is case sensitive.
csv_metadata_trim_set = ""
@ -67,6 +68,10 @@ The CSV input data format parses documents containing comma-separated values int
## The separator between csv fields
## By default, the parser assumes a comma (",")
## Please note that if you use invalid delimiters (e.g. "\u0000"), commas
## will be changed to "\ufffd", the invalid delimiters changed to a comma
## during parsing, and afterwards the invalid characters and commas are
## returned to their original values.
csv_delimiter = ","
## The character reserved for marking a row as a comment row
@ -81,6 +86,9 @@ The CSV input data format parses documents containing comma-separated values int
## will be added as fields.
csv_tag_columns = []
## Set to true to let the column tags overwrite the metadata and default tags.
csv_tag_overwrite = false
## The column to extract the name of the metric from. Will not be
## included as field in metric.
csv_measurement_column = ""
@ -115,16 +123,18 @@ The CSV input data format parses documents containing comma-separated values int
## Helpful when e.g. reading whole files in each gather-cycle.
# csv_reset_mode = "none"
```
### csv_timestamp_column, csv_timestamp_format
By default the current time will be used for all created metrics, to set the
By default, the current time will be used for all created metrics, to set the
time using the JSON document you can use the `csv_timestamp_column` and
`csv_timestamp_format` options together to set the time to a value in the parsed
document.
The `csv_timestamp_column` option specifies the column name containing the
time value and `csv_timestamp_format` must be set to a Go "reference time"
which is defined to be the specific time: `Mon Jan 2 15:04:05 MST 2006`.
The `csv_timestamp_column` option specifies the key containing the time value
and `csv_timestamp_format` must be set to `unix`, `unix_ms`, `unix_us`,
`unix_ns`, or a format string in using the Go "reference time" which is defined
to be the **specific time**: `Mon Jan 2 15:04:05 MST 2006`.
Consult the Go [time][time parse] package for details and additional examples
on how to set the time format.
@ -134,10 +144,14 @@ on how to set the time format.
One metric is created for each row with the columns added as fields. The type
of the field is automatically determined based on the contents of the value.
In addition to the options above, you can use [metric filtering][] to skip over
columns and rows.
## Examples
Config:
```
```toml
[[inputs.file]]
files = ["example"]
data_format = "csv"
@ -147,12 +161,78 @@ Config:
```
Input:
```
```csv
measurement,cpu,time_user,time_system,time_idle,time
cpu,cpu0,42,42,42,2018-09-13T13:03:28Z
```
Output:
```
```text
cpu cpu=cpu0,time_user=42,time_system=42,time_idle=42 1536869008000000000
```
Config:
```toml
[[inputs.file]]
files = ["example"]
data_format = "csv"
csv_metadata_rows = 2
csv_metadata_separators = [":", "="]
csv_metadata_trim_set = " #"
csv_header_row_count = 1
csv_tag_columns = ["Version","cpu"]
csv_timestamp_column = "time"
csv_timestamp_format = "2006-01-02T15:04:05Z07:00"
```
Input:
```csv
# Version=1.1
# File Created: 2021-11-17T07:02:45+10:00
Version,measurement,cpu,time_user,time_system,time_idle,time
1.2,cpu,cpu0,42,42,42,2018-09-13T13:03:28Z
```
Output:
```text
cpu,cpu=cpu0,File\ Created=2021-11-17T07:02:45+10:00,Version=1.1 time_user=42,time_system=42,time_idle=42 1536869008000000000
```
Config:
```toml
[[inputs.file]]
files = ["example"]
data_format = "csv"
csv_metadata_rows = 2
csv_metadata_separators = [":", "="]
csv_metadata_trim_set = " #"
csv_header_row_count = 1
csv_tag_columns = ["Version","cpu"]
csv_tag_overwrite = true
csv_timestamp_column = "time"
csv_timestamp_format = "2006-01-02T15:04:05Z07:00"
```
Input:
```csv
# Version=1.1
# File Created: 2021-11-17T07:02:45+10:00
Version,measurement,cpu,time_user,time_system,time_idle,time
1.2,cpu,cpu0,42,42,42,2018-09-13T13:03:28Z
```
Output:
```text
cpu,cpu=cpu0,File\ Created=2021-11-17T07:02:45+10:00,Version=1.2 time_user=42,time_system=42,time_idle=42 1536869008000000000
```
[time parse]: https://pkg.go.dev/time#Parse
[metric filtering]: /docs/CONFIGURATION.md#metric-filtering

315
content/telegraf/v1.27/data_formats/input/dropwizard.md
View File

@ -1,18 +1,24 @@
---
title: Dropwizard input data format
list_title: Dropwizard
description: Use the `dropwizard` input data format to parse Dropwizard JSON representations into Telegraf metrics.
menu:
telegraf_1_27_ref:
name: Dropwizard
weight: 30
weight: 10
parent: Input data formats
aliases:
- /telegraf/v1.27/data_formats/template-patterns/
metadata: [Dropwizard parser plugin]
---
The `dropwizard` data format can parse a [Dropwizard JSON representation](http://metrics.dropwizard.io/3.1.0/manual/json/) representation of a single metrics registry. By default, tags are parsed from metric names as if they were actual InfluxDB Line Protocol keys (`measurement<,tag_set>`) which can be overridden using custom [template patterns](#templates). All field value types are supported, including `string`, `number` and `boolean`.
Use the `dropwizard` input data format to parse the [JSON Dropwizard][dropwizard]
representation of a single dropwizard metric registry into Telegraf metrics. By default, tags are
parsed from metric names as if they were actual InfluxDB line protocol keys
(`measurement<,tag_set>`) which can be overridden by defining a custom [template
pattern][templates]. All field value types are supported, `string`, `number` and
`boolean`.
[templates]: /docs/TEMPLATE_PATTERN.md
[dropwizard]: http://metrics.dropwizard.io/3.1.0/manual/json/
## Configuration
@ -38,7 +44,7 @@ The `dropwizard` data format can parse a [Dropwizard JSON representation](http:/
## 2. filter + template + extra tag(s)
## 3. filter + template with field key
## 4. default template
## By providing an empty template array, templating is disabled and measurements are parsed as influxdb line protocol keys (measurement<,tag_set>)
## By providing an empty template array, templating is disabled and measurements are parsed as InfluxDB line protocol keys (measurement<,tag_set>)
templates = []
## You may use an appropriate [gjson path](https://github.com/tidwall/gjson#path-syntax)
@ -60,76 +66,75 @@ The `dropwizard` data format can parse a [Dropwizard JSON representation](http:/
# tag2 = "tags.tag2"
```
## Examples
A typical JSON of a dropwizard metric registry:
```json
{
"version": "3.0.0",
"counters" : {
"measurement,tag1=green" : {
"count" : 1
}
},
"meters" : {
"measurement" : {
"count" : 1,
"m15_rate" : 1.0,
"m1_rate" : 1.0,
"m5_rate" : 1.0,
"mean_rate" : 1.0,
"units" : "events/second"
}
},
"gauges" : {
"measurement" : {
"value" : 1
}
},
"histograms" : {
"measurement" : {
"count" : 1,
"max" : 1.0,
"mean" : 1.0,
"min" : 1.0,
"p50" : 1.0,
"p75" : 1.0,
"p95" : 1.0,
"p98" : 1.0,
"p99" : 1.0,
"p999" : 1.0,
"stddev" : 1.0
}
},
"timers" : {
"measurement" : {
"count" : 1,
"max" : 1.0,
"mean" : 1.0,
"min" : 1.0,
"p50" : 1.0,
"p75" : 1.0,
"p95" : 1.0,
"p98" : 1.0,
"p99" : 1.0,
"p999" : 1.0,
"stddev" : 1.0,
"m15_rate" : 1.0,
"m1_rate" : 1.0,
"m5_rate" : 1.0,
"mean_rate" : 1.0,
"duration_units" : "seconds",
"rate_units" : "calls/second"
}
}
"version": "3.0.0",
"counters" : {
"measurement,tag1=green" : {
"count" : 1
}
},
"meters" : {
"measurement" : {
"count" : 1,
"m15_rate" : 1.0,
"m1_rate" : 1.0,
"m5_rate" : 1.0,
"mean_rate" : 1.0,
"units" : "events/second"
}
},
"gauges" : {
"measurement" : {
"value" : 1
}
},
"histograms" : {
"measurement" : {
"count" : 1,
"max" : 1.0,
"mean" : 1.0,
"min" : 1.0,
"p50" : 1.0,
"p75" : 1.0,
"p95" : 1.0,
"p98" : 1.0,
"p99" : 1.0,
"p999" : 1.0,
"stddev" : 1.0
}
},
"timers" : {
"measurement" : {
"count" : 1,
"max" : 1.0,
"mean" : 1.0,
"min" : 1.0,
"p50" : 1.0,
"p75" : 1.0,
"p95" : 1.0,
"p98" : 1.0,
"p99" : 1.0,
"p999" : 1.0,
"stddev" : 1.0,
"m15_rate" : 1.0,
"m1_rate" : 1.0,
"m5_rate" : 1.0,
"mean_rate" : 1.0,
"duration_units" : "seconds",
"rate_units" : "calls/second"
}
}
}
```
Would get translated into 4 different measurements:
```
```text
measurement,metric_type=counter,tag1=green count=1
measurement,metric_type=meter count=1,m15_rate=1.0,m1_rate=1.0,m5_rate=1.0,mean_rate=1.0
measurement,metric_type=gauge value=1
@ -137,32 +142,34 @@ measurement,metric_type=histogram count=1,max=1.0,mean=1.0,min=1.0,p50=1.0,p75=1
measurement,metric_type=timer count=1,max=1.0,mean=1.0,min=1.0,p50=1.0,p75=1.0,p95=1.0,p98=1.0,p99=1.0,p999=1.0,stddev=1.0,m15_rate=1.0,m1_rate=1.0,m5_rate=1.0,mean_rate=1.0
```
You may also parse a dropwizard registry from any JSON document which contains a dropwizard registry in some inner field.
Eg. to parse the following JSON document:
You may also parse a dropwizard registry from any JSON document which contains a
dropwizard registry in some inner field. Eg. to parse the following JSON
document:
```json
{
"time" : "2017-02-22T14:33:03.662+02:00",
"tags" : {
"tag1" : "green",
"tag2" : "yellow"
},
"metrics" : {
"counters" : {
"measurement" : {
"count" : 1
}
},
"meters" : {},
"gauges" : {},
"histograms" : {},
"timers" : {}
}
"time" : "2017-02-22T14:33:03.662+02:00",
"tags" : {
"tag1" : "green",
"tag2" : "yellow"
},
"metrics" : {
"counters" : {
"measurement" : {
"count" : 1
}
},
"meters" : {},
"gauges" : {},
"histograms" : {},
"timers" : {}
}
}
```
and translate it into:
```
```text
measurement,metric_type=counter,tag1=green,tag2=yellow count=1 1487766783662000000
```
@ -178,143 +185,3 @@ dropwizard_tags_path = "tags"
# tag1 = "tags.tag1"
# tag2 = "tags.tag2"
```
## Templates <!--This content is duplicated in /telegraf/v1.27/data_formats/input/graphite/-->
Template patterns are a mini language that describes how a dot-delimited
string should be mapped to and from [metrics](/telegraf/v1.27/concepts/metrics/).
A template has the following format:
```
"host.mytag.mytag.measurement.measurement.field*"
```
You can set the following keywords:
- `measurement`: Specifies that this section of the graphite bucket corresponds
to the measurement name. This can be specified multiple times.
- `field`: Specifies that this section of the graphite bucket corresponds
to the field name. This can be specified multiple times.
- `measurement*`: Specifies that all remaining elements of the graphite bucket
correspond to the measurement name.
- `field*`: Specifies that all remaining elements of the graphite bucket
correspond to the field name.
{{% note %}}
`field*` can't be used in conjunction with `measurement*`.
{{% /note %}}
Any part of the template that isn't a keyword is treated as a tag key, which can also be used multiple times.
### Examples
#### Measurement and tag templates
The most basic template is to specify a single transformation to apply to all
incoming metrics.
##### Template
```toml
templates = [
"region.region.measurement*"
]
```
##### Resulting transformation
```
us.west.cpu.load 100
=> cpu.load,region=us.west value=100
```
You can also specify multiple templates using [filters](#filter-templates).
```toml
templates = [
"*.*.* region.region.measurement", # <- all 3-part measurements will match this one.
"*.*.*.* region.region.host.measurement", # <- all 4-part measurements will match this one.
]
```
#### Field templates
The field keyword tells Telegraf to give the metric that field name.
##### Template
```toml
separator = "_"
templates = [
"measurement.measurement.field.field.region"
]
```
##### Resulting transformation
```
cpu.usage.idle.percent.eu-east 100
=> cpu_usage,region=eu-east idle_percent=100
```
You can also derive the field key from all remaining elements of the graphite
bucket by specifying `field*`.
##### Template
```toml
separator = "_"
templates = [
"measurement.measurement.region.field*"
]
```
##### Resulting transformation
```
cpu.usage.eu-east.idle.percentage 100
=> cpu_usage,region=eu-east idle_percentage=100
```
#### Filter templates
You can also filter templates based on the name of the bucket
using a wildcard.
##### Template
```toml
templates = [
"cpu.* measurement.measurement.region",
"mem.* measurement.measurement.host"
]
```
##### Resulting transformation
```
cpu.load.eu-east 100
=> cpu_load,region=eu-east value=100
mem.cached.localhost 256
=> mem_cached,host=localhost value=256
```
#### Adding tags
You can add additional tags to a metric that don't exist on the received metric by specifying them after the pattern. Tags have the same format as the line protocol.
Separate multiple tags with commas.
##### Template
```toml
templates = [
"measurement.measurement.field.region datacenter=1a"
]
```
##### Resulting transformation
```
cpu.usage.idle.eu-east 100
=> cpu_usage,region=eu-east,datacenter=1a idle=100

71
content/telegraf/v1.27/data_formats/input/form_urlencoded.md Normal file
View File

@ -0,0 +1,71 @@
---
title: Form URL-encoded input data format
list_title: Form URL-encoded
description:
Use the `form-urlencoded` data format to parse `application/x-www-form-urlencoded`
data, such as HTTP query strings.
menu:
telegraf_1_27_ref:
name: Form URL-encoded
weight: 10
parent: Input data formats
metadata: [Form URLencoded parser plugin]
---
Use the `form-urlencoded` data format to parse `application/x-www-form-urlencoded`
data, such as HTTP query strings.
A common use case is to pair it with the [http_listener_v2](/telegraf/v1.27/plugins/#input-http_listener_v2) input plugin to parse
the HTTP request body or query parameters.
## Configuration
```toml
[[inputs.http_listener_v2]]
## Address and port to host HTTP listener on
service_address = ":8080"
## Part of the request to consume. Available options are "body" and
## "query".
data_source = "body"
## Data format to consume.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "form_urlencoded"
## Array of key names which should be collected as tags.
## By default, keys with string value are ignored if not marked as tags.
form_urlencoded_tag_keys = ["tag1"]
```
## Examples
### Basic parsing
Config:
```toml
[[inputs.http_listener_v2]]
name_override = "mymetric"
service_address = ":8080"
data_source = "query"
data_format = "form_urlencoded"
form_urlencoded_tag_keys = ["tag1"]
```
Request:
```bash
curl -i -XGET 'http://localhost:8080/telegraf?tag1=foo&field1=0.42&field2=42'
```
Output:
```text
mymetric,tag1=foo field1=0.42,field2=42
```
[query string]: https://en.wikipedia.org/wiki/Query_string
[http_listener_v2]: /plugins/inputs/http_listener_v2

150
content/telegraf/v1.27/data_formats/input/graphite.md
View File

@ -1,18 +1,16 @@
---
title: Graphite input data format
description: Use the Graphite data format to translate Graphite dot buckets directly into Telegraf measurement names, with a single value field, and without any tags.
list_title: Graphite
description: Use the `graphite` input data format to parse Graphite dot buckets into Telegraf metrics.
menu:
telegraf_1_27_ref:
name: Graphite
weight: 40
weight: 10
parent: Input data formats
aliases:
- /telegraf/v1.27/data_formats/template-patterns/
---
The Graphite data format translates Graphite *dot* buckets directly into
Telegraf measurement names, with a single value field, and without any tags.
Use the `graphite` input data format to parse graphite _dot_ buckets directly into
Telegraf metrics with a measurement name, a single field, and optional tags.
By default, the separator is left as `.`, but this can be changed using the
`separator` argument. For more advanced options, Telegraf supports specifying
[templates](#templates) to translate graphite buckets into Telegraf metrics.
@ -55,141 +53,7 @@ By default, the separator is left as `.`, but this can be changed using the
## Templates
Template patterns are a mini language that describes how a dot-delimited
string should be mapped to and from [metrics](/telegraf/v1.27/concepts/metrics/).
[Template patterns](/{{% latest "telegraf" %}}/concepts/metrics/) specify how a dot-delimited
string should be mapped to and from [metrics](/{{% latest "telegraf" %}}/configure_plugins/template-patterns/).
A template has the following format:
```
"host.mytag.mytag.measurement.measurement.field*"
```
You can set the following keywords:
- `measurement`: Specifies that this section of the graphite bucket corresponds
to the measurement name. This can be specified multiple times.
- `field`: Specifies that this section of the graphite bucket corresponds
to the field name. This can be specified multiple times.
- `measurement*`: Specifies that all remaining elements of the graphite bucket
correspond to the measurement name.
- `field*`: Specifies that all remaining elements of the graphite bucket
correspond to the field name.
{{% note %}}
`field*` can't be used in conjunction with `measurement*`.
{{% /note %}}
Any part of the template that isn't a keyword is treated as a tag key, which can also be used multiple times.
### Examples
#### Measurement and tag templates
The most basic template is to specify a single transformation to apply to all
incoming metrics.
##### Template <!--This content is duplicated in /telegraf/v1.27/data_formats/input/graphite/-->
```toml
templates = [
"region.region.measurement*"
]
```
##### Resulting transformation
```
us.west.cpu.load 100
=> cpu.load,region=us.west value=100
```
You can also specify multiple templates using [filters](#filter-templates).
```toml
templates = [
"*.*.* region.region.measurement", # <- all 3-part measurements will match this one.
"*.*.*.* region.region.host.measurement", # <- all 4-part measurements will match this one.
]
```
#### Field templates
The field keyword tells Telegraf to give the metric that field name.
##### Template
```toml
separator = "_"
templates = [
"measurement.measurement.field.field.region"
]
```
##### Resulting transformation
```
cpu.usage.idle.percent.eu-east 100
=> cpu_usage,region=eu-east idle_percent=100
```
You can also derive the field key from all remaining elements of the graphite
bucket by specifying `field*`.
##### Template
```toml
separator = "_"
templates = [
"measurement.measurement.region.field*"
]
```
##### Resulting transformation
```
cpu.usage.eu-east.idle.percentage 100
=> cpu_usage,region=eu-east idle_percentage=100
```
#### Filter templates
You can also filter templates based on the name of the bucket
using a wildcard.
##### Template
```toml
templates = [
"cpu.* measurement.measurement.region",
"mem.* measurement.measurement.host"
]
```
##### Resulting transformation
```
cpu.load.eu-east 100
=> cpu_load,region=eu-east value=100
mem.cached.localhost 256
=> mem_cached,host=localhost value=256
```
#### Adding tags
You can add additional tags to a metric that don't exist on the received metric by specifying them after the pattern. Tags have the same format as the line protocol.
Separate multiple tags with commas.
##### Template
```toml
templates = [
"measurement.measurement.field.region datacenter=1a"
]
```
##### Resulting transformation
```
cpu.usage.idle.eu-east 100
=> cpu_usage,region=eu-east,datacenter=1a idle=100
```

97
content/telegraf/v1.27/data_formats/input/grok.md
View File

@ -1,31 +1,32 @@
---
title: Grok input data format
description: Use the grok data format to parse line-delimited data using a regular expression-like language.
list_title: Grok
description: Use the `grok` data format to parse line-delimited data using a regular expression-like language.
menu:
telegraf_1_27_ref:
name: Grok
weight: 40
weight: 10
parent: Input data formats
---
The grok data format parses line-delimited data using a regular expression-like
Use the `grok` data format to parse line-delimited data using a regular expression-like
language.
For an introduction to grok patterns, see [Grok Basics](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#_grok_basics)
in the Logstash documentation. The grok parser uses a slightly modified version of logstash "grok"
in the Logstash documentation. The grok parser uses a slightly modified version of logstash **grok**
patterns, using the format:
```
```text
%{<capture_syntax>[:<semantic_name>][:<modifier>]}
```
The `capture_syntax` defines the grok pattern that is used to parse the input
The `capture_syntax` defines the grok pattern used to parse the input
line and the `semantic_name` is used to name the field or tag. The extension
`modifier` controls the data type that the parsed item is converted to or
other special handling.
By default, all named captures are converted into string fields.
By default all named captures are converted into string fields.
If a pattern does not have a semantic name it will not be captured.
Timestamp modifiers can be used to convert captures to the timestamp of the
parsed metric. If no timestamp is parsed the metric will be created using the
current time.
@ -55,6 +56,7 @@ You must capture at least one field per line.
- ts-httpd ("02/Jan/2006:15:04:05 -0700")
- ts-epoch (seconds since unix epoch, may contain decimal)
- ts-epochnano (nanoseconds since unix epoch)
- ts-epochmilli (milliseconds since unix epoch)
- ts-syslog ("Jan 02 15:04:05", parsed time is set to the current year)
- ts-"CUSTOM"
@ -67,11 +69,19 @@ See https://golang.org/pkg/time/#Parse for more details.
Telegraf has many of its own [built-in patterns](https://github.com/influxdata/telegraf/blob/master/plugins/parsers/grok/influx_patterns.go),
as well as support for most of
[Logstash's core patterns](https://github.com/logstash-plugins/logstash-patterns-core/blob/main/patterns/ecs-v1/grok-patterns).
_Golang regular expressions do not support lookahead or lookbehind.
Logstash patterns that depend on these are not supported._
_Golang regular expressions do not support _lookahead_ or _lookbehind_.
Logstash patterns that depend on these aren't supported._
For help building and testing patterns, see [tips for creating patterns](#tips-for-creating-patterns).
<!-- TOC -->
- [Configuration](#configuration)
- [Timestamp Examples](#timestamp-examples)
- [TOML Escaping](#toml-escaping)
- [Tips for creating patterns](#tips-for-creating-patterns)
- [Performance](#performance)
## Configuration
```toml
@ -115,13 +125,20 @@ For help building and testing patterns, see [tips for creating patterns](#tips-f
## 2. "Canada/Eastern" -- Unix TZ values like those found in https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
## 3. UTC -- or blank/unspecified, will return timestamp in UTC
grok_timezone = "Canada/Eastern"
## When set to "disable" timestamp will not incremented if there is a
## duplicate.
# grok_unique_timestamp = "auto"
## Enable multiline messages to be processed.
# grok_multiline = false
```
### Timestamp examples
### Timestamp Examples
This example input and config parses a file using a custom timestamp conversion:
```
```text
2017-02-21 13:10:34 value=42
```
@ -132,7 +149,7 @@ This example input and config parses a file using a custom timestamp conversion:
This example input and config parses a file using a timestamp in unix time:
```
```text
1466004605 value=42
1466004605.123456789 value=42
```
@ -144,7 +161,7 @@ This example input and config parses a file using a timestamp in unix time:
This example parses a file using a built-in conversion and a custom pattern:
```
```text
Wed Apr 12 13:10:34 PST 2017 value=42
```
@ -156,24 +173,42 @@ Wed Apr 12 13:10:34 PST 2017 value=42
'''
```
For cases where the timestamp itself is without offset, the `timezone` config var is available
to denote an offset. By default (with `timezone` either omit, blank or set to `"UTC"`), the times
are processed as if in the UTC timezone. If specified as `timezone = "Local"`, the timestamp
will be processed based on the current machine timezone configuration. Lastly, if using a
timezone from the list of Unix [timezones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones),
grok will offset the timestamp accordingly.
This example input and config parses a file using a custom timestamp conversion
that doesn't match any specific standard:
### TOML escaping
```text
21/02/2017 13:10:34 value=42
```
```toml
[[inputs.file]]
grok_patterns = ['%{MY_TIMESTAMP:timestamp:ts-"02/01/2006 15:04:05"} value=%{NUMBER:value:int}']
grok_custom_patterns = '''
MY_TIMESTAMP (?:\d{2}.\d{2}.\d{4} \d{2}:\d{2}:\d{2})
'''
```
For cases where the timestamp itself is without offset, the `timezone` config
var is available to denote an offset. By default (with `timezone` either omit,
blank or set to `"UTC"`), the times are processed as if in the UTC timezone. If
specified as `timezone = "Local"`, the timestamp will be processed based on the
current machine timezone configuration. Lastly, if using a timezone from the
list of Unix
[timezones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones), grok
will offset the timestamp accordingly.
#### TOML Escaping
When saving patterns to the configuration file, keep in mind the different TOML
[string](https://github.com/toml-lang/toml#string) types and the escaping
rules for each. These escaping rules must be applied in addition to the
escaping required by the grok syntax. Using the TOML multi-line literal
syntax (`'''`) may be useful.
escaping required by the grok syntax. Using the Multi-line line literal
syntax with `'''` may be useful.
The following config examples will parse this input file:
```
```text
|42|\uD83D\uDC2F|'telegraf'|
```
@ -189,6 +224,7 @@ backslash must be escaped, requiring us to escape the backslash a second time.
We cannot use a literal TOML string for the pattern, because we cannot match a
`'` within it. However, it works well for the custom pattern.
```toml
[[inputs.file]]
grok_patterns = ["\\|%{NUMBER:value:int}\\|%{UNICODE_ESCAPE:escape}\\|'%{WORD:name}'\\|"]
@ -196,6 +232,7 @@ We cannot use a literal TOML string for the pattern, because we cannot match a
```
A multi-line literal string allows us to encode the pattern:
```toml
[[inputs.file]]
grok_patterns = ['''
@ -231,3 +268,15 @@ We recommend the following steps for building and testing a new pattern with Tel
5. If successful, add the next token to the data file, update the pattern configuration in Telegraf, and then retest.
6. Continue one token at a time until the entire line is successfully parsed.
#### Performance
Performance depends heavily on the regular expressions that you use, but there
are a few techniques that can help:
- Avoid using patterns such as `%{DATA}` that will always match.
- If possible, add `^` and `$` anchors to your pattern:
```toml
[[inputs.file]]
grok_patterns = ["^%{COMBINED_LOG_FORMAT}$"]
```

22
content/telegraf/v1.27/data_formats/input/influx.md
View File

@ -1,20 +1,17 @@
---
title: InfluxDB Line Protocol input data format
description: Use the InfluxDB Line Protocol input data format to parse InfluxDB metrics directly into Telegraf metrics.
title: InfluxDB line protocol input data format
list_title: InfluxDB line protocol
description: Use the `influx` line protocol input data format to parse InfluxDB metrics directly into Telegraf metrics.
menu:
telegraf_1_27_ref:
name: InfluxDB Line Protocol input
weight: 60
name: InfluxDB line protocol
weight: 10
parent: Input data formats
---
There are no additional configuration options for InfluxDB [line protocol][]. The
InfluxDB metrics are parsed directly into Telegraf metrics.
Use the `influx` line protocol input data format to parse InfluxDB [line protocol](/influxdb/cloud-serverless/reference/syntax/line-protocol/) data into Telegraf [metrics](/telegraf/v1.27/metrics/).
[line protocol]: /{{< latest "influxdb" "v1" >}}/write_protocols/line/
### Configuration
## Configuration
```toml
[[inputs.file]]
@ -25,4 +22,9 @@ InfluxDB metrics are parsed directly into Telegraf metrics.
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "influx"
## Influx line protocol parser
## 'internal' is the default. 'upstream' is a newer parser that is faster
## and more memory efficient.
## influx_parser_type = "internal"
```

120
content/telegraf/v1.27/data_formats/input/json.md
View File

@ -1,23 +1,27 @@
---
title: JSON input data format
description: Use the JSON input data format to parse [JSON][json] objects, or an array of objects, into Telegraf metric fields.
list_title: JSON
description: |
The `json` input data format parses JSON objects, or an array of objects, into Telegraf metrics.
For most cases, use the JSON v2 input data format instead.
menu:
telegraf_1_27_ref:
name: JSON input
weight: 70
name: JSON
weight: 10
parent: Input data formats
---
{{% note %}}
The following information applies to the legacy JSON input data format. For most cases, we recommend using the [JSON v2 input data format](/{{< latest "telegraf" >}}/data_formats/input/json_v2/) instead.
The following information applies to the legacy JSON input data format.
For most cases, use the [JSON v2 input data format](/{{< latest "telegraf" >}}/data_formats/input/json_v2/) instead.
{{% /note %}}
The JSON input data format parses a [JSON][json] object or an array of objects
into Telegraf metric fields.
The `json` data format parses a [JSON][json] object or an array of objects into
metric fields.
**NOTE:** All JSON numbers are converted to float fields. JSON String are
ignored unless specified in the `tag_key` or `json_string_fields` options.
**NOTE:** All JSON numbers are converted to float fields. JSON strings and
booleans are ignored unless specified in the `tag_key` or `json_string_fields`
options.
## Configuration
@ -31,20 +35,27 @@ ignored unless specified in the `tag_key` or `json_string_fields` options.
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "json"
## When strict is true and a JSON array is being parsed, all objects within the
## array must be valid
json_strict = true
## Query is a GJSON path that specifies a specific chunk of JSON to be
## parsed, if not specified the whole document will be parsed.
##
## GJSON query paths are described here:
## https://github.com/tidwall/gjson#path-syntax
## https://github.com/tidwall/gjson/tree/v1.3.0#path-syntax
json_query = ""
## Tag keys is an array of keys that should be added as tags.
## Tag keys is an array of keys that should be added as tags. Matching keys
## are no longer saved as fields. Supports wildcard glob matching.
tag_keys = [
"my_tag_1",
"my_tag_2"
"my_tag_2",
"tags_*",
"tag*"
]
## String fields is an array of keys that should be added as string fields.
## Array of glob pattern strings or booleans keys that should be added as string fields.
json_string_fields = []
## Name key is the key to use as the measurement name.
@ -54,25 +65,44 @@ ignored unless specified in the `tag_key` or `json_string_fields` options.
## metric.
json_time_key = ""
## Time format is the time layout that should be used to interprete the
## json_time_key. The time must be `unix`, `unix_ms` or a time in the
## "reference time".
## Time format is the time layout that should be used to interpret the json_time_key.
## The time must be `unix`, `unix_ms`, `unix_us`, `unix_ns`, or a time in the
## "reference time". To define a different format, arrange the values from
## the "reference time" in the example to match the format you will be
## using. For more information on the "reference time", visit
## https://golang.org/pkg/time/#Time.Format
## ex: json_time_format = "Mon Jan 2 15:04:05 -0700 MST 2006"
## json_time_format = "2006-01-02T15:04:05Z07:00"
## json_time_format = "01/02/2006 15:04:05"
## json_time_format = "unix"
## json_time_format = "unix_ms"
json_time_format = ""
## Timezone allows you to provide an override for timestamps that
## don't already include an offset
## e.g. 04/06/2016 12:41:45
##
## Default: "" which renders UTC
## Options are as follows:
## 1. Local -- interpret based on machine localtime
## 2. "America/New_York" -- Unix TZ values like those found in https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
## 3. UTC -- or blank/unspecified, will return timestamp in UTC
json_timezone = ""
```
### `json_query`
### json_query
The `json_query` is a [GJSON][gjson] path that can be used to limit the
portion of the overall JSON document that should be parsed. The result of the
query should contain a JSON object or an array of objects.
The `json_query` is a [GJSON][gjson] path that can be used to transform the
JSON document before being parsed. The query is performed before any other
options are applied and the new document produced will be parsed instead of the
original document, as such, the result of the query should be a JSON object or
an array of objects.
Consult the GJSON [path syntax][gjson syntax] for details and examples.
Consult the GJSON [path syntax][gjson syntax] for details and examples, and
consider using the [GJSON playground][gjson playground] for developing and
debugging your query.
### json_time_key, json_time_format
### json_time_key, json_time_format, json_timezone
By default the current time will be used for all created metrics, to set the
time using the JSON document you can use the `json_time_key` and
@ -80,17 +110,25 @@ time using the JSON document you can use the `json_time_key` and
document.
The `json_time_key` option specifies the key containing the time value and
`json_time_format` must be set to `unix`, `unix_ms`, or the Go "reference
time" which is defined to be the specific time: `Mon Jan 2 15:04:05 MST 2006`.
`json_time_format` must be set to `unix`, `unix_ms`, `unix_us`, `unix_ns`, or
the Go "reference time" which is defined to be the specific time:
`Mon Jan 2 15:04:05 MST 2006`.
Consult the Go [time][time parse] package for details and additional examples
on how to set the time format.
When parsing times that don't include a timezone specifier, times are assumed to
be UTC. To default to another timezone, or to local time, specify the
`json_timezone` option. This option should be set to a [Unix TZ
value](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones), such as
`America/New_York`, to `Local` to utilize the system timezone, or to `UTC`.
## Examples
### Basic parsing
### Basic Parsing
Config:
```toml
[[inputs.file]]
files = ["example"]
@ -99,6 +137,7 @@ Config:
```
Input:
```json
{
"a": 5,
@ -110,23 +149,26 @@ Input:
```
Output:
```
```text
myjsonmetric a=5,b_c=6
```
### Name, tags, and string fields
### Name, Tags, and String Fields
Config:
```toml
[[inputs.file]]
files = ["example"]
json_name_key = "name"
tag_keys = ["my_tag_1"]
json_string_fields = ["my_field"]
json_string_fields = ["b_my_field"]
data_format = "json"
```
Input:
```json
{
"a": 5,
@ -140,8 +182,9 @@ Input:
```
Output:
```
my_json,my_tag_1=foo a=5,b_c=6,my_field="description"
```text
my_json,my_tag_1=foo a=5,b_c=6,b_my_field="description"
```
### Arrays
@ -150,6 +193,7 @@ If the JSON data is an array, then each object within the array is parsed with
the configured settings.
Config:
```toml
[[inputs.file]]
files = ["example"]
@ -159,6 +203,7 @@ Config:
```
Input:
```json
[
{
@ -166,20 +211,21 @@ Input:
"b": {
"c": 6,
"time":"04 Jan 06 15:04 MST"
},
}
},
{
"a": 7,
"b": {
"c": 8,
"time":"11 Jan 07 15:04 MST"
},
}
}
]
```
Output:
```
```text
file a=5,b_c=6 1136387040000000000
file a=7,b_c=8 1168527840000000000
```
@ -189,16 +235,18 @@ file a=7,b_c=8 1168527840000000000
The `json_query` option can be used to parse a subset of the document.
Config:
```toml
[[inputs.file]]
files = ["example"]
data_format = "json_v2"
data_format = "json"
tag_keys = ["first"]
json_string_fields = ["last"]
json_query = "obj.friends"
```
Input:
```json
{
"obj": {
@ -216,7 +264,8 @@ Input:
```
Output:
```
```text
file,first=Dale last="Murphy",age=44
file,first=Roger last="Craig",age=68
file,first=Jane last="Murphy",age=47
@ -224,5 +273,6 @@ file,first=Jane last="Murphy",age=47
[gjson]: https://github.com/tidwall/gjson
[gjson syntax]: https://github.com/tidwall/gjson#path-syntax
[gjson playground]: https://gjson.dev/
[json]: https://www.json.org/
[time parse]: https://golang.org/pkg/time/#Parse

16
content/telegraf/v1.27/data_formats/input/json_v2.md
View File

@ -1,19 +1,19 @@
---
title: JSON v2 input data format
description: Use the JSON v2 input data format to parse [JSON][json] objects, or an array of objects, into Telegraf metric fields.
list_title: JSON v2
description: Use the `json_v2` input data format to parse [JSON][json] objects, or an array of objects, into Telegraf metrics.
menu:
telegraf_1_27_ref:
name: JSON v2 input
weight: 70
name: JSON v2
weight: 10
parent: Input data formats
---
The JSON v2 input data format parses a [JSON][json] object or an array of objects into Telegraf metric fields.
This parser takes valid JSON input and turns it into metrics.
Use the `json_v2` input data format to parse a [JSON][json] object or an array of objects into Telegraf metrics.
The query syntax supported is [GJSON Path Syntax](https://github.com/tidwall/gjson/blob/v1.7.5/SYNTAX.md),
Use to [this playground](https://gjson.dev/) to test out your GJSON path.
The parser supports [GJSON Path Syntax](https://github.com/tidwall/gjson/blob/v1.7.5/SYNTAX.md) for querying JSON.
To test your GJSON path, use [GJSON Playground](https://gjson.dev/).
You can find multiple examples [here](https://github.com/influxdata/telegraf/tree/master/plugins/parsers/json_v2/testdata) in the Telegraf repository.

17
content/telegraf/v1.27/data_formats/input/logfmt.md
View File

@ -1,15 +1,15 @@
---
title: Logfmt input data format
list_title: Logfmt
description: Use the `logfmt` input data format to parse logfmt data into Telegraf metrics.
menu:
telegraf_1_27_ref:
name: logfmt
weight: 80
weight: 10
parent: Input data formats
---
The `logfmt` data format parses [logfmt] data into Telegraf metrics.
Use the `logfmt` data format to parse [logfmt] data into Telegraf metrics.
[logfmt]: https://brandur.org/logfmt
@ -25,19 +25,18 @@ The `logfmt` data format parses [logfmt] data into Telegraf metrics.
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "logfmt"
## Set the name of the created metric, if unset the name of the plugin will
## be used.
metric_name = "logfmt"
## Array of key names which should be collected as tags. Globs accepted.
logfmt_tag_keys = ["method","host"]
```
## Metrics
Each key-value pair in the line is added to a new metric as a field. The type
Each key/value pair in the line is added to a new metric as a field. The type
of the field is automatically determined based on the contents of the value.
## Examples
```
```text
- method=GET host=example.org ts=2018-07-24T19:43:40.275Z connect=4ms service=8ms status=200 bytes=1653
+ logfmt method="GET",host="example.org",ts="2018-07-24T19:43:40.275Z",connect="4ms",service="8ms",status=200i,bytes=1653i
+ logfmt,host=example.org,method=GET ts="2018-07-24T19:43:40.275Z",connect="4ms",service="8ms",status=200i,bytes=1653i
```

10
content/telegraf/v1.27/data_formats/input/nagios.md
View File

@ -1,17 +1,15 @@
---
title: Nagios input data format
description: Use the Nagios input data format to parse the output of Nagios plugins into Telegraf metrics.
list_title: Nagios
description: Use the `nagios` input data format to parse the output of Nagios plugins into Telegraf metrics.
menu:
telegraf_1_27_ref:
name: Nagios
weight: 90
weight: 10
parent: Input data formats
---
# Nagios
The Nagios input data format parses the output of
Use the `nagios` input data format to parse the output of
[Nagios plugins](https://www.nagios.org/downloads/nagios-plugins/) into
Telegraf metrics.

39
content/telegraf/v1.27/data_formats/input/opentsdb.md Normal file
View File

@ -0,0 +1,39 @@
---
title: OpenTSDB Telnet "PUT" API input data format
list_title: OpenTSDB Telnet PUT API
description:
Use the `opentsdb` data format to parse OpenTSDB Telnet `PUT` API data into Telegraf metrics.
menu:
telegraf_1_27_ref:
name: OpenTSDB
weight: 10
parent: Input data formats
metadata: []
---
Use the `opentsdb` data format to parse [OpenTSDB Telnet `PUT` API](http://opentsdb.net/docs/build/html/api_telnet/put.html) data into
Telegraf metrics. There are no additional configuration options for OpenTSDB.
For more detail on the format, see:
- [OpenTSDB Telnet "PUT" API guide](http://opentsdb.net/docs/build/html/api_telnet/put.html)
- [OpenTSDB data specification](http://opentsdb.net/docs/build/html/user_guide/writing/index.html#data-specification)
## Configuration
```toml
[[inputs.file]]
files = ["example"]
## Data format to consume.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "opentsdb"
```
## Example
```opentsdb
put sys.cpu.user 1356998400 42.5 host=webserver01 cpu=0
```

65
content/telegraf/v1.27/data_formats/input/prometheus-remote-write.md
View File

@ -1,16 +1,16 @@
---
title: Prometheus Remote Write input data format
description: |
Use the Prometheus Remote Write input data format to write samples directly into Telegraf metrics.
list_title: Prometheus Remote Write
description:
Use the `prometheusremotewrite` input data format to parse Prometheus Remote Write samples into Telegraf metrics.
menu:
telegraf_1_27_ref:
name: Prometheus Remote Write
weight: 40
weight: 10
parent: Input data formats
---
Use the Prometheus Remote Write plugin to convert [Prometheus Remote Write](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write) samples directly into Telegraf metrics.
Use the `prometheusremotewrite` input data format to parse [Prometheus Remote Write](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write) samples into Telegraf metrics.
{{% note %}}
If you are using InfluxDB 1.x and the [Prometheus Remote Write endpoint](https://github.com/influxdata/telegraf/blob/master/plugins/parsers/prometheusremotewrite/README.md
@ -19,44 +19,51 @@ For the metrics to completely align with the 1.x endpoint, add a Starlark proces
{{% /note %}}
### Configuration
Converts prometheus remote write samples directly into Telegraf metrics. It can
be used with [http_listener_v2](/plugins/inputs/http_listener_v2). There are no
additional configuration options for Prometheus Remote Write Samples.
Use the [`inputs.http_listener_v2`](/telegraf/v1.27/plugins/#input-http_listener_v2) plug and set `data_format = "prometheusremotewrite"`
## Configuration
```toml
[[inputs.http_listener_v2]]
## Address and port to host HTTP listener on
service_address = ":1234"
## Path to listen to.
path = "/receive"
## Paths to listen to.
paths = ["/receive"]
## Data format to consume.
data_format = "prometheusremotewrite"
```
### Example
## Example Input
**Example Input**
```
```json
prompb.WriteRequest{
Timeseries: []*prompb.TimeSeries{
{
Labels: []*prompb.Label{
{Name: "__name__", Value: "go_gc_duration_seconds"},
{Name: "instance", Value: "localhost:9090"},
{Name: "job", Value: "prometheus"},
{Name: "quantile", Value: "0.99"},
},
Samples: []prompb.Sample{
{Value: 4.63, Timestamp: time.Date(2020, 4, 1, 0, 0, 0, 0, time.UTC).UnixNano()},
},
},
},
}
Timeseries: []*prompb.TimeSeries{
{
Labels: []*prompb.Label{
{Name: "__name__", Value: "go_gc_duration_seconds"},
{Name: "instance", Value: "localhost:9090"},
{Name: "job", Value: "prometheus"},
{Name: "quantile", Value: "0.99"},
},
Samples: []prompb.Sample{
{Value: 4.63, Timestamp: time.Date(2020, 4, 1, 0, 0, 0, 0, time.UTC).UnixNano()},
},
},
},
}
```
**Example Output**
```
## Example Output
```text
prometheus_remote_write,instance=localhost:9090,job=prometheus,quantile=0.99 go_gc_duration_seconds=4.63 1614889298859000000
```
[here]: https://github.com/influxdata/telegraf/tree/master/plugins/parsers/prometheusremotewrite#for-alignment-with-the-influxdb-v1x-prometheus-remote-write-spec
## For alignment with the [InfluxDB v1.x Prometheus Remote Write Spec](https://docs.influxdata.com/influxdb/v1.8/supported_protocols/prometheus/#how-prometheus-metrics-are-parsed-in-influxdb)
- Use the [Starlark processor rename prometheus remote write script](https://github.com/influxdata/telegraf/blob/master/plugins/processors/starlark/testdata/rename_prometheus_remote_write.star) to rename the measurement name to the fieldname and rename the fieldname to value.

20
content/telegraf/v1.27/data_formats/input/value.md
View File

@ -1,21 +1,22 @@
---
title: Value input data format
list_title: Value
description: Use the `value` input data format to parse single values into Telegraf metrics.
menu:
telegraf_1_27_ref:
name: Value
weight: 100
weight: 10
parent: Input data formats
---
The "value" input data format translates single values into Telegraf metrics. This
is done by assigning a measurement name and setting a single field ("value")
as the parsed metric.
Use the `value` input data format to parse single values into Telegraf metrics.
## Configuration
Specify the measurement name and a field to use as the parsed metric.
> To specify the measurement name for your metric, set `name_override`; otherwise, the input plugin name (for example, "exec") is used as the measurement name.
You **must** tell Telegraf what type of metric to collect by using the
`data_type` configuration option. Available data type options are:
@ -24,10 +25,6 @@ You **must** tell Telegraf what type of metric to collect by using the
3. string
4. boolean
> **Note:** It is also recommended that you set `name_override` to a measurement
name that makes sense for your metric; otherwise, it will just be set to the
name of the plugin.
```toml
[[inputs.exec]]
## Commands array
@ -36,6 +33,9 @@ name of the plugin.
## override the default metric name of "exec"
name_override = "entropy_available"
## override the field name of "value"
# value_field_name = "value"
## Data format to consume.
## Each data format has its own unique set of configuration options, read
## more about them here:

8
content/telegraf/v1.27/data_formats/input/wavefront.md
View File

@ -1,15 +1,15 @@
---
title: Wavefront input data format
description: Use the Wavefront input data format to parse Wavefront data into Telegraf metrics.
list_title: Wavefront
description: Use the `wavefront` input data format to parse Wavefront data into Telegraf metrics.
menu:
telegraf_1_27_ref:
name: Wavefront
weight: 110
weight: 10
parent: Input data formats
---
The Wavefront input data format parses Wavefront data into Telegraf metrics.
Use the `wavefront` input data format to parse Wavefront data into Telegraf metrics.
For more information on the Wavefront native data format, see
[Wavefront Data Format](https://docs.wavefront.com/wavefront_data_format.html) in the Wavefront documentation.

10
content/telegraf/v1.27/data_formats/input/xml.md
View File

@ -1,16 +1,16 @@
---
title: XML input data format
description: Use the XML input data format to parse XML data into Telegraf metrics.
list_title: XML
description: Use the `xml` input data format to parse XML data into Telegraf metrics.
menu:
telegraf_1_27_ref:
name: XML
weight: 110
weight: 10
parent: Input data formats
metadata: [XPath parser plugin]
---
The XML input data format parses XML data into Telegraf metrics.
Use the `xml` input data format, provided by the [XPath parser plugin](https://github.com/influxdata/telegraf/tree/master/plugins/parsers/xpath), with XPath expressions to parse XML data into Telegraf metrics.
## Configuration

629
content/telegraf/v1.27/data_formats/input/xpath_json.md Normal file
View File

@ -0,0 +1,629 @@
---
title: XPath JSON input data format
list_title: XPath JSON
description:
Use the `xpath_json` input data format and XPath expressions to parse JSON into Telegraf metrics.
menu:
telegraf_1_27_ref:
name: XPath JSON
weight: 10
parent: Input data formats
metadata: [XPath parser plugin]
---
Use the `xpath_json` input data format, provided by the [XPath parser plugin](https://github.com/influxdata/telegraf/tree/master/plugins/parsers/xpath), with [XPath][xpath] expressions to parse JSON data into Telegraf metrics.
For information about supported XPath functions, see [the underlying XPath library][xpath lib].
**NOTE:** The type of fields are specified using [XPath functions][xpath
lib]. The only exceptions are _integer_ fields that need to be specified in a
`fields_int` section.
## Supported data formats
| name | `data_format` setting | comment |
| --------------------------------------- | --------------------- | ------- |
| [Extensible Markup Language (XML)][xml] | `"xml"` | |
| [JSON][json] | `"xpath_json"` | |
| [MessagePack][msgpack] | `"xpath_msgpack"` | |
| [Protocol-buffers][protobuf] | `"xpath_protobuf"` | [see additional parameters](#protocol-buffers-additional-settings)|
### Protocol-buffers additional settings
For using the protocol-buffer format you need to specify additional
(_mandatory_) properties for the parser. Those options are described here.
#### `xpath_protobuf_file` (mandatory)
Use this option to specify the name of the protocol-buffer definition file
(`.proto`).
#### `xpath_protobuf_type` (mandatory)
This option contains the top-level message file to use for deserializing the
data to be parsed. Usually, this is constructed from the `package` name in the
protocol-buffer definition file and the `message` name as `<package
name>.<message name>`.
#### `xpath_protobuf_import_paths` (optional)
In case you import other protocol-buffer definitions within your `.proto` file
(i.e. you use the `import` statement) you can use this option to specify paths
to search for the imported definition file(s). By default the imports are only
searched in `.` which is the current-working-directory, i.e. usually the
directory you are in when starting telegraf.
Imagine you do have multiple protocol-buffer definitions (e.g. `A.proto`,
`B.proto` and `C.proto`) in a directory (e.g. `/data/my_proto_files`) where your
top-level file (e.g. `A.proto`) imports at least one other definition
```protobuf
syntax = "proto3";
package foo;
import "B.proto";
message Measurement {
...
}
```
You should use the following setting
```toml
[[inputs.file]]
files = ["example.dat"]
data_format = "xpath_protobuf"
xpath_protobuf_file = "A.proto"
xpath_protobuf_type = "foo.Measurement"
xpath_protobuf_import_paths = [".", "/data/my_proto_files"]
...
```
#### `xpath_protobuf_skip_bytes` (optional)
This option allows to skip a number of bytes before trying to parse
the protocol-buffer message. This is useful in cases where the raw data
has a header e.g. for the message length or in case of GRPC messages.
This is a list of known headers and the corresponding values for
`xpath_protobuf_skip_bytes`
| name | setting | comment |
| --------------------------------------- | ------- | ------- |
| [GRPC protocol][GRPC] | 5 | GRPC adds a 5-byte header for _Length-Prefixed-Messages_ |
| [PowerDNS logging][PDNS] | 2 | Sent messages contain a 2-byte header containing the message length |
[GRPC]: https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md
[PDNS]: https://docs.powerdns.com/recursor/lua-config/protobuf.html
## Configuration
```toml
[[inputs.file]]
files = ["example.xml"]
## Data format to consume.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "xml"
## PROTOCOL-BUFFER definitions
## Protocol-buffer definition file
# xpath_protobuf_file = "sparkplug_b.proto"
## Name of the protocol-buffer message type to use in a fully qualified form.
# xpath_protobuf_type = "org.eclipse.tahu.protobuf.Payload"
## List of paths to use when looking up imported protocol-buffer definition files.
# xpath_protobuf_import_paths = ["."]
## Number of (header) bytes to ignore before parsing the message.
# xpath_protobuf_skip_bytes = 0
## Print the internal XML document when in debug logging mode.
## This is especially useful when using the parser with non-XML formats like protocol-buffers
## to get an idea on the expression necessary to derive fields etc.
# xpath_print_document = false
## Allow the results of one of the parsing sections to be empty.
## Useful when not all selected files have the exact same structure.
# xpath_allow_empty_selection = false
## Get native data-types for all data-format that contain type information.
## Currently, protobuf, msgpack and JSON support native data-types
# xpath_native_types = false
## Multiple parsing sections are allowed
[[inputs.file.xpath]]
## Optional: XPath-query to select a subset of nodes from the XML document.
# metric_selection = "/Bus/child::Sensor"
## Optional: XPath-query to set the metric (measurement) name.
# metric_name = "string('example')"
## Optional: Query to extract metric timestamp.
## If not specified the time of execution is used.
# timestamp = "/Gateway/Timestamp"
## Optional: Format of the timestamp determined by the query above.
## This can be any of "unix", "unix_ms", "unix_us", "unix_ns" or a valid Golang
## time format. If not specified, a "unix" timestamp (in seconds) is expected.
# timestamp_format = "2006-01-02T15:04:05Z"
## Optional: Timezone of the parsed time
## This will locate the parsed time to the given timezone. Please note that
## for times with timezone-offsets (e.g. RFC3339) the timestamp is unchanged.
## This is ignored for all (unix) timestamp formats.
# timezone = "UTC"
## Optional: List of fields to convert to hex-strings if they are
## containing byte-arrays. This might be the case for e.g. protocol-buffer
## messages encoding data as byte-arrays. Wildcard patterns are allowed.
## By default, all byte-array-fields are converted to string.
# fields_bytes_as_hex = []
## Tag definitions using the given XPath queries.
[inputs.file.xpath.tags]
name = "substring-after(Sensor/@name, ' ')"
device = "string('the ultimate sensor')"
## Integer field definitions using XPath queries.
[inputs.file.xpath.fields_int]
consumers = "Variable/@consumers"
## Non-integer field definitions using XPath queries.
## The field type is defined using XPath expressions such as number(), boolean() or string(). If no conversion is performed the field will be of type string.
[inputs.file.xpath.fields]
temperature = "number(Variable/@temperature)"
power = "number(Variable/@power)"
frequency = "number(Variable/@frequency)"
ok = "Mode != 'ok'"
```
In this configuration mode, you explicitly specify the field and tags you want
to scrape from your data.
A configuration can contain muliple _xpath_ subsections (for example, the file plugin
to process the xml-string multiple times). Consult the [XPath syntax][xpath] and
the [underlying library's functions][xpath lib] for details and help regarding
XPath queries. Consider using an XPath tester such as [xpather.com][xpather] or
[Code Beautify's XPath Tester][xpath tester] for help developing and debugging
your query.
## Configuration (batch)
Alternatively to the configuration above, fields can also be specified in a
batch way. So contrary to specify the fields in a section, you can define a
`name` and a `value` selector used to determine the name and value of the fields
in the metric.
```toml
[[inputs.file]]
files = ["example.xml"]
## Data format to consume.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "xml"
## PROTOCOL-BUFFER definitions
## Protocol-buffer definition file
# xpath_protobuf_file = "sparkplug_b.proto"
## Name of the protocol-buffer message type to use in a fully qualified form.
# xpath_protobuf_type = "org.eclipse.tahu.protobuf.Payload"
## List of paths to use when looking up imported protocol-buffer definition files.
# xpath_protobuf_import_paths = ["."]
## Print the internal XML document when in debug logging mode.
## This is especially useful when using the parser with non-XML formats like protocol-buffers
## to get an idea on the expression necessary to derive fields etc.
# xpath_print_document = false
## Allow the results of one of the parsing sections to be empty.
## Useful when not all selected files have the exact same structure.
# xpath_allow_empty_selection = false
## Get native data-types for all data-format that contain type information.
## Currently, protobuf, msgpack and JSON support native data-types
# xpath_native_types = false
## Multiple parsing sections are allowed
[[inputs.file.xpath]]
## Optional: XPath-query to select a subset of nodes from the XML document.
metric_selection = "/Bus/child::Sensor"
## Optional: XPath-query to set the metric (measurement) name.
# metric_name = "string('example')"
## Optional: Query to extract metric timestamp.
## If not specified the time of execution is used.
# timestamp = "/Gateway/Timestamp"
## Optional: Format of the timestamp determined by the query above.
## This can be any of "unix", "unix_ms", "unix_us", "unix_ns" or a valid Golang
## time format. If not specified, a "unix" timestamp (in seconds) is expected.
# timestamp_format = "2006-01-02T15:04:05Z"
## Field specifications using a selector.
field_selection = "child::*"
## Optional: Queries to specify field name and value.
## These options are only to be used in combination with 'field_selection'!
## By default the node name and node content is used if a field-selection
## is specified.
# field_name = "name()"
# field_value = "."
## Optional: Expand field names relative to the selected node
## This allows to flatten out nodes with non-unique names in the subtree
# field_name_expansion = false
## Tag specifications using a selector.
## tag_selection = "child::*"
## Optional: Queries to specify tag name and value.
## These options are only to be used in combination with 'tag_selection'!
## By default the node name and node content is used if a tag-selection
## is specified.
# tag_name = "name()"
# tag_value = "."
## Optional: Expand tag names relative to the selected node
## This allows to flatten out nodes with non-unique names in the subtree
# tag_name_expansion = false
## Tag definitions using the given XPath queries.
[inputs.file.xpath.tags]
name = "substring-after(Sensor/@name, ' ')"
device = "string('the ultimate sensor')"
```
**Please note**: The resulting fields are _always_ of type string.
It is also possible to specify a mixture of the two alternative ways of
specifying fields. In this case, _explicitly_ defined tags and fields take
_precedence_ over the batch instances if both use the same tag or field name.
### metric_selection (optional)
You can specify a [XPath][xpath] query to select a subset of nodes from the XML
document, each used to generate a new metrics with the specified fields, tags
etc.
Relative queries in subsequent queries are relative to the
`metric_selection`. To specify absolute paths, start the query with a
slash (`/`).
Specifying `metric_selection` is optional. If not specified, all relative queries
are relative to the root node of the XML document.
### metric_name (optional)
By specifying `metric_name` you can override the metric/measurement name with
the result of the given [XPath][xpath] query. If not specified, the default
metric name is used.
### timestamp, timestamp_format, timezone (optional)
By default, the current time is used for all created metrics. To set the
time from values in the XML document you can specify a [XPath][xpath] query in
`timestamp` and set the format in `timestamp_format`.
The `timestamp_format` can be set to `unix`, `unix_ms`, `unix_us`, `unix_ns`, or
an accepted [Go "reference time"][time const]. Consult the Go [time][time parse]
package for details and additional examples on how to set the time format. If
`timestamp_format` is omitted `unix` format is assumed as result of the
`timestamp` query.
The `timezone` setting is used to locate the parsed time in the given
timezone. This is helpful for cases where the time does not contain timezone
information, e.g. `2023-03-09 14:04:40` and is not located in _UTC_, which is
the default setting. It is also possible to set the `timezone` to `Local` which
used the configured host timezone.
For time formats with timezone information, e.g. RFC3339, the resulting
timestamp is unchanged. The `timezone` setting is ignored for all `unix`
timestamp formats.
### tags sub-section
[XPath][xpath] queries in the `tag name = query` format to add tags to the
metrics. The specified path can be absolute (starting with `/`) or
relative. Relative paths use the currently selected node as reference.
__NOTE:__ Results of tag-queries will always be converted to strings.
### fields_int sub-section
[XPath][xpath] queries in the `field name = query` format to add integer typed
fields to the metrics. The specified path can be absolute (starting with `/`) or
relative. Relative paths use the currently selected node as reference.
__NOTE:__ Results of field_int-queries will always be converted to
__int64__. The conversion fails in case the query result is not convertible.
### fields sub-section
[XPath][xpath] queries in the `field name = query` format to add non-integer
fields to the metrics. The specified path can be absolute (starting with `/`) or
relative. Relative paths use the currently selected node as reference.
The type of the field is specified in the [XPath][xpath] query using the type
conversion functions of XPath such as `number()`, `boolean()` or `string()` If
no conversion is performed in the query the field will be of type string.
__NOTE: Path conversion functions always succeed even if you convert a text
to float.__
### field_selection, field_name, field_value (optional)
You can specify a [XPath][xpath] query to select a set of nodes forming the
fields of the metric. The specified path can be absolute (starting with `/`) or
relative to the currently selected node. Each node selected by `field_selection`
forms a new field within the metric.
The _name_ and the _value_ of each field can be specified using the optional
`field_name` and `field_value` queries. The queries are relative to the selected
field if not starting with `/`. If not specified the field's _name_ defaults to
the node name and the field's _value_ defaults to the content of the selected
field node.
__NOTE__: `field_name` and `field_value` queries are only evaluated if a
`field_selection` is specified.
Specifying `field_selection` is optional. This is an alternative way to specify
fields especially for documents where the node names are not known a priori or
if there is a large number of fields to be specified. These options can also be
combined with the field specifications above.
__NOTE: Path conversion functions always succeed even if you convert a text
to float.__
### field_name_expansion (optional)
When _true_, field names selected with `field_selection` are expanded to a
_path_ relative to the _selected node_. This is necessary if we select all
leaf nodes as fields and those leaf nodes do not have unique names. That is in
case you have duplicate names in the fields you select you should set this to
`true`.
### tag_selection, tag_name, tag_value (optional)
You can specify a [XPath][xpath] query to select a set of nodes forming the tags
of the metric. The specified path can be absolute (starting with `/`) or
relative to the currently selected node. Each node selected by `tag_selection`
forms a new tag within the metric.
The _name_ and the _value_ of each tag can be specified using the optional
`tag_name` and `tag_value` queries. The queries are relative to the selected tag
if not starting with `/`. If not specified the tag's _name_ defaults to the node
name and the tag's _value_ defaults to the content of the selected tag node.
__NOTE__: `tag_name` and `tag_value` queries are only evaluated if a
`tag_selection` is specified.
Specifying `tag_selection` is optional. This is an alternative way to specify
tags especially for documents where the node names are not known a priori or if
there is a large number of tags to be specified. These options can also be
combined with the tag specifications above.
### tag_name_expansion (optional)
When _true_, tag names selected with `tag_selection` are expanded to a _path_
relative to the _selected node_. This is necessary if we e.g. select all leaf
nodes as tags and those leaf nodes do not have unique names. That is in case you
have duplicate names in the tags you select you should set this to `true`.
## Examples
This `example.xml` file is used in the configuration examples below:
```xml
<?xml version="1.0"?>
<Gateway>
<Name>Main Gateway</Name>
<Timestamp>2020-08-01T15:04:03Z</Timestamp>
<Sequence>12</Sequence>
<Status>ok</Status>
</Gateway>
<Bus>
<Sensor name="Sensor Facility A">
<Variable temperature="20.0"/>
<Variable power="123.4"/>
<Variable frequency="49.78"/>
<Variable consumers="3"/>
<Mode>busy</Mode>
</Sensor>
<Sensor name="Sensor Facility B">
<Variable temperature="23.1"/>
<Variable power="14.3"/>
<Variable frequency="49.78"/>
<Variable consumers="1"/>
<Mode>standby</Mode>
</Sensor>
<Sensor name="Sensor Facility C">
<Variable temperature="19.7"/>
<Variable power="0.02"/>
<Variable frequency="49.78"/>
<Variable consumers="0"/>
<Mode>error</Mode>
</Sensor>
</Bus>
```
### Basic Parsing
This example shows the basic usage of the xml parser.
Config:
```toml
[[inputs.file]]
files = ["example.xml"]
data_format = "xml"
[[inputs.file.xpath]]
[inputs.file.xpath.tags]
gateway = "substring-before(/Gateway/Name, ' ')"
[inputs.file.xpath.fields_int]
seqnr = "/Gateway/Sequence"
[inputs.file.xpath.fields]
ok = "/Gateway/Status = 'ok'"
```
Output:
```text
file,gateway=Main,host=Hugin seqnr=12i,ok=true 1598610830000000000
```
In the _tags_ definition the XPath function `substring-before()` is used to only
extract the sub-string before the space. To get the integer value of
`/Gateway/Sequence` we have to use the _fields_int_ section as there is no XPath
expression to convert node values to integers (only float).
The `ok` field is filled with a boolean by specifying a query comparing the
query result of `/Gateway/Status` with the string _ok_. Use the type conversions
available in the XPath syntax to specify field types.
### Time and metric names
This is an example for using time and name of the metric from the XML document
itself.
Config:
```toml
[[inputs.file]]
files = ["example.xml"]
data_format = "xml"
[[inputs.file.xpath]]
metric_name = "name(/Gateway/Status)"
timestamp = "/Gateway/Timestamp"
timestamp_format = "2006-01-02T15:04:05Z"
[inputs.file.xpath.tags]
gateway = "substring-before(/Gateway/Name, ' ')"
[inputs.file.xpath.fields]
ok = "/Gateway/Status = 'ok'"
```
Output:
```text
Status,gateway=Main,host=Hugin ok=true 1596294243000000000
```
Additionally to the basic parsing example, the metric name is defined as the
name of the `/Gateway/Status` node and the timestamp is derived from the XML
document instead of using the execution time.
### Multi-node selection
For XML documents containing metrics for e.g. multiple devices (like `Sensor`s
in the _example.xml_), multiple metrics can be generated using node
selection. This example shows how to generate a metric for each _Sensor_ in the
example.
Config:
```toml
[[inputs.file]]
files = ["example.xml"]
data_format = "xml"
[[inputs.file.xpath]]
metric_selection = "/Bus/child::Sensor"
metric_name = "string('sensors')"
timestamp = "/Gateway/Timestamp"
timestamp_format = "2006-01-02T15:04:05Z"
[inputs.file.xpath.tags]
name = "substring-after(@name, ' ')"
[inputs.file.xpath.fields_int]
consumers = "Variable/@consumers"
[inputs.file.xpath.fields]
temperature = "number(Variable/@temperature)"
power = "number(Variable/@power)"
frequency = "number(Variable/@frequency)"
ok = "Mode != 'error'"
```
Output:
```text
sensors,host=Hugin,name=Facility\ A consumers=3i,frequency=49.78,ok=true,power=123.4,temperature=20 1596294243000000000
sensors,host=Hugin,name=Facility\ B consumers=1i,frequency=49.78,ok=true,power=14.3,temperature=23.1 1596294243000000000
sensors,host=Hugin,name=Facility\ C consumers=0i,frequency=49.78,ok=false,power=0.02,temperature=19.7 1596294243000000000
```
Using the `metric_selection` option we select all `Sensor` nodes in the XML
document. Please note that all field and tag definitions are relative to these
selected nodes. An exception is the timestamp definition which is relative to
the root node of the XML document.
### Batch field processing with multi-node selection
For XML documents containing metrics with a large number of fields or where the
fields are not known before (e.g. an unknown set of `Variable` nodes in the
_example.xml_), field selectors can be used. This example shows how to generate
a metric for each _Sensor_ in the example with fields derived from the
_Variable_ nodes.
Config:
```toml
[[inputs.file]]
files = ["example.xml"]
data_format = "xml"
[[inputs.file.xpath]]
metric_selection = "/Bus/child::Sensor"
metric_name = "string('sensors')"
timestamp = "/Gateway/Timestamp"
timestamp_format = "2006-01-02T15:04:05Z"
field_selection = "child::Variable"
field_name = "name(@*[1])"
field_value = "number(@*[1])"
[inputs.file.xpath.tags]
name = "substring-after(@name, ' ')"
```
Output:
```text
sensors,host=Hugin,name=Facility\ A consumers=3,frequency=49.78,power=123.4,temperature=20 1596294243000000000
sensors,host=Hugin,name=Facility\ B consumers=1,frequency=49.78,power=14.3,temperature=23.1 1596294243000000000
sensors,host=Hugin,name=Facility\ C consumers=0,frequency=49.78,power=0.02,temperature=19.7 1596294243000000000
```
Using the `metric_selection` option we select all `Sensor` nodes in the XML
document. For each _Sensor_ we then use `field_selection` to select all child
nodes of the sensor as _field-nodes_ Please note that the field selection is
relative to the selected nodes. For each selected _field-node_ we use
`field_name` and `field_value` to determining the field's name and value,
respectively. The `field_name` derives the name of the first attribute of the
node, while `field_value` derives the value of the first attribute and converts
the result to a number.
[xpath lib]: https://github.com/antchfx/xpath
[json]: https://www.json.org/
[msgpack]: https://msgpack.org/
[protobuf]: https://developers.google.com/protocol-buffers
[xml]: https://www.w3.org/XML/
[xpath]: https://www.w3.org/TR/xpath/
[xpather]: http://xpather.com/
[xpath tester]: https://codebeautify.org/Xpath-Tester
[time const]: https://golang.org/pkg/time/#pkg-constants
[time parse]: https://golang.org/pkg/time/#Parse

629
content/telegraf/v1.27/data_formats/input/xpath_msgpack.md Normal file
View File

@ -0,0 +1,629 @@
---
title: XPath MessagePack input data format
list_title: XPath MessagePack
description:
Use the `xpath_msgpack` input data format and XPath expressions to parse MessagePack data into Telegraf metrics.
menu:
telegraf_1_27_ref:
name: XPath MessagePack
weight: 10
parent: Input data formats
metadata: [XPath parser plugin]
---
Use the `xpath_msgpack` input data format, provided by the [XPath parser plugin](https://github.com/influxdata/telegraf/tree/master/plugins/parsers/xpath), with XPath expressions to parse MessagePack data into Telegraf metrics.
For information about supported XPath functions, see [the underlying XPath library][xpath lib].
**NOTE:** The type of fields are specified using [XPath functions][xpath
lib]. The only exceptions are _integer_ fields that need to be specified in a
`fields_int` section.
## Supported data formats
| name | `data_format` setting | comment |
| --------------------------------------- | --------------------- | ------- |
| [Extensible Markup Language (XML)][xml] | `"xml"` | |
| [JSON][json] | `"xpath_json"` | |
| [MessagePack][msgpack] | `"xpath_msgpack"` | |
| [Protocol-buffers][protobuf] | `"xpath_protobuf"` | [see additional parameters](#protocol-buffers-additional-settings)|
### Protocol-buffers additional settings
For using the protocol-buffer format you need to specify additional
(_mandatory_) properties for the parser. Those options are described here.
#### `xpath_protobuf_file` (mandatory)
Use this option to specify the name of the protocol-buffer definition file
(`.proto`).
#### `xpath_protobuf_type` (mandatory)
This option contains the top-level message file to use for deserializing the
data to be parsed. Usually, this is constructed from the `package` name in the
protocol-buffer definition file and the `message` name as `<package
name>.<message name>`.
#### `xpath_protobuf_import_paths` (optional)
In case you import other protocol-buffer definitions within your `.proto` file
(i.e. you use the `import` statement) you can use this option to specify paths
to search for the imported definition file(s). By default the imports are only
searched in `.` which is the current-working-directory, i.e. usually the
directory you are in when starting telegraf.
Imagine you do have multiple protocol-buffer definitions (e.g. `A.proto`,
`B.proto` and `C.proto`) in a directory (e.g. `/data/my_proto_files`) where your
top-level file (e.g. `A.proto`) imports at least one other definition
```protobuf
syntax = "proto3";
package foo;
import "B.proto";
message Measurement {
...
}
```
You should use the following setting
```toml
[[inputs.file]]
files = ["example.dat"]
data_format = "xpath_protobuf"
xpath_protobuf_file = "A.proto"
xpath_protobuf_type = "foo.Measurement"
xpath_protobuf_import_paths = [".", "/data/my_proto_files"]
...
```
#### `xpath_protobuf_skip_bytes` (optional)
This option allows to skip a number of bytes before trying to parse
the protocol-buffer message. This is useful in cases where the raw data
has a header e.g. for the message length or in case of GRPC messages.
This is a list of known headers and the corresponding values for
`xpath_protobuf_skip_bytes`
| name | setting | comment |
| --------------------------------------- | ------- | ------- |
| [GRPC protocol][GRPC] | 5 | GRPC adds a 5-byte header for _Length-Prefixed-Messages_ |
| [PowerDNS logging][PDNS] | 2 | Sent messages contain a 2-byte header containing the message length |
[GRPC]: https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md
[PDNS]: https://docs.powerdns.com/recursor/lua-config/protobuf.html
## Configuration
```toml
[[inputs.file]]
files = ["example.xml"]
## Data format to consume.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "xml"
## PROTOCOL-BUFFER definitions
## Protocol-buffer definition file
# xpath_protobuf_file = "sparkplug_b.proto"
## Name of the protocol-buffer message type to use in a fully qualified form.
# xpath_protobuf_type = "org.eclipse.tahu.protobuf.Payload"
## List of paths to use when looking up imported protocol-buffer definition files.
# xpath_protobuf_import_paths = ["."]
## Number of (header) bytes to ignore before parsing the message.
# xpath_protobuf_skip_bytes = 0
## Print the internal XML document when in debug logging mode.
## This is especially useful when using the parser with non-XML formats like protocol-buffers
## to get an idea on the expression necessary to derive fields etc.
# xpath_print_document = false
## Allow the results of one of the parsing sections to be empty.
## Useful when not all selected files have the exact same structure.
# xpath_allow_empty_selection = false
## Get native data-types for all data-format that contain type information.
## Currently, protobuf, msgpack and JSON support native data-types
# xpath_native_types = false
## Multiple parsing sections are allowed
[[inputs.file.xpath]]
## Optional: XPath-query to select a subset of nodes from the XML document.
# metric_selection = "/Bus/child::Sensor"
## Optional: XPath-query to set the metric (measurement) name.
# metric_name = "string('example')"
## Optional: Query to extract metric timestamp.
## If not specified the time of execution is used.
# timestamp = "/Gateway/Timestamp"
## Optional: Format of the timestamp determined by the query above.
## This can be any of "unix", "unix_ms", "unix_us", "unix_ns" or a valid Golang
## time format. If not specified, a "unix" timestamp (in seconds) is expected.
# timestamp_format = "2006-01-02T15:04:05Z"
## Optional: Timezone of the parsed time
## This will locate the parsed time to the given timezone. Please note that
## for times with timezone-offsets (e.g. RFC3339) the timestamp is unchanged.
## This is ignored for all (unix) timestamp formats.
# timezone = "UTC"
## Optional: List of fields to convert to hex-strings if they are
## containing byte-arrays. This might be the case for e.g. protocol-buffer
## messages encoding data as byte-arrays. Wildcard patterns are allowed.
## By default, all byte-array-fields are converted to string.
# fields_bytes_as_hex = []
## Tag definitions using the given XPath queries.
[inputs.file.xpath.tags]
name = "substring-after(Sensor/@name, ' ')"
device = "string('the ultimate sensor')"
## Integer field definitions using XPath queries.
[inputs.file.xpath.fields_int]
consumers = "Variable/@consumers"
## Non-integer field definitions using XPath queries.
## The field type is defined using XPath expressions such as number(), boolean() or string(). If no conversion is performed the field will be of type string.
[inputs.file.xpath.fields]
temperature = "number(Variable/@temperature)"
power = "number(Variable/@power)"
frequency = "number(Variable/@frequency)"
ok = "Mode != 'ok'"
```
In this configuration mode, you explicitly specify the field and tags you want
to scrape from your data.
A configuration can contain muliple _xpath_ subsections (for example, the file plugin
to process the xml-string multiple times). Consult the [XPath syntax][xpath] and
the [underlying library's functions][xpath lib] for details and help regarding
XPath queries. Consider using an XPath tester such as [xpather.com][xpather] or
[Code Beautify's XPath Tester][xpath tester] for help developing and debugging
your query.
## Configuration (batch)
Alternatively to the configuration above, fields can also be specified in a
batch way. So contrary to specify the fields in a section, you can define a
`name` and a `value` selector used to determine the name and value of the fields
in the metric.
```toml
[[inputs.file]]
files = ["example.xml"]
## Data format to consume.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "xml"
## PROTOCOL-BUFFER definitions
## Protocol-buffer definition file
# xpath_protobuf_file = "sparkplug_b.proto"
## Name of the protocol-buffer message type to use in a fully qualified form.
# xpath_protobuf_type = "org.eclipse.tahu.protobuf.Payload"
## List of paths to use when looking up imported protocol-buffer definition files.
# xpath_protobuf_import_paths = ["."]
## Print the internal XML document when in debug logging mode.
## This is especially useful when using the parser with non-XML formats like protocol-buffers
## to get an idea on the expression necessary to derive fields etc.
# xpath_print_document = false
## Allow the results of one of the parsing sections to be empty.
## Useful when not all selected files have the exact same structure.
# xpath_allow_empty_selection = false
## Get native data-types for all data-format that contain type information.
## Currently, protobuf, msgpack and JSON support native data-types
# xpath_native_types = false
## Multiple parsing sections are allowed
[[inputs.file.xpath]]
## Optional: XPath-query to select a subset of nodes from the XML document.
metric_selection = "/Bus/child::Sensor"
## Optional: XPath-query to set the metric (measurement) name.
# metric_name = "string('example')"
## Optional: Query to extract metric timestamp.
## If not specified the time of execution is used.
# timestamp = "/Gateway/Timestamp"
## Optional: Format of the timestamp determined by the query above.
## This can be any of "unix", "unix_ms", "unix_us", "unix_ns" or a valid Golang
## time format. If not specified, a "unix" timestamp (in seconds) is expected.
# timestamp_format = "2006-01-02T15:04:05Z"
## Field specifications using a selector.
field_selection = "child::*"
## Optional: Queries to specify field name and value.
## These options are only to be used in combination with 'field_selection'!
## By default the node name and node content is used if a field-selection
## is specified.
# field_name = "name()"
# field_value = "."
## Optional: Expand field names relative to the selected node
## This allows to flatten out nodes with non-unique names in the subtree
# field_name_expansion = false
## Tag specifications using a selector.
## tag_selection = "child::*"
## Optional: Queries to specify tag name and value.
## These options are only to be used in combination with 'tag_selection'!
## By default the node name and node content is used if a tag-selection
## is specified.
# tag_name = "name()"
# tag_value = "."
## Optional: Expand tag names relative to the selected node
## This allows to flatten out nodes with non-unique names in the subtree
# tag_name_expansion = false
## Tag definitions using the given XPath queries.
[inputs.file.xpath.tags]
name = "substring-after(Sensor/@name, ' ')"
device = "string('the ultimate sensor')"
```
**Please note**: The resulting fields are _always_ of type string.
It is also possible to specify a mixture of the two alternative ways of
specifying fields. In this case, _explicitly_ defined tags and fields take
_precedence_ over the batch instances if both use the same tag or field name.
### metric_selection (optional)
You can specify a [XPath][xpath] query to select a subset of nodes from the XML
document, each used to generate a new metrics with the specified fields, tags
etc.
Relative queries in subsequent queries are relative to the
`metric_selection`. To specify absolute paths, start the query with a
slash (`/`).
Specifying `metric_selection` is optional. If not specified, all relative queries
are relative to the root node of the XML document.
### metric_name (optional)
By specifying `metric_name` you can override the metric/measurement name with
the result of the given [XPath][xpath] query. If not specified, the default
metric name is used.
### timestamp, timestamp_format, timezone (optional)
By default, the current time is used for all created metrics. To set the
time from values in the XML document you can specify a [XPath][xpath] query in
`timestamp` and set the format in `timestamp_format`.
The `timestamp_format` can be set to `unix`, `unix_ms`, `unix_us`, `unix_ns`, or
an accepted [Go "reference time"][time const]. Consult the Go [time][time parse]
package for details and additional examples on how to set the time format. If
`timestamp_format` is omitted `unix` format is assumed as result of the
`timestamp` query.
The `timezone` setting will be used to locate the parsed time in the given
timezone. This is helpful for cases where the time does not contain timezone
information, e.g. `2023-03-09 14:04:40` and is not located in _UTC_, which is
the default setting. It is also possible to set the `timezone` to `Local` which
used the configured host timezone.
For time formats with timezone information, e.g. RFC3339, the resulting
timestamp is unchanged. The `timezone` setting is ignored for all `unix`
timestamp formats.
### tags sub-section
[XPath][xpath] queries in the `tag name = query` format to add tags to the
metrics. The specified path can be absolute (starting with `/`) or
relative. Relative paths use the currently selected node as reference.
__NOTE:__ Results of tag-queries will always be converted to strings.
### fields_int sub-section
[XPath][xpath] queries in the `field name = query` format to add integer typed
fields to the metrics. The specified path can be absolute (starting with `/`) or
relative. Relative paths use the currently selected node as reference.
__NOTE:__ Results of field_int-queries will always be converted to
__int64__. The conversion will fail in case the query result is not convertible!
### fields sub-section
[XPath][xpath] queries in the `field name = query` format to add non-integer
fields to the metrics. The specified path can be absolute (starting with `/`) or
relative. Relative paths use the currently selected node as reference.
The type of the field is specified in the [XPath][xpath] query using the type
conversion functions of XPath such as `number()`, `boolean()` or `string()` If
no conversion is performed in the query the field will be of type string.
__NOTE: Path conversion functions will always succeed even if you convert a text
to float!__
### field_selection, field_name, field_value (optional)
You can specify a [XPath][xpath] query to select a set of nodes forming the
fields of the metric. The specified path can be absolute (starting with `/`) or
relative to the currently selected node. Each node selected by `field_selection`
forms a new field within the metric.
The _name_ and the _value_ of each field can be specified using the optional
`field_name` and `field_value` queries. The queries are relative to the selected
field if not starting with `/`. If not specified the field's _name_ defaults to
the node name and the field's _value_ defaults to the content of the selected
field node.
__NOTE__: `field_name` and `field_value` queries are only evaluated if a
`field_selection` is specified.
Specifying `field_selection` is optional. This is an alternative way to specify
fields especially for documents where the node names are not known a priori or
if there is a large number of fields to be specified. These options can also be
combined with the field specifications above.
__NOTE: Path conversion functions will always succeed even if you convert a text
to float!__
### field_name_expansion (optional)
When _true_, field names selected with `field_selection` are expanded to a
_path_ relative to the _selected node_. This is necessary if we e.g. select all
leaf nodes as fields and those leaf nodes do not have unique names. That is in
case you have duplicate names in the fields you select you should set this to
`true`.
### tag_selection, tag_name, tag_value (optional)
You can specify a [XPath][xpath] query to select a set of nodes forming the tags
of the metric. The specified path can be absolute (starting with `/`) or
relative to the currently selected node. Each node selected by `tag_selection`
forms a new tag within the metric.
The _name_ and the _value_ of each tag can be specified using the optional
`tag_name` and `tag_value` queries. The queries are relative to the selected tag
if not starting with `/`. If not specified the tag's _name_ defaults to the node
name and the tag's _value_ defaults to the content of the selected tag node.
__NOTE__: `tag_name` and `tag_value` queries are only evaluated if a
`tag_selection` is specified.
Specifying `tag_selection` is optional. This is an alternative way to specify
tags especially for documents where the node names are not known a priori or if
there is a large number of tags to be specified. These options can also be
combined with the tag specifications above.
### tag_name_expansion (optional)
When _true_, tag names selected with `tag_selection` are expanded to a _path_
relative to the _selected node_. This is necessary if we e.g. select all leaf
nodes as tags and those leaf nodes do not have unique names. That is in case you
have duplicate names in the tags you select you should set this to `true`.
## Examples
This `example.xml` file is used in the configuration examples below:
```xml
<?xml version="1.0"?>
<Gateway>
<Name>Main Gateway</Name>
<Timestamp>2020-08-01T15:04:03Z</Timestamp>
<Sequence>12</Sequence>
<Status>ok</Status>
</Gateway>
<Bus>
<Sensor name="Sensor Facility A">
<Variable temperature="20.0"/>
<Variable power="123.4"/>
<Variable frequency="49.78"/>
<Variable consumers="3"/>
<Mode>busy</Mode>
</Sensor>
<Sensor name="Sensor Facility B">
<Variable temperature="23.1"/>
<Variable power="14.3"/>
<Variable frequency="49.78"/>
<Variable consumers="1"/>
<Mode>standby</Mode>
</Sensor>
<Sensor name="Sensor Facility C">
<Variable temperature="19.7"/>
<Variable power="0.02"/>
<Variable frequency="49.78"/>
<Variable consumers="0"/>
<Mode>error</Mode>
</Sensor>
</Bus>
```
### Basic Parsing
This example shows the basic usage of the xml parser.
Config:
```toml
[[inputs.file]]
files = ["example.xml"]
data_format = "xml"
[[inputs.file.xpath]]
[inputs.file.xpath.tags]
gateway = "substring-before(/Gateway/Name, ' ')"
[inputs.file.xpath.fields_int]
seqnr = "/Gateway/Sequence"
[inputs.file.xpath.fields]
ok = "/Gateway/Status = 'ok'"
```
Output:
```text
file,gateway=Main,host=Hugin seqnr=12i,ok=true 1598610830000000000
```
In the _tags_ definition the XPath function `substring-before()` is used to only
extract the sub-string before the space. To get the integer value of
`/Gateway/Sequence` we have to use the _fields_int_ section as there is no XPath
expression to convert node values to integers (only float).
The `ok` field is filled with a boolean by specifying a query comparing the
query result of `/Gateway/Status` with the string _ok_. Use the type conversions
available in the XPath syntax to specify field types.
### Time and metric names
This is an example for using time and name of the metric from the XML document
itself.
Config:
```toml
[[inputs.file]]
files = ["example.xml"]
data_format = "xml"
[[inputs.file.xpath]]
metric_name = "name(/Gateway/Status)"
timestamp = "/Gateway/Timestamp"
timestamp_format = "2006-01-02T15:04:05Z"
[inputs.file.xpath.tags]
gateway = "substring-before(/Gateway/Name, ' ')"
[inputs.file.xpath.fields]
ok = "/Gateway/Status = 'ok'"
```
Output:
```text
Status,gateway=Main,host=Hugin ok=true 1596294243000000000
```
Additionally to the basic parsing example, the metric name is defined as the
name of the `/Gateway/Status` node and the timestamp is derived from the XML
document instead of using the execution time.
### Multi-node selection
For XML documents containing metrics for e.g. multiple devices (like `Sensor`s
in the _example.xml_), multiple metrics can be generated using node
selection. This example shows how to generate a metric for each _Sensor_ in the
example.
Config:
```toml
[[inputs.file]]
files = ["example.xml"]
data_format = "xml"
[[inputs.file.xpath]]
metric_selection = "/Bus/child::Sensor"
metric_name = "string('sensors')"
timestamp = "/Gateway/Timestamp"
timestamp_format = "2006-01-02T15:04:05Z"
[inputs.file.xpath.tags]
name = "substring-after(@name, ' ')"
[inputs.file.xpath.fields_int]
consumers = "Variable/@consumers"
[inputs.file.xpath.fields]
temperature = "number(Variable/@temperature)"
power = "number(Variable/@power)"
frequency = "number(Variable/@frequency)"
ok = "Mode != 'error'"
```
Output:
```text
sensors,host=Hugin,name=Facility\ A consumers=3i,frequency=49.78,ok=true,power=123.4,temperature=20 1596294243000000000
sensors,host=Hugin,name=Facility\ B consumers=1i,frequency=49.78,ok=true,power=14.3,temperature=23.1 1596294243000000000
sensors,host=Hugin,name=Facility\ C consumers=0i,frequency=49.78,ok=false,power=0.02,temperature=19.7 1596294243000000000
```
Using the `metric_selection` option we select all `Sensor` nodes in the XML
document. Please note that all field and tag definitions are relative to these
selected nodes. An exception is the timestamp definition which is relative to
the root node of the XML document.
### Batch field processing with multi-node selection
For XML documents containing metrics with a large number of fields or where the
fields are not known before (e.g. an unknown set of `Variable` nodes in the
_example.xml_), field selectors can be used. This example shows how to generate
a metric for each _Sensor_ in the example with fields derived from the
_Variable_ nodes.
Config:
```toml
[[inputs.file]]
files = ["example.xml"]
data_format = "xml"
[[inputs.file.xpath]]
metric_selection = "/Bus/child::Sensor"
metric_name = "string('sensors')"
timestamp = "/Gateway/Timestamp"
timestamp_format = "2006-01-02T15:04:05Z"
field_selection = "child::Variable"
field_name = "name(@*[1])"
field_value = "number(@*[1])"
[inputs.file.xpath.tags]
name = "substring-after(@name, ' ')"
```
Output:
```text
sensors,host=Hugin,name=Facility\ A consumers=3,frequency=49.78,power=123.4,temperature=20 1596294243000000000
sensors,host=Hugin,name=Facility\ B consumers=1,frequency=49.78,power=14.3,temperature=23.1 1596294243000000000
sensors,host=Hugin,name=Facility\ C consumers=0,frequency=49.78,power=0.02,temperature=19.7 1596294243000000000
```
Using the `metric_selection` option we select all `Sensor` nodes in the XML
document. For each _Sensor_ we then use `field_selection` to select all child
nodes of the sensor as _field-nodes_ Please note that the field selection is
relative to the selected nodes. For each selected _field-node_ we use
`field_name` and `field_value` to determining the field's name and value,
respectively. The `field_name` derives the name of the first attribute of the
node, while `field_value` derives the value of the first attribute and converts
the result to a number.
[xpath lib]: https://github.com/antchfx/xpath
[json]: https://www.json.org/
[msgpack]: https://msgpack.org/
[protobuf]: https://developers.google.com/protocol-buffers
[xml]: https://www.w3.org/XML/
[xpath]: https://www.w3.org/TR/xpath/
[xpather]: http://xpather.com/
[xpath tester]: https://codebeautify.org/Xpath-Tester
[time const]: https://golang.org/pkg/time/#pkg-constants
[time parse]: https://golang.org/pkg/time/#Parse

629
content/telegraf/v1.27/data_formats/input/xpath_protobuf.md Normal file
View File

@ -0,0 +1,629 @@
---
title: XPath Protocol Buffers input data format
list_title: XPath Protocol Buffers
description:
Use the `xpath_protobuf` input data format and XPath expressions to parse protobuf (Protocol Buffer) data into Telegraf metrics.
menu:
telegraf_1_27_ref:
name: XPath Protocol Buffers
weight: 10
parent: Input data formats
metadata: [XPath parser plugin]
---
Use the `xpath_protobuf` input data format, provided by the [XPath parser plugin](https://github.com/influxdata/telegraf/tree/master/plugins/parsers/xpath), with XPath expressions to parse Protocol Buffer data into Telegraf metrics.
For information about supported XPath functions, see [the underlying XPath library][xpath lib].
**NOTE:** The type of fields are specified using [XPath functions][xpath
lib]. The only exceptions are _integer_ fields that need to be specified in a
`fields_int` section.
## Supported data formats
| name | `data_format` setting | comment |
| --------------------------------------- | --------------------- | ------- |
| [Extensible Markup Language (XML)][xml] | `"xml"` | |
| [JSON][json] | `"xpath_json"` | |
| [MessagePack][msgpack] | `"xpath_msgpack"` | |
| [Protocol-buffers][protobuf] | `"xpath_protobuf"` | [see additional parameters](#protocol-buffers-additional-settings)|
### Protocol-buffers additional settings
For using the protocol-buffer format you need to specify additional
(_mandatory_) properties for the parser. Those options are described here.
#### `xpath_protobuf_file` (mandatory)
Use this option to specify the name of the protocol-buffer definition file
(`.proto`).
#### `xpath_protobuf_type` (mandatory)
This option contains the top-level message file to use for deserializing the
data to be parsed. Usually, this is constructed from the `package` name in the
protocol-buffer definition file and the `message` name as `<package
name>.<message name>`.
#### `xpath_protobuf_import_paths` (optional)
In case you import other protocol-buffer definitions within your `.proto` file
(i.e. you use the `import` statement) you can use this option to specify paths
to search for the imported definition file(s). By default the imports are only
searched in `.` which is the current-working-directory, i.e. usually the
directory you are in when starting telegraf.
Imagine you do have multiple protocol-buffer definitions (e.g. `A.proto`,
`B.proto` and `C.proto`) in a directory (e.g. `/data/my_proto_files`) where your
top-level file (e.g. `A.proto`) imports at least one other definition
```protobuf
syntax = "proto3";
package foo;
import "B.proto";
message Measurement {
...
}
```
You should use the following setting
```toml
[[inputs.file]]
files = ["example.dat"]
data_format = "xpath_protobuf"
xpath_protobuf_file = "A.proto"
xpath_protobuf_type = "foo.Measurement"
xpath_protobuf_import_paths = [".", "/data/my_proto_files"]
...
```
#### `xpath_protobuf_skip_bytes` (optional)
This option allows to skip a number of bytes before trying to parse
the protocol-buffer message. This is useful in cases where the raw data
has a header e.g. for the message length or in case of GRPC messages.
This is a list of known headers and the corresponding values for
`xpath_protobuf_skip_bytes`
| name | setting | comment |
| --------------------------------------- | ------- | ------- |
| [GRPC protocol][GRPC] | 5 | GRPC adds a 5-byte header for _Length-Prefixed-Messages_ |
| [PowerDNS logging][PDNS] | 2 | Sent messages contain a 2-byte header containing the message length |
[GRPC]: https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md
[PDNS]: https://docs.powerdns.com/recursor/lua-config/protobuf.html
## Configuration
```toml
[[inputs.file]]
files = ["example.xml"]
## Data format to consume.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "xml"
## PROTOCOL-BUFFER definitions
## Protocol-buffer definition file
# xpath_protobuf_file = "sparkplug_b.proto"
## Name of the protocol-buffer message type to use in a fully qualified form.
# xpath_protobuf_type = "org.eclipse.tahu.protobuf.Payload"
## List of paths to use when looking up imported protocol-buffer definition files.
# xpath_protobuf_import_paths = ["."]
## Number of (header) bytes to ignore before parsing the message.
# xpath_protobuf_skip_bytes = 0
## Print the internal XML document when in debug logging mode.
## This is especially useful when using the parser with non-XML formats like protocol-buffers
## to get an idea on the expression necessary to derive fields etc.
# xpath_print_document = false
## Allow the results of one of the parsing sections to be empty.
## Useful when not all selected files have the exact same structure.
# xpath_allow_empty_selection = false
## Get native data-types for all data-format that contain type information.
## Currently, protobuf, msgpack and JSON support native data-types
# xpath_native_types = false
## Multiple parsing sections are allowed
[[inputs.file.xpath]]
## Optional: XPath-query to select a subset of nodes from the XML document.
# metric_selection = "/Bus/child::Sensor"
## Optional: XPath-query to set the metric (measurement) name.
# metric_name = "string('example')"
## Optional: Query to extract metric timestamp.
## If not specified the time of execution is used.
# timestamp = "/Gateway/Timestamp"
## Optional: Format of the timestamp determined by the query above.
## This can be any of "unix", "unix_ms", "unix_us", "unix_ns" or a valid Golang
## time format. If not specified, a "unix" timestamp (in seconds) is expected.
# timestamp_format = "2006-01-02T15:04:05Z"
## Optional: Timezone of the parsed time
## This will locate the parsed time to the given timezone. Please note that
## for times with timezone-offsets (e.g. RFC3339) the timestamp is unchanged.
## This is ignored for all (unix) timestamp formats.
# timezone = "UTC"
## Optional: List of fields to convert to hex-strings if they are
## containing byte-arrays. This might be the case for e.g. protocol-buffer
## messages encoding data as byte-arrays. Wildcard patterns are allowed.
## By default, all byte-array-fields are converted to string.
# fields_bytes_as_hex = []
## Tag definitions using the given XPath queries.
[inputs.file.xpath.tags]
name = "substring-after(Sensor/@name, ' ')"
device = "string('the ultimate sensor')"
## Integer field definitions using XPath queries.
[inputs.file.xpath.fields_int]
consumers = "Variable/@consumers"
## Non-integer field definitions using XPath queries.
## The field type is defined using XPath expressions such as number(), boolean() or string(). If no conversion is performed the field will be of type string.
[inputs.file.xpath.fields]
temperature = "number(Variable/@temperature)"
power = "number(Variable/@power)"
frequency = "number(Variable/@frequency)"
ok = "Mode != 'ok'"
```
In this configuration mode, you explicitly specify the field and tags you want
to scrape from your data.
A configuration can contain muliple _xpath_ subsections (for example, the file plugin
to process the xml-string multiple times). Consult the [XPath syntax][xpath] and
the [underlying library's functions][xpath lib] for details and help regarding
XPath queries. Consider using an XPath tester such as [xpather.com][xpather] or
[Code Beautify's XPath Tester][xpath tester] for help developing and debugging
your query.
## Configuration (batch)
Alternatively to the configuration above, fields can also be specified in a
batch way. So contrary to specify the fields in a section, you can define a
`name` and a `value` selector used to determine the name and value of the fields
in the metric.
```toml
[[inputs.file]]
files = ["example.xml"]
## Data format to consume.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "xml"
## PROTOCOL-BUFFER definitions
## Protocol-buffer definition file
# xpath_protobuf_file = "sparkplug_b.proto"
## Name of the protocol-buffer message type to use in a fully qualified form.
# xpath_protobuf_type = "org.eclipse.tahu.protobuf.Payload"
## List of paths to use when looking up imported protocol-buffer definition files.
# xpath_protobuf_import_paths = ["."]
## Print the internal XML document when in debug logging mode.
## This is especially useful when using the parser with non-XML formats like protocol-buffers
## to get an idea on the expression necessary to derive fields etc.
# xpath_print_document = false
## Allow the results of one of the parsing sections to be empty.
## Useful when not all selected files have the exact same structure.
# xpath_allow_empty_selection = false
## Get native data-types for all data-format that contain type information.
## Currently, protobuf, msgpack and JSON support native data-types
# xpath_native_types = false
## Multiple parsing sections are allowed
[[inputs.file.xpath]]
## Optional: XPath-query to select a subset of nodes from the XML document.
metric_selection = "/Bus/child::Sensor"
## Optional: XPath-query to set the metric (measurement) name.
# metric_name = "string('example')"
## Optional: Query to extract metric timestamp.
## If not specified the time of execution is used.
# timestamp = "/Gateway/Timestamp"
## Optional: Format of the timestamp determined by the query above.
## This can be any of "unix", "unix_ms", "unix_us", "unix_ns" or a valid Golang
## time format. If not specified, a "unix" timestamp (in seconds) is expected.
# timestamp_format = "2006-01-02T15:04:05Z"
## Field specifications using a selector.
field_selection = "child::*"
## Optional: Queries to specify field name and value.
## These options are only to be used in combination with 'field_selection'!
## By default the node name and node content is used if a field-selection
## is specified.
# field_name = "name()"
# field_value = "."
## Optional: Expand field names relative to the selected node
## This allows to flatten out nodes with non-unique names in the subtree
# field_name_expansion = false
## Tag specifications using a selector.
## tag_selection = "child::*"
## Optional: Queries to specify tag name and value.
## These options are only to be used in combination with 'tag_selection'!
## By default the node name and node content is used if a tag-selection
## is specified.
# tag_name = "name()"
# tag_value = "."
## Optional: Expand tag names relative to the selected node
## This allows to flatten out nodes with non-unique names in the subtree
# tag_name_expansion = false
## Tag definitions using the given XPath queries.
[inputs.file.xpath.tags]
name = "substring-after(Sensor/@name, ' ')"
device = "string('the ultimate sensor')"
```
**Please note**: The resulting fields are _always_ of type string.
It is also possible to specify a mixture of the two alternative ways of
specifying fields. In this case, _explicitly_ defined tags and fields take
_precedence_ over the batch instances if both use the same tag or field name.
### metric_selection (optional)
You can specify a [XPath][xpath] query to select a subset of nodes from the XML
document, each used to generate a new metrics with the specified fields, tags
etc.
Relative queries in subsequent queries are relative to the
`metric_selection`. To specify absolute paths, start the query with a
slash (`/`).
Specifying `metric_selection` is optional. If not specified, all relative queries
are relative to the root node of the XML document.
### metric_name (optional)
By specifying `metric_name` you can override the metric/measurement name with
the result of the given [XPath][xpath] query. If not specified, the default
metric name is used.
### timestamp, timestamp_format, timezone (optional)
By default, the current time is used for all created metrics. To set the
time from values in the XML document you can specify a [XPath][xpath] query in
`timestamp` and set the format in `timestamp_format`.
The `timestamp_format` can be set to `unix`, `unix_ms`, `unix_us`, `unix_ns`, or
an accepted [Go "reference time"][time const]. Consult the Go [time][time parse]
package for details and additional examples on how to set the time format. If
`timestamp_format` is omitted `unix` format is assumed as result of the
`timestamp` query.
The `timezone` setting will be used to locate the parsed time in the given
timezone. This is helpful for cases where the time does not contain timezone
information, e.g. `2023-03-09 14:04:40` and is not located in _UTC_, which is
the default setting. It is also possible to set the `timezone` to `Local` which
used the configured host timezone.
For time formats with timezone information, e.g. RFC3339, the resulting
timestamp is unchanged. The `timezone` setting is ignored for all `unix`
timestamp formats.
### tags sub-section
[XPath][xpath] queries in the `tag name = query` format to add tags to the
metrics. The specified path can be absolute (starting with `/`) or
relative. Relative paths use the currently selected node as reference.
__NOTE:__ Results of tag-queries will always be converted to strings.
### fields_int sub-section
[XPath][xpath] queries in the `field name = query` format to add integer typed
fields to the metrics. The specified path can be absolute (starting with `/`) or
relative. Relative paths use the currently selected node as reference.
__NOTE:__ Results of field_int-queries will always be converted to
__int64__. The conversion will fail in case the query result is not convertible!
### fields sub-section
[XPath][xpath] queries in the `field name = query` format to add non-integer
fields to the metrics. The specified path can be absolute (starting with `/`) or
relative. Relative paths use the currently selected node as reference.
The type of the field is specified in the [XPath][xpath] query using the type
conversion functions of XPath such as `number()`, `boolean()` or `string()` If
no conversion is performed in the query the field will be of type string.
__NOTE: Path conversion functions will always succeed even if you convert a text
to float!__
### field_selection, field_name, field_value (optional)
You can specify a [XPath][xpath] query to select a set of nodes forming the
fields of the metric. The specified path can be absolute (starting with `/`) or
relative to the currently selected node. Each node selected by `field_selection`
forms a new field within the metric.
The _name_ and the _value_ of each field can be specified using the optional
`field_name` and `field_value` queries. The queries are relative to the selected
field if not starting with `/`. If not specified the field's _name_ defaults to
the node name and the field's _value_ defaults to the content of the selected
field node.
__NOTE__: `field_name` and `field_value` queries are only evaluated if a
`field_selection` is specified.
Specifying `field_selection` is optional. This is an alternative way to specify
fields especially for documents where the node names are not known a priori or
if there is a large number of fields to be specified. These options can also be
combined with the field specifications above.
__NOTE: Path conversion functions will always succeed even if you convert a text
to float!__
### field_name_expansion (optional)
When _true_, field names selected with `field_selection` are expanded to a
_path_ relative to the _selected node_. This is necessary if we e.g. select all
leaf nodes as fields and those leaf nodes do not have unique names. That is in
case you have duplicate names in the fields you select you should set this to
`true`.
### tag_selection, tag_name, tag_value (optional)
You can specify a [XPath][xpath] query to select a set of nodes forming the tags
of the metric. The specified path can be absolute (starting with `/`) or
relative to the currently selected node. Each node selected by `tag_selection`
forms a new tag within the metric.
The _name_ and the _value_ of each tag can be specified using the optional
`tag_name` and `tag_value` queries. The queries are relative to the selected tag
if not starting with `/`. If not specified the tag's _name_ defaults to the node
name and the tag's _value_ defaults to the content of the selected tag node.
__NOTE__: `tag_name` and `tag_value` queries are only evaluated if a
`tag_selection` is specified.
Specifying `tag_selection` is optional. This is an alternative way to specify
tags especially for documents where the node names are not known a priori or if
there is a large number of tags to be specified. These options can also be
combined with the tag specifications above.
### tag_name_expansion (optional)
When _true_, tag names selected with `tag_selection` are expanded to a _path_
relative to the _selected node_. This is necessary if we e.g. select all leaf
nodes as tags and those leaf nodes do not have unique names. That is in case you
have duplicate names in the tags you select you should set this to `true`.
## Examples
This `example.xml` file is used in the configuration examples below:
```xml
<?xml version="1.0"?>
<Gateway>
<Name>Main Gateway</Name>
<Timestamp>2020-08-01T15:04:03Z</Timestamp>
<Sequence>12</Sequence>
<Status>ok</Status>
</Gateway>
<Bus>
<Sensor name="Sensor Facility A">
<Variable temperature="20.0"/>
<Variable power="123.4"/>
<Variable frequency="49.78"/>
<Variable consumers="3"/>
<Mode>busy</Mode>
</Sensor>
<Sensor name="Sensor Facility B">
<Variable temperature="23.1"/>
<Variable power="14.3"/>
<Variable frequency="49.78"/>
<Variable consumers="1"/>
<Mode>standby</Mode>
</Sensor>
<Sensor name="Sensor Facility C">
<Variable temperature="19.7"/>
<Variable power="0.02"/>
<Variable frequency="49.78"/>
<Variable consumers="0"/>
<Mode>error</Mode>
</Sensor>
</Bus>
```
### Basic Parsing
This example shows the basic usage of the xml parser.
Config:
```toml
[[inputs.file]]
files = ["example.xml"]
data_format = "xml"
[[inputs.file.xpath]]
[inputs.file.xpath.tags]
gateway = "substring-before(/Gateway/Name, ' ')"
[inputs.file.xpath.fields_int]
seqnr = "/Gateway/Sequence"
[inputs.file.xpath.fields]
ok = "/Gateway/Status = 'ok'"
```
Output:
```text
file,gateway=Main,host=Hugin seqnr=12i,ok=true 1598610830000000000
```
In the _tags_ definition the XPath function `substring-before()` is used to only
extract the sub-string before the space. To get the integer value of
`/Gateway/Sequence` we have to use the _fields_int_ section as there is no XPath
expression to convert node values to integers (only float).
The `ok` field is filled with a boolean by specifying a query comparing the
query result of `/Gateway/Status` with the string _ok_. Use the type conversions
available in the XPath syntax to specify field types.
### Time and metric names
This is an example for using time and name of the metric from the XML document
itself.
Config:
```toml
[[inputs.file]]
files = ["example.xml"]
data_format = "xml"
[[inputs.file.xpath]]
metric_name = "name(/Gateway/Status)"
timestamp = "/Gateway/Timestamp"
timestamp_format = "2006-01-02T15:04:05Z"
[inputs.file.xpath.tags]
gateway = "substring-before(/Gateway/Name, ' ')"
[inputs.file.xpath.fields]
ok = "/Gateway/Status = 'ok'"
```
Output:
```text
Status,gateway=Main,host=Hugin ok=true 1596294243000000000
```
Additionally to the basic parsing example, the metric name is defined as the
name of the `/Gateway/Status` node and the timestamp is derived from the XML
document instead of using the execution time.
### Multi-node selection
For XML documents containing metrics for e.g. multiple devices (like `Sensor`s
in the _example.xml_), multiple metrics can be generated using node
selection. This example shows how to generate a metric for each _Sensor_ in the
example.
Config:
```toml
[[inputs.file]]
files = ["example.xml"]
data_format = "xml"
[[inputs.file.xpath]]
metric_selection = "/Bus/child::Sensor"
metric_name = "string('sensors')"
timestamp = "/Gateway/Timestamp"
timestamp_format = "2006-01-02T15:04:05Z"
[inputs.file.xpath.tags]
name = "substring-after(@name, ' ')"
[inputs.file.xpath.fields_int]
consumers = "Variable/@consumers"
[inputs.file.xpath.fields]
temperature = "number(Variable/@temperature)"
power = "number(Variable/@power)"
frequency = "number(Variable/@frequency)"
ok = "Mode != 'error'"
```
Output:
```text
sensors,host=Hugin,name=Facility\ A consumers=3i,frequency=49.78,ok=true,power=123.4,temperature=20 1596294243000000000
sensors,host=Hugin,name=Facility\ B consumers=1i,frequency=49.78,ok=true,power=14.3,temperature=23.1 1596294243000000000
sensors,host=Hugin,name=Facility\ C consumers=0i,frequency=49.78,ok=false,power=0.02,temperature=19.7 1596294243000000000
```
Using the `metric_selection` option we select all `Sensor` nodes in the XML
document. Please note that all field and tag definitions are relative to these
selected nodes. An exception is the timestamp definition which is relative to
the root node of the XML document.
### Batch field processing with multi-node selection
For XML documents containing metrics with a large number of fields or where the
fields are not known before (e.g. an unknown set of `Variable` nodes in the
_example.xml_), field selectors can be used. This example shows how to generate
a metric for each _Sensor_ in the example with fields derived from the
_Variable_ nodes.
Config:
```toml
[[inputs.file]]
files = ["example.xml"]
data_format = "xml"
[[inputs.file.xpath]]
metric_selection = "/Bus/child::Sensor"
metric_name = "string('sensors')"
timestamp = "/Gateway/Timestamp"
timestamp_format = "2006-01-02T15:04:05Z"
field_selection = "child::Variable"
field_name = "name(@*[1])"
field_value = "number(@*[1])"
[inputs.file.xpath.tags]
name = "substring-after(@name, ' ')"
```
Output:
```text
sensors,host=Hugin,name=Facility\ A consumers=3,frequency=49.78,power=123.4,temperature=20 1596294243000000000
sensors,host=Hugin,name=Facility\ B consumers=1,frequency=49.78,power=14.3,temperature=23.1 1596294243000000000
sensors,host=Hugin,name=Facility\ C consumers=0,frequency=49.78,power=0.02,temperature=19.7 1596294243000000000
```
Using the `metric_selection` option we select all `Sensor` nodes in the XML
document. For each _Sensor_ we then use `field_selection` to select all child
nodes of the sensor as _field-nodes_ Please note that the field selection is
relative to the selected nodes. For each selected _field-node_ we use
`field_name` and `field_value` to determining the field's name and value,
respectively. The `field_name` derives the name of the first attribute of the
node, while `field_value` derives the value of the first attribute and converts
the result to a number.
[xpath lib]: https://github.com/antchfx/xpath
[json]: https://www.json.org/
[msgpack]: https://msgpack.org/
[protobuf]: https://developers.google.com/protocol-buffers
[xml]: https://www.w3.org/XML/
[xpath]: https://www.w3.org/TR/xpath/
[xpather]: http://xpather.com/
[xpath tester]: https://codebeautify.org/Xpath-Tester
[time const]: https://golang.org/pkg/time/#pkg-constants
[time parse]: https://golang.org/pkg/time/#Parse

2
content/telegraf/v1.27/data_formats/output/_index.md
View File

@ -1,9 +1,9 @@
---
title: Telegraf output data formats
list_title: Output data formats
description: Telegraf serializes metrics into output data formats.
menu:
telegraf_1_27_ref:
name: Output data formats
weight: 1
parent: Data formats

6
content/telegraf/v1.27/data_formats/output/carbon2.md
View File

@ -1,15 +1,15 @@
---
title: Carbon2 output data format
description: Use the Carbon2 output data format (serializer) to convert Telegraf metrics into the Carbon2 format.
list_title: Carbon2
description: Use the `carbon2` output data format (serializer) to format and output Telegraf metrics as Carbon2 format.
menu:
telegraf_1_27_ref:
name: Carbon2
weight: 10
parent: Output data formats
---
The `carbon2` output data format (serializer) translates the Telegraf metric format to the [Carbon2 format](http://metrics20.org/implementations/).
Use the `carbon2` output data format (serializer) to format and output Telegraf metrics as [Carbon2 format](http://metrics20.org/implementations/).
### Configuration

21
content/telegraf/v1.27/data_formats/output/graphite.md
View File

@ -1,18 +1,19 @@
---
title: Graphite output data format
description: Use the Graphite output data format to serialize data from Telegraf metrics.
list_title: Graphite
description: Use the `graphite` output data format (serializer) to format and output Telegraf metrics as Graphite Message Format.
menu:
telegraf_1_27_ref:
name: Graphite output
weight: 20
name: Graphite
weight: 10
parent: Output data formats
identifier: output-data-format-graphite
---
The Graphite data format is serialized from Telegraf metrics using either the
template pattern or tag support method. You can select between the two
methods using the [`graphite_tag_support`](#graphite_tag_support) option. When set, the tag support method is used,
otherwise the [template pattern][templates]) option is used.
Use the `graphite` output data format (serializer) to format and output Telegraf metrics as [Graphite Message Format](https://graphite.readthedocs.io/en/latest/feeding-carbon.html#step-3-understanding-the-graphite-message-format).
The serializer uses either the _template pattern_ method (_default_) or the _tag support_ method.
To use the tag support method, set the [`graphite_tag_support`](#graphite_tag_support) option.
## Configuration
@ -54,6 +55,6 @@ cpu.usage_user;cpu=cpu-total;dc=us-east-1;host=tars 0.89 1455320690
cpu.usage_idle;cpu=cpu-total;dc=us-east-1;host=tars 98.09 1455320690
```
### templates
### Templates
For more information on templates and template patterns, see [Template patterns](/telegraf/v1.27/data_formats/input/graphite/#templates).
To learn more about using templates and template patterns, see [Template patterns](/telegraf/v1.27/configure_plugins/template-patterns/).

14
content/telegraf/v1.27/data_formats/output/influx.md
View File

@ -1,15 +1,17 @@
---
title: InfluxDB Line Protocol output data format
description: The `influx` data format outputs metrics into the InfluxDB Line Protocol format.
title: InfluxDB line protocol output data format
list_title: InfluxDB line protocol
description: Use the `influx` output data format (serializer) to format and output metrics as InfluxDB line protocol format.
menu:
telegraf_1_27_ref:
name: InfluxDB Line Protocol
weight: 30
name: InfluxDB line protocol
weight: 10
parent: Output data formats
identifier: output-data-format-influx
---
The `influx` output data format outputs metrics into [InfluxDB Line Protocol][line protocol]. InfluxData recommends this data format unless another format is required for interoperability.
Use the `influx` output data format (serializer) to format and output metrics as [InfluxDB line protocol][line protocol].
InfluxData recommends this data format unless another format is required for interoperability.
## Configuration

9
content/telegraf/v1.27/data_formats/output/json.md
View File

@ -1,15 +1,16 @@
---
title: JSON output data format
description: Telegraf's `json` output data format converts metrics into JSON documents.
list_title: JSON
description: Use the `json` output data format (serializer) to format and output Telegraf metrics as JSON documents.
menu:
telegraf_1_27_ref:
name: JSON
weight: 40
weight: 10
parent: Output data formats
identifier: output-data-format-json
---
The `json` output data format serializes Telegraf metrics into JSON documents.
Use the `json` output data format (serializer) to format and output Telegraf metrics as [JSON](https://www.json.org/json-en.html) documents.
## Configuration

4
content/telegraf/v1.27/data_formats/output/messagepack.md
View File

@ -1,9 +1,9 @@
---
title: MessagePack output data format
description: Use the MessagePack output data format (serializer) to convert Telegraf metrics into MessagePack format.
list_title: MessagePack
description: Use the `msgpack` output data format (serializer) to convert Telegraf metrics into MessagePack format.
menu:
telegraf_1_27_ref:
name: MessagePack
weight: 10
parent: Output data formats

12
content/telegraf/v1.27/data_formats/output/nowmetric.md
View File

@ -1,15 +1,15 @@
---
title: ServiceNow Metrics output data format
description: Use the ServiceNow Metrics output data format (serializer) to output metrics in the ServiceNow Operational Intelligence format.
title: ServiceNow metrics output data format
list_title: ServiceNow metrics
description: Use the `nowmetric` ServiceNow metrics output data format (serializer) to output Telegraf metrics as ServiceNow Operational Intelligence format.
menu:
telegraf_1_27_ref:
name: ServiceNow Metrics
weight: 50
name: ServiceNow metrics
weight: 10
parent: Output data formats
---
The ServiceNow Metrics output data format (serializer) outputs metrics in the [ServiceNow Operational Intelligence format](https://docs.servicenow.com/bundle/kingston-it-operations-management/page/product/event-management/reference/mid-POST-metrics.html).
The `nowmetric` output data format (serializer) outputs Telegraf metrics as [ServiceNow Operational Intelligence format](https://docs.servicenow.com/bundle/kingston-it-operations-management/page/product/event-management/reference/mid-POST-metrics.html).
It can be used to write to a file using the File output plugin, or for sending metrics to a MID Server with Enable REST endpoint activated using the standard telegraf HTTP output.
If you're using the HTTP output plugin, this serializer knows how to batch the metrics so you don't end up with an HTTP POST per metric.

15
content/telegraf/v1.27/data_formats/output/splunkmetric.md
View File

@ -1,16 +1,17 @@
---
title: SplunkMetric output data format
description: The SplunkMetric serializer formats and outputs data in a format that can be consumed by a Splunk metrics index.
title: Splunk metrics output data format
list_title: Splunk metrics
description: Use the `splunkmetric` metric output data format (serializer) to output Telegraf metrics in a format that can be consumed by a Splunk metrics index.
menu:
telegraf_1_27_ref:
name: SplunkMetric
weight: 60
name: Splunk metric
weight: 10
parent: Output data formats
---
The SplunkMetric serializer formats and outputs the metric data in a format that can be consumed by a Splunk metrics index.
It can be used to write to a file using the file output, or for sending metrics to a HEC using the standard Telegraf HTTP output.
Use the `splunkmetric` output data format (serializer) to output Telegraf metrics in a format that can be consumed by a Splunk metrics index.
The output data format can write to a file using the file output, or send metrics to a HEC using the standard Telegraf HTTP output.
If you're using the HTTP output, this serializer knows how to batch the metrics so you don't end up with an HTTP POST per metric.

1
content/telegraf/v1.27/glossary.md
View File

@ -3,7 +3,6 @@ title: Telegraf glossary
description: This section includes definitions of important terms for related to Telegraf.
menu:
telegraf_1_27_ref:
name: Glossary
weight: 79
---

18
content/telegraf/v1.27/metrics.md
View File

@ -2,15 +2,15 @@
title: Telegraf metrics
description: Telegraf metrics are internal representations used to model data during processing and are based on InfluxDB's data model. Each metric component includes the measurement name, tags, fields, and timestamp.
menu:
telegraf_1_27:
telegraf_1_27_ref:
name: Metrics
weight: 10
parent: Concepts
draft: true
---
Telegraf metrics are the internal representation used to model data during
processing. These metrics are closely based on InfluxDB's data model and contain
processing.
These metrics are closely based on InfluxDB's data model and contain
four main components:
- **Measurement name**: Description and namespace for the metric.
@ -20,10 +20,12 @@ four main components:
metric data.
- **Timestamp**: Date and time associated with the fields.
This metric type exists only in memory and must be converted to a concrete
representation in order to be transmitted or viewed. Telegraf provides [output data formats][output data formats] (also known as *serializers*) for these conversions. Telegraf's default serializer converts to [InfluxDB Line
Protocol][line protocol], which provides a high performance and one-to-one
Telegraf metrics exist only in memory and must be converted to a concrete
representation to be transmitted or viewed.
Telegraf provides [output data formats][output data formats] (also known as *serializers*) for these conversions.
Telegraf's default serializer converts to [InfluxDB line
protocol][line protocol], which provides a high performance and one-to-one
direct mapping from Telegraf metrics.
[output data formats]: /telegraf/v1.27/data_formats/output/
[line protocol]: /telegraf/v1.27/data_formats/output/influx/
[output data formats]: /{{% latest "telegraf" %}}/data_formats/output/
[line protocol]: /{{% latest "telegraf" %}}/data_formats/output/influx/