From 3d8b49503697fad83fe79c680e293ea623036ef6 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Tue, 19 May 2020 17:08:57 -0600 Subject: [PATCH 1/6] WIP write annotated CSV --- content/v2.0/reference/glossary.md | 27 ++++- content/v2.0/write-data/csv.md | 183 +++++++++++++++++++++++++++++ 2 files changed, 206 insertions(+), 4 deletions(-) create mode 100644 content/v2.0/write-data/csv.md diff --git a/content/v2.0/reference/glossary.md b/content/v2.0/reference/glossary.md index 0c9a97443..daee14047 100644 --- a/content/v2.0/reference/glossary.md +++ b/content/v2.0/reference/glossary.md @@ -90,6 +90,8 @@ Related entries: [implicit block](#implicit-block), [explicit block](#explicit-b A data type with two possible values: true or false. By convention, you can express `true` as the integer `1` and false as the integer `0` (zero). +In [annotated CSV](/v2.0/reference/syntax/annotated-csv/), columns that contain +boolean values are annotated with the `boolean` datatype. ### bucket @@ -337,8 +339,10 @@ Related entries: [block](#block) ### float -A float represents real numbers and is written with a decimal point dividing the integer and fractional parts. -For example, 1.0, 3.14. +A real number written with a decimal point dividing the integer and fractional parts (`1.0`, `3.14`, `-20.1`). +InfluxDB supports 64-bit float values. +In [annotated CSV](/v2.0/reference/syntax/annotated-csv/), columns that contain +float values are annotated with the `double` datatype. ### flush interval @@ -466,9 +470,14 @@ Related entries: [aggregator plugin](#aggregator-plugin), [collection interval]( An entity comprising data on a server (or virtual server in cloud computing). -### int (data type) +### integer -A data type that represents an integer, a whole number that's positive, negative, or zero. +A whole number that is positive, negative, or zero (`0`, `-5`, `143`). +InfluxDB supports 64-bit integers (minimum: `-9223372036854775808`, maximum: `9223372036854775807`). +In [annotated CSV](/v2.0/reference/syntax/annotated-csv/), columns that contain +integers are annotated with the `long` datatype. + +Related entries: [unsigned integer](#unsigned-integer) ## J @@ -945,6 +954,8 @@ A stream includes a series of tables over a sequence of time intervals. ### string A data type used to represent text. +In [annotated CSV](/v2.0/reference/syntax/annotated-csv/), columns that contain +string values are annotated with the `string` datatype. ## T @@ -1094,6 +1105,14 @@ InfluxDB supports the following unix timestamp precisions: Related entries: [timestamp](#timestamp), [RFC3339 timestamp](#rfc3339-timestamp) +### unsigned integer +A whole number that is positive or zero (`0`, `143`). Also known as a "uinteger." +InfluxDB supports 64-bit unsigned integers (minimum: `0`, maximum: `18446744073709551615`). +In [annotated CSV](/v2.0/reference/syntax/annotated-csv/), columns that contain +integers are annotated with the `unisgnedLong` datatype. + +Related entries: [integer](#integer) + ### user InfluxDB users are granted permission to access to InfluxDB. diff --git a/content/v2.0/write-data/csv.md b/content/v2.0/write-data/csv.md new file mode 100644 index 000000000..b2a3d65a1 --- /dev/null +++ b/content/v2.0/write-data/csv.md @@ -0,0 +1,183 @@ +--- +title: Write CSV data to InfluxDB +description: > + Use the [`influx write` command](/v2.0/reference/cli/influx/write/) to write CSV data + to InfluxDB. Include annotations with the CSV data determine how the data translates + into [line protocol](/v2.0/reference/syntax/line-protocol/). +menu: + v2_0: + name: Write CSV data + parent: Write data +weight: 104 +related: + - /v2.0/reference/syntax/line-protocol/ + - /v2.0/reference/syntax/annotated-csv/ + - /v2.0/reference/cli/influx/write/ +--- + +Use the [`influx write` command](/v2.0/reference/cli/influx/write/) to write CSV data +to InfluxDB. Include annotations with the CSV data to specify how the data translates +into [line protocol](/v2.0/reference/syntax/line-protocol/). + +InfluxDB requires the following for each point written: + +- measurement +- field set +- timestamp +- (optional) tag set + +## CSV Annotations + +{{% note %}} +When writing CSV data to InfluxDB, the `influx write` command supports all annotations +supported and returned by [Flux Annotated CSV](/v2.0/reference/syntax/annotated-csv/). +Additional annotation options described in this article are specific to the +`influx write` command and **cannot** be read directly with the +[Flux `csv.from()` function](/v2.0/reference/flux/stdlib/csv/from/). +{{% /note %}} + +- [datatype](#datatype) +- [constant](#constant) + +### datatype +Use the `#datatype` annotation to specify which [line protocol element](/v2.0/reference/syntax/line-protocol/#elements-of-line-protocol) +each column represents. +To explicitly define a column as a **field** of a specific data type, use the field +type in the annotation. + +| Data type | Resulting line protocol | +|:---------- |:----------------------- | +| [measurement](#measurement) | Column is the **measurement** | +| [tag](#tag) | Column is a **tag** | +| [dateTime](#datetime) | Column is the **timestamp** | +| [field](#field) | Column is a **field** | +| [ignored](#ignored) | Column is ignored | +| [string](#field-types) | Column is a **string field** | +| [double](#field-types) | Column is a **float field** | +| [long](#field-types) | Column is an **integer field** | +| [unsignedLong](#field-types) | Column is an **unsigned integer field** | +| [boolean](#field-types) | Column is a **boolean field** | + +#### measurement +Indicates the column is the **measurement** for the row. + +#### tag +Indicates the column is a **tag**. +The **column label** is the **tag key**. +The **column value** is the **tag value**. + +#### dateTime +Indicates the column is the **timestamp** for the row. +You can also use `time` as an alias for `dateTime`. +By default, all timestamps are UTC. +Use the [`#timezone` annotation](#timezone) to adjust timestamps to a specific timezone. + +{{% note %}} +There can only be **one** `dateTime` column. +{{% /note %}} + +The `influx write` command converts timestamps to [Unix timestamps](/v2.0/reference/glossary/#unix-timestamp). +Append the timestamp format to the `dateTime` datatype with (`:`). + +```csv +#datatype dateTime:RFC3339 +#datatype dateTime:RFC3339Nano +#datatype dateTime:number +#datatype dateTime:2006-01-02 +``` + +##### Supported timestamp formats + +| Timestamp format | Description | Example | +|:---------------- |:----------- |:------- | +| **RFC3339** | RFC3339 timestamp | `2020-01-01T00:00:00Z` | +| **RFC3339Nano** | RFC3339 timestamp | `2020-01-01T00:00:00.000000000Z` | +| **number** | Unix timestamp | `1577836800000000000` | +| **2006-01-02** | YYYY-MM-DD date | `2020-01-01` | + +{{% note %}} +If using the `number` timestamp format and timestamps are **not nanosecond Unix timestamps**, +use the [`--precision` flag](/v2.0/reference/cli/influx/write/#flags) with the +`influx write` command to specify the timestamp precision. +{{% /note %}} + +#### field +Indicates the column is a **field** and auto-detects the field type. +The **column label** is the **field key**. +The **column value** is the **field value**. + +#### ignored +The column is ignored and not written to InfluxDB. + +#### Field types +The column is a **field** of a specified type. +The **column label** is the **field key**. +The **column value** is the **field value**. + +- `string`: column is a **[string](/v2.0/reference/glossary/#string) field**. +- `double`: column is a **[float](/v2.0/reference/glossary/#float) field**. +- `long`: column is an **[integer](/v2.0/reference/glossary/#integer) field**. +- `unsignedLong`: column is an **[unsigned integer](/v2.0/reference/glossary/#unsigned-integer) field**. +- `boolean`: column is a **[boolean](/v2.0/reference/glossary/#boolean) field**. + +### constant +Use the `#constant` annotation to define a constant column label and value for each row. +The `#constant` annotation provides a way to supply elements required to write +to InfluxDB if they don't exist in the CSV data. + +Use the following syntax to define constants: + +``` +#constant ,, +``` + +{{% note %}} +For constants with `measurement` and `dateTime` datatypes, the second value in +the constant definition is the **column-value**. +{{% /note %}} + +##### Example constants +``` +#constant measurement,m +#constant tag,dataSource,csv +``` + +### timezone +Use the `#timezone` annotation to update timestamps to a specific timezone. +By default, timestamps are parsed as UTC. +Use the `±HHMM` format to the offset relative to UTC. + +##### Timezone examples +| Timezone | Offset | +|:-------- | ------: | +| US Mountain Daylight Time | `-0600` | +| Central European Summer Time | `+0200` | +| Australia Eastern Standard Time | `+1000` | + +##### Timezone annotation example +``` +#timezone -0600 +``` + + +## Define custom column separator +``` +sep=; +``` + +--- + + - Annotation shorthand + - include column label, datatype, and default value in one line. + - Set constants + - Custom column separator + +- Write command + - inject annotation headers + - skip headers + +- Example commands + - Write the raw results of a Flux query + - Simple annotated CSV with #datatype annotation + - Annotated CSV with #datatype and CSV annotations + - Include defaults with the #datatype annotation From 1be8f826ca6c4d1a28aa5454c89b63e0568a64b2 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Wed, 20 May 2020 16:00:37 -0600 Subject: [PATCH 2/6] created new extended annotated csv doc, outline for write csv doc --- .../_index.md} | 7 +- .../syntax/annotated-csv/extended.md | 278 ++++++++++++++++++ content/v2.0/write-data/csv.md | 146 +-------- 3 files changed, 283 insertions(+), 148 deletions(-) rename content/v2.0/reference/syntax/{annotated-csv.md => annotated-csv/_index.md} (99%) create mode 100644 content/v2.0/reference/syntax/annotated-csv/extended.md diff --git a/content/v2.0/reference/syntax/annotated-csv.md b/content/v2.0/reference/syntax/annotated-csv/_index.md similarity index 99% rename from content/v2.0/reference/syntax/annotated-csv.md rename to content/v2.0/reference/syntax/annotated-csv/_index.md index 9b1a8db49..0466536cb 100644 --- a/content/v2.0/reference/syntax/annotated-csv.md +++ b/content/v2.0/reference/syntax/annotated-csv/_index.md @@ -1,6 +1,5 @@ --- -title: Annotated CSV syntax -list_title: Annotated CSV +title: Annotated CSV description: > InfluxDB and Flux return query results in annotated CSV format. You can also read annotated CSV directly from Flux with the `csv.from()` function @@ -9,10 +8,12 @@ weight: 103 menu: v2_0_ref: parent: Syntax - name: Annotated CSV v2.0/tags: [csv, syntax] aliases: - /v2.0/reference/annotated-csv/ +related: + - /v2.0/reference/flux/stdlib/csv/from/ + - /v2.0/reference/syntax/annotated-csv/extended/ --- InfluxDB and Flux return query results in annotated CSV format. diff --git a/content/v2.0/reference/syntax/annotated-csv/extended.md b/content/v2.0/reference/syntax/annotated-csv/extended.md new file mode 100644 index 000000000..87a3b74cf --- /dev/null +++ b/content/v2.0/reference/syntax/annotated-csv/extended.md @@ -0,0 +1,278 @@ +--- +title: Extended annotated CSV +description: > + Use the [`influx write` command](/v2.0/reference/cli/influx/write/) to write CSV data + to InfluxDB. Include annotations with the CSV data determine how the data translates + into [line protocol](/v2.0/reference/syntax/line-protocol/). +menu: + v2_0_ref: + name: Extended annotated CSV + parent: Annotated CSV +weight: 201 +v2.0/tags: [csv, syntax, write] +related: + - /v2.0/write-data/csv/ + - /v2.0/reference/cli/influx/write/ + - /v2.0/reference/syntax/line-protocol/ + - /v2.0/reference/syntax/annotated-csv/ +--- + +**Extended annotated CSV** provides additional annotations and options that specify +how CSV data should be converted to [line protocol](/v2.0/reference/syntax/line-protocol/) +and written to InfluxDB. +Extended annotated CSV supports all [Annotated CSV](/v2.0/reference/syntax/annotated-csv/) +annotations. + +{{% warn %}} +The Flux [`csv.from` function](/v2.0/reference/flux/stdlib/csv/from/) only supports +**Annotated CSV**, not **Extended annotated CSV**. +{{% /warn %}} + +To write data to InfluxDB, line protocol must include the following: + +- [measurement](/v2.0/reference/syntax/line-protocol/#measurement) +- [field set](/v2.0/reference/syntax/line-protocol/#field-set) +- [timestamp](/v2.0/reference/syntax/line-protocol/#timestamp) _(Optional but recommended)_ +- [tag set](/v2.0/reference/syntax/line-protocol/#tag-set) _(Optional)_ + +Extended annotated CSV annotations identify which element of line protocol each +column represents and how the data should be converted. + +## CSV Annotations +Extended annotated CSV extends and adds the following annotations: + +- [datatype](#datatype) +- [constant](#constant) +- [timezone](#timezone) + +### datatype +Use the `#datatype` annotation to specify which [line protocol element](/v2.0/reference/syntax/line-protocol/#elements-of-line-protocol) +each column represents. +To explicitly define a column as a **field** of a specific data type, use the field +type in the annotation. + +| Data type | Resulting line protocol | +|:---------- |:----------------------- | +| [measurement](#measurement) | Column is the **measurement** | +| [tag](#tag) | Column is a **tag** | +| [dateTime](#datetime) | Column is the **timestamp** | +| [field](#field) | Column is a **field** | +| [ignored](#ignored) | Column is ignored | +| [string](#string) | Column is a **string field** | +| [double](#double) | Column is a **float field** | +| [long](#long) | Column is an **integer field** | +| [unsignedLong](#unsignedlong) | Column is an **unsigned integer field** | +| [boolean](#boolean) | Column is a **boolean field** | + +#### measurement +Indicates the column is the **measurement**. + +#### tag +Indicates the column is a **tag**. +The **column label** is the **tag key**. +The **column value** is the **tag value**. + +#### dateTime +Indicates the column is the **timestamp**. +`time` is as an alias for `dateTime`. +By default, all timestamps are UTC. +Use the [`#timezone` annotation](#timezone) to adjust timestamps to a specific timezone. + +{{% note %}} +There can only be **one** `dateTime` column. +{{% /note %}} + +The `influx write` command converts timestamps to [Unix timestamps](/v2.0/reference/glossary/#unix-timestamp). +Append the timestamp format to the `dateTime` datatype with (`:`). + +```csv +#datatype dateTime:RFC3339 +#datatype dateTime:RFC3339Nano +#datatype dateTime:number +#datatype dateTime:2006-01-02 +``` + +##### Supported timestamp formats + +| Timestamp format | Description | Example | +|:---------------- |:----------- |:------- | +| **RFC3339** | RFC3339 timestamp | `2020-01-01T00:00:00Z` | +| **RFC3339Nano** | RFC3339 timestamp | `2020-01-01T00:00:00.000000000Z` | +| **number** | Unix timestamp | `1577836800000000000` | +| **2006-01-02** | YYYY-MM-DD date | `2020-01-01` | + +{{% note %}} +If using the `number` timestamp format and timestamps are **not nanosecond Unix timestamps**, +use the [`--precision` flag](/v2.0/reference/cli/influx/write/#flags) with the +`influx write` command to specify the timestamp precision. +{{% /note %}} + +#### field +Indicates the column is a **field** and auto-detects the field type. +The **column label** is the **field key**. +The **column value** is the **field value**. + +#### ignored +The column is ignored and not written to InfluxDB. + +#### Field types +The column is a **field** of a specified type. +The **column label** is the **field key**. +The **column value** is the **field value**. + +##### string +Column is a **[string](/v2.0/reference/glossary/#string) field**. + +##### double +Column is a **[float](/v2.0/reference/glossary/#float) field**. +By default, InfluxDB expects float values that use a period (`.`) to separate the +fraction from the whole number. +If column values include or use other separators, such as commas (`,`) to visually +separate large numbers into groups, specify the following **float separators**: + +- **fraction separator**: Separates the fraction from the whole number +- **ignored separator**: Visually separates the whole number into groups but should + be ignored when parsing the float value. + +Use the following syntax to specify **float separators**: + +```sh +# Syntax + + +# Example +., + +# With the float separators above +# 1,200,000.15 => 1200000.15 +``` + +Append **float separators** to the `double` datatype annotation with a colon (`:`). +For example: + +``` +#datatype "fieldName|double:.," +``` + +{{% note %}} +If your **float separators** include a comma (`,`), wrap the column annotation in double +quotes (`""`) to prevent the comma from being parsed as column separator or delimitter. +You can also [define a custom column separator](#define-custom-column-separator). +{{% /note %}} + +##### long +Column is an **[integer](/v2.0/reference/glossary/#integer) field**. + +##### unsignedLong +Column is an **[unsigned integer](/v2.0/reference/glossary/#unsigned-integer) field**. + +##### boolean +Column is a **[boolean](/v2.0/reference/glossary/#boolean) field**. +If column values are not [supported boolean values](/v2.0/reference/syntax/line-protocol/#boolean), +specify the **boolean format** with the following syntax: + +```sh +# Syntax +: + +# Example +y,Y:n,N + +# With the boolean format above +# y => true, Y => true, n => false, N => false +``` + +Append the **boolean format** to the `boolean` datatype annotation with a colon (`:`). +For example: + +``` +#datatype "fieldName|boolean:y,Y:n,N" +``` + +{{% note %}} +If your **boolean format** contains commas (`,`), wrap the column annotation in double +quotes (`""`) to prevent the commas from being parsed as column separators or delimiters. +You can also [define a custom column separator](#define-custom-column-separator). +{{% /note %}} + +### constant +Use the `#constant` annotation to define a constant column label and value for each row. +The `#constant` annotation provides a way to supply +[line protocol elements](/v2.0/reference/syntax/line-protocol/#elements-of-line-protocol) +that don't exist in the CSV data. + +Use the following syntax to define constants: + +``` +#constant ,, +``` + +To provide multiple constants, include each `#constant` annotations on a separate line. + +``` +#constant measurement,m +#constant tag,dataSource,csv +``` + +{{% note %}} +For constants with `measurement` and `dateTime` datatypes, the second value in +the constant definition is the **column-value**. +{{% /note %}} + +### timezone +Use the `#timezone` annotation to update timestamps to a specific timezone. +By default, timestamps are parsed as UTC. +Use the `±HHmm` format to specify the timezone offset relative to UTC. + +##### Timezone examples +| Timezone | Offset | +|:-------- | ------: | +| US Mountain Daylight Time | `-0600` | +| Central European Summer Time | `+0200` | +| Australia Eastern Standard Time | `+1000` | +| Apia Daylight Time | `+1400` | + +##### Timezone annotation example +``` +#timezone -0600 +``` + +## Define custom column separator +If columns are delimited using a character other than a comma, use the `sep` +keyword to define a custom separator for your CSV file. + +``` +sep=; +``` + +## Annotation shorthand +Extended annotated CSV supports **annotation shorthand**. +Include the column label, datatype, and _(optional)_ default value in each column +header row using the following syntax: + +``` +|| +``` + +##### Example annotation shorthand +``` +m|measurement|general,location|tag,temp|double,pm|long,time|dateTime:RFC3339 +weather,San Francisco,51.9,38,2020-01-01T00:00:00Z +weather,New York,18.2,15,2020-01-01T00:00:00Z +,Hong Kong,53.6,171,2020-01-01T00:00:00Z +``` + +##### The shorthand explained +- The `m` column represents the **measurement** and has a default value of `general`. +- The `location` column is a **tag** with no default value. +- The `temp` column is a **field** with **float** (`double`) values and no default value. +- The `pm` column is a **field** with **integer** (`long`) values and no default value. +- The `time` column represents the **timestamp**, uses the **RFC3339** timestamp format, + and has no default value. + +##### Resulting line protocol +``` +weather,location=San\ Francisco temp=51.9,pm=38i 1577836800000000000 +weather,location=New\ York temp=18.2,pm=18i 1577836800000000000 +general,location=Hong\ Kong temp=53.6,pm=171i 1577836800000000000 +``` diff --git a/content/v2.0/write-data/csv.md b/content/v2.0/write-data/csv.md index b2a3d65a1..60dbadf38 100644 --- a/content/v2.0/write-data/csv.md +++ b/content/v2.0/write-data/csv.md @@ -26,151 +26,7 @@ InfluxDB requires the following for each point written: - timestamp - (optional) tag set -## CSV Annotations - -{{% note %}} -When writing CSV data to InfluxDB, the `influx write` command supports all annotations -supported and returned by [Flux Annotated CSV](/v2.0/reference/syntax/annotated-csv/). -Additional annotation options described in this article are specific to the -`influx write` command and **cannot** be read directly with the -[Flux `csv.from()` function](/v2.0/reference/flux/stdlib/csv/from/). -{{% /note %}} - -- [datatype](#datatype) -- [constant](#constant) - -### datatype -Use the `#datatype` annotation to specify which [line protocol element](/v2.0/reference/syntax/line-protocol/#elements-of-line-protocol) -each column represents. -To explicitly define a column as a **field** of a specific data type, use the field -type in the annotation. - -| Data type | Resulting line protocol | -|:---------- |:----------------------- | -| [measurement](#measurement) | Column is the **measurement** | -| [tag](#tag) | Column is a **tag** | -| [dateTime](#datetime) | Column is the **timestamp** | -| [field](#field) | Column is a **field** | -| [ignored](#ignored) | Column is ignored | -| [string](#field-types) | Column is a **string field** | -| [double](#field-types) | Column is a **float field** | -| [long](#field-types) | Column is an **integer field** | -| [unsignedLong](#field-types) | Column is an **unsigned integer field** | -| [boolean](#field-types) | Column is a **boolean field** | - -#### measurement -Indicates the column is the **measurement** for the row. - -#### tag -Indicates the column is a **tag**. -The **column label** is the **tag key**. -The **column value** is the **tag value**. - -#### dateTime -Indicates the column is the **timestamp** for the row. -You can also use `time` as an alias for `dateTime`. -By default, all timestamps are UTC. -Use the [`#timezone` annotation](#timezone) to adjust timestamps to a specific timezone. - -{{% note %}} -There can only be **one** `dateTime` column. -{{% /note %}} - -The `influx write` command converts timestamps to [Unix timestamps](/v2.0/reference/glossary/#unix-timestamp). -Append the timestamp format to the `dateTime` datatype with (`:`). - -```csv -#datatype dateTime:RFC3339 -#datatype dateTime:RFC3339Nano -#datatype dateTime:number -#datatype dateTime:2006-01-02 -``` - -##### Supported timestamp formats - -| Timestamp format | Description | Example | -|:---------------- |:----------- |:------- | -| **RFC3339** | RFC3339 timestamp | `2020-01-01T00:00:00Z` | -| **RFC3339Nano** | RFC3339 timestamp | `2020-01-01T00:00:00.000000000Z` | -| **number** | Unix timestamp | `1577836800000000000` | -| **2006-01-02** | YYYY-MM-DD date | `2020-01-01` | - -{{% note %}} -If using the `number` timestamp format and timestamps are **not nanosecond Unix timestamps**, -use the [`--precision` flag](/v2.0/reference/cli/influx/write/#flags) with the -`influx write` command to specify the timestamp precision. -{{% /note %}} - -#### field -Indicates the column is a **field** and auto-detects the field type. -The **column label** is the **field key**. -The **column value** is the **field value**. - -#### ignored -The column is ignored and not written to InfluxDB. - -#### Field types -The column is a **field** of a specified type. -The **column label** is the **field key**. -The **column value** is the **field value**. - -- `string`: column is a **[string](/v2.0/reference/glossary/#string) field**. -- `double`: column is a **[float](/v2.0/reference/glossary/#float) field**. -- `long`: column is an **[integer](/v2.0/reference/glossary/#integer) field**. -- `unsignedLong`: column is an **[unsigned integer](/v2.0/reference/glossary/#unsigned-integer) field**. -- `boolean`: column is a **[boolean](/v2.0/reference/glossary/#boolean) field**. - -### constant -Use the `#constant` annotation to define a constant column label and value for each row. -The `#constant` annotation provides a way to supply elements required to write -to InfluxDB if they don't exist in the CSV data. - -Use the following syntax to define constants: - -``` -#constant ,, -``` - -{{% note %}} -For constants with `measurement` and `dateTime` datatypes, the second value in -the constant definition is the **column-value**. -{{% /note %}} - -##### Example constants -``` -#constant measurement,m -#constant tag,dataSource,csv -``` - -### timezone -Use the `#timezone` annotation to update timestamps to a specific timezone. -By default, timestamps are parsed as UTC. -Use the `±HHMM` format to the offset relative to UTC. - -##### Timezone examples -| Timezone | Offset | -|:-------- | ------: | -| US Mountain Daylight Time | `-0600` | -| Central European Summer Time | `+0200` | -| Australia Eastern Standard Time | `+1000` | - -##### Timezone annotation example -``` -#timezone -0600 -``` - - -## Define custom column separator -``` -sep=; -``` - ---- - - - Annotation shorthand - - include column label, datatype, and default value in one line. - - Set constants - - Custom column separator +- Extended annotated CSV - Write command - inject annotation headers From 4e82a5f551e8bfb92490c85e39bc6ff2f7132719 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Tue, 26 May 2020 14:52:08 -0600 Subject: [PATCH 3/6] finalized procedural docs for writing with csv --- .../syntax/annotated-csv/extended.md | 92 +++- content/v2.0/write-data/csv.md | 438 +++++++++++++++++- 2 files changed, 508 insertions(+), 22 deletions(-) diff --git a/content/v2.0/reference/syntax/annotated-csv/extended.md b/content/v2.0/reference/syntax/annotated-csv/extended.md index 87a3b74cf..489b889c0 100644 --- a/content/v2.0/reference/syntax/annotated-csv/extended.md +++ b/content/v2.0/reference/syntax/annotated-csv/extended.md @@ -75,8 +75,11 @@ The **column value** is the **tag value**. #### dateTime Indicates the column is the **timestamp**. `time` is as an alias for `dateTime`. +If the [timestamp format](#supported-timestamp-formats) includes a time zone, +the parsed timestamp respects the time zone. By default, all timestamps are UTC. -Use the [`#timezone` annotation](#timezone) to adjust timestamps to a specific timezone. +You can also use the [`#timezone` annotation](#timezone) to adjust timestamps to +a specific time zone. {{% note %}} There can only be **one** `dateTime` column. @@ -99,7 +102,6 @@ Append the timestamp format to the `dateTime` datatype with (`:`). | **RFC3339** | RFC3339 timestamp | `2020-01-01T00:00:00Z` | | **RFC3339Nano** | RFC3339 timestamp | `2020-01-01T00:00:00.000000000Z` | | **number** | Unix timestamp | `1577836800000000000` | -| **2006-01-02** | YYYY-MM-DD date | `2020-01-01` | {{% note %}} If using the `number` timestamp format and timestamps are **not nanosecond Unix timestamps**, @@ -107,6 +109,11 @@ use the [`--precision` flag](/v2.0/reference/cli/influx/write/#flags) with the `influx write` command to specify the timestamp precision. {{% /note %}} +##### Custom timestamp formats +To specify a custom timestamp format, use timestamp formats as described in the +[Go time package](https://golang.org/pkg/time). +For example: `2020-01-01`. + #### field Indicates the column is a **field** and auto-detects the field type. The **column label** is the **field key**. @@ -120,6 +127,12 @@ The column is a **field** of a specified type. The **column label** is the **field key**. The **column value** is the **field value**. +- [string](#string) +- [double](#double) +- [long](#long) +- [unsignedLong](#unsignedlong) +- [boolean](#boolean) + ##### string Column is a **[string](/v2.0/reference/glossary/#string) field**. @@ -156,15 +169,81 @@ For example: {{% note %}} If your **float separators** include a comma (`,`), wrap the column annotation in double -quotes (`""`) to prevent the comma from being parsed as column separator or delimitter. +quotes (`""`) to prevent the comma from being parsed as column separator or delimiter. You can also [define a custom column separator](#define-custom-column-separator). {{% /note %}} ##### long Column is an **[integer](/v2.0/reference/glossary/#integer) field**. +If column values contain separators such as periods (`.`) or commas (`,`), specify +the following **integer separators**: + +- **fraction separator**: Separates the fraction from the whole number. + _**Integer values are truncated at the fraction separator when converted to line protocol.**_ +- **ignored separator**: Visually separates the whole number into groups but should + be ignored when parsing the integer value. + +Use the following syntax to specify **integer separators**: + +```sh +# Syntax + + +# Example +., + +# With the integer separators above +# 1,200,000.00 => 1200000i +``` + +Append **integer separators** to the `long` datatype annotation with a colon (`:`). +For example: + +``` +#datatype "fieldName|long:.," +``` + +{{% note %}} +If your **integer separators** include a comma (`,`), wrap the column annotation in double +quotes (`""`) to prevent the comma from being parsed as column separator or delimiter. +You can also [define a custom column separator](#define-custom-column-separator). +{{% /note %}} ##### unsignedLong -Column is an **[unsigned integer](/v2.0/reference/glossary/#unsigned-integer) field**. +Column is an **[unsigned integer (uinteger)](/v2.0/reference/glossary/#unsigned-integer) field**. +If column values contain separators such as periods (`.`) or commas (`,`), specify +the following **uinteger separators**: + +- **fraction separator**: Separates the fraction from the whole number. + _**Uinteger values are truncated at the fraction separator when converted to line protocol.**_ +- **ignored separator**: Visually separates the whole number into groups but should + be ignored when parsing the uinteger value. + +Use the following syntax to specify **uinteger separators**: + +```sh +# Syntax + + +# Example +., + +# With the uinteger separators above +# 1,200,000.00 => 1200000u +``` + +Append **uinteger separators** to the `long` datatype annotation with a colon (`:`). +For example: + +``` +#datatype "fieldName|usignedLong:.," +``` + +{{% note %}} +If your **uinteger separators** include a comma (`,`), wrap the column annotation in double +quotes (`""`) to prevent the comma from being parsed as column separator or delimiter. +You can also [define a custom column separator](#define-custom-column-separator). +{{% /note %}} ##### boolean Column is a **[boolean](/v2.0/reference/glossary/#boolean) field**. @@ -176,10 +255,11 @@ specify the **boolean format** with the following syntax: : # Example -y,Y:n,N +y,Y,1:n,N,0 # With the boolean format above -# y => true, Y => true, n => false, N => false +# y => true, Y => true, 1 => true +# n => false, N => false, 0 => false ``` Append the **boolean format** to the `boolean` datatype annotation with a colon (`:`). diff --git a/content/v2.0/write-data/csv.md b/content/v2.0/write-data/csv.md index 60dbadf38..55817263d 100644 --- a/content/v2.0/write-data/csv.md +++ b/content/v2.0/write-data/csv.md @@ -16,24 +16,430 @@ related: --- Use the [`influx write` command](/v2.0/reference/cli/influx/write/) to write CSV data -to InfluxDB. Include annotations with the CSV data to specify how the data translates -into [line protocol](/v2.0/reference/syntax/line-protocol/). +to InfluxDB. Include [Extended annotated CSV](/v2.0/reference/syntax/annotated-csv/extended/) +annotations to specify how the data translates into [line protocol](/v2.0/reference/syntax/line-protocol/). +Include annotations in the CSV file or inject them using the `--header` flag of +the `influx write` command. -InfluxDB requires the following for each point written: +##### On this page +- [CSV Annotations](#csv-annotations) +- [Inject annotation headers](#inject-annotation-headers) +- [Skip annotation headers](#skip-annotation-headers) +- [Process input as CSV](#process-input-as-csv) +- [Specify CSV character encoding](#specify-csv-character-encoding) +- [Skip rows with errors](#skip-rows-with-errors) +- [Advanced examples](#advanced-examples) -- measurement -- field set -- timestamp -- (optional) tag set +##### Example write command +```sh +influx write -b example-bucket -f path/to/example.csv +``` -- Extended annotated CSV +##### example.csv +``` +#datatype measurement,tag,float,dateTime:RFC3339 +m,host,used_percent,time +mem,host1,64.23,2020-01-01T00:00:00Z +mem,host2,72.01,2020-01-01T00:00:00Z +mem,host1,62.61,2020-01-01T00:00:10Z +mem,host2,72.98,2020-01-01T00:00:10Z +mem,host1,63.40,2020-01-01T00:00:20Z +mem,host2,73.77,2020-01-01T00:00:20Z +``` -- Write command - - inject annotation headers - - skip headers +##### Resulting line protocol +``` +mem,host=host1 used_percent=64.23 1577836800000000000 +mem,host=host2 used_percent=72.01 1577836800000000000 +mem,host=host1 used_percent=62.61 1577836810000000000 +mem,host=host2 used_percent=72.98 1577836810000000000 +mem,host=host1 used_percent=63.40 1577836820000000000 +mem,host=host2 used_percent=73.77 1577836820000000000 +``` -- Example commands - - Write the raw results of a Flux query - - Simple annotated CSV with #datatype annotation - - Annotated CSV with #datatype and CSV annotations - - Include defaults with the #datatype annotation +{{% note %}} +To test the CSV to line protocol conversion process, include the `--dryrun` flag +with the `influx write` command to print the resulting line protocol to stdout +rather than write to InfluxDB. +{{% /note %}} + +## CSV Annotations +Use **CSV annotations** to specify which element of line protocol each CSV column +represents and how to format the data. CSV annotations are rows at the beginning +of a CSV file that describe column properties. + +The `influx write` command supports [Extended annotated CSV](/v2.0/reference/syntax/annotated-csv/extended) +which provides options for specifying how CSV data should be converted into line +protocol and how data is formatted. + +To write data to InfluxDB, data must include the following: + +- [measurement](/v2.0/reference/syntax/line-protocol/#measurement) +- [field set](/v2.0/reference/syntax/line-protocol/#field-set) +- [timestamp](/v2.0/reference/syntax/line-protocol/#timestamp) _(Optional but recommended)_ +- [tag set](/v2.0/reference/syntax/line-protocol/#tag-set) _(Optional)_ + +Use CSV annotations to specify which of these elements each column represents. + +## Write raw query results back to InfluxDB +Flux returns query results in [Annotated CSV](/v2.0/reference/syntax/annotated-csv/). +These results include all annotations necessary to write the data back to InfluxDB. + +## Inject annotation headers +If the CSV data you want to write to InfluxDB does not contain the annotations +required to properly convert the data to line protocol, use the `--header` flag +to inject annotation rows into the CSV data. + +```sh +influx write -b example-bucket \ + -f path/to/example.csv \ + --header "#constant measurement,birds" \ + --header "#datatype dataTime:2006-01-02,long,tag" +``` + +{{< flex >}} +{{% flex-content %}} +##### example.csv +``` +date,sighted,loc +2020-01-01,12,Boise +2020-06-01,78,Boise +2020-01-01,54,Seattle +2020-06-01,112,Seattle +2020-01-01,9,Detroit +2020-06-01,135,Detroit +``` +{{% /flex-content %}} +{{% flex-content %}} +##### Resulting line protocol +``` +birds,loc=Boise sighted=12 1577836800000000000 +birds,loc=Boise sighted=78 1590969600000000000 +birds,loc=Seattle sighted=54 1577836800000000000 +birds,loc=Seattle sighted=112 1590969600000000000 +birds,loc=Detroit sighted=9 1577836800000000000 +birds,loc=Detroit sighted=135 1590969600000000000 +``` +{{% /flex-content %}} +{{< /flex >}} + +## Skip annotation headers +Some CSV data may include annotations that conflict with annotations necessary to +write CSV data to InfluxDB. +Use the `--skipHeader` flag to specify the **number of rows to skip** at the +beginning of the CSV data. + +```sh +influx write -b example-bucket \ + -f path/to/example.csv \ + --skipHeader=2 +``` + +## Process input as CSV +The `influx write` command automatically processes files with the `.csv` extension as CSV files. +If your CSV file uses a different extension, use the `--format` flat to explicitly +declare the format of the input file. + +```sh +influx write -b example-bucket \ + -f path/to/example.txt \ + --format csv +``` + +{{% note %}} +The `influx write` command assumes all input files are line protocol unless they +include the `.csv` extension or you declare the `csv`. +{{% /note %}} + +## Specify CSV character encoding +The `influx write` command assumes CSV files contain UTF-8 encoded characters. +If your CSV data uses different character encoding, specify the encoding +with the `--encoding`. + +```sh +influx write -b example-bucket \ + -f path/to/example.csv \ + --encoding "UTF-16" +``` + +## Skip rows with errors +If a row in your CSV data is missing an +[element required to write to InfluxDB](/v2.0/reference/syntax/line-protocol/#elements-of-line-protocol) +or data is incorrectly formatted, when processing the row, the `influx write` command +returns an error and cancels the write request. +To skip rows with errors, use the `--skipRowOnError` flag. + +```sh +influx write -b example-bucket \ + -f path/to/example.csv \ + --skipRowOnError +``` + +{{% warn %}} +Skipped rows are ignored and are not written to InfluxDB. +{{% /warn %}} + +## Advanced examples + +- [Define constants](#define-constants) +- [Annotation shorthand](#annotation-shorthand) +- [Use alternate numeric formats](#use-alternate-numeric-formats) +- [Use alternate boolean format](#use-alternate-boolean-format) +- [Use different timestamp formats](#use-different-timestamp-formats) + +--- + +### Define constants +Use the Extended annotated CSV [`#constant` annotation](/v2.0/reference/syntax/annotated-csv/extended/#constant) +to add a column and value to each row in the CSV data. + +{{< flex >}} +{{% flex-content %}} +##### CSV with constants +``` +#constant measurement,example +#constant tag,source,csv +#datatype long,dateTime:RFC3339 +count,time +1,2020-01-01T00:00:00Z +4,2020-01-02T00:00:00Z +9,2020-01-03T00:00:00Z +18,2020-01-04T00:00:00Z +``` +{{% /flex-content %}} +{{% flex-content %}} +##### Resulting line protocol +``` +example,source=csv count=1 1577836800000000000 +example,source=csv count=4 1577923200000000000 +example,source=csv count=9 1578009600000000000 +example,source=csv count=18 1578096000000000000 +``` +{{% /flex-content %}} +{{< /flex >}} + +--- + +### Annotation shorthand +Extended annotated CSV supports [annotation shorthand](/v2.0/reference/syntax/annotated-csv/extended/#annotation-shorthand), +which lets you define the **column label**, **datatype**, and **default value** in the column header. + +{{< flex >}} +{{% flex-content %}} +##### CSV with annotation shorthand +``` +m|measurement,count|long|0,time|dateTime:RFC3339 +example,1,2020-01-01T00:00:00Z +example,4,2020-01-02T00:00:00Z +example,,2020-01-03T00:00:00Z +example,18,2020-01-04T00:00:00Z +``` +{{% /flex-content %}} +{{% flex-content %}} +##### Resulting line protocol +``` +example count=1 1577836800000000000 +example count=4 1577923200000000000 +example count=0 1578009600000000000 +example count=18 1578096000000000000 +``` +{{% /flex-content %}} +{{< /flex >}} + +#### Replace column header with annotation shorthand +It's possible to replace the column header row in a CSV file with annotation +shorthand without modifying the CSV file. +This lets you define column data types and default values while writing to InfluxDB. + +To replace an existing column header row with annotation shorthand: + +1. Use the `--skipHeader` flag to ignore the existing column header row. +2. Use the `--header` flag to inject a new column header row that uses annotation shorthand. + + +```sh +influx write -b example-bucket \ + -f example.csv \ + --skipHeader=1 + --header="m|measurement,count|long|0,time|dateTime:RFC3339" +``` + +{{< flex >}} +{{% flex-content %}} +##### Unmodified example.csv +``` +m,count,time +example,1,2020-01-01T00:00:00Z +example,4,2020-01-02T00:00:00Z +example,,2020-01-03T00:00:00Z +example,18,2020-01-04T00:00:00Z +``` +{{% /flex-content %}} +{{% flex-content %}} +##### Resulting line protocol +``` +example count=1 1577836800000000000 +example count=4 1577923200000000000 +example count=0 1578009600000000000 +example count=18 1578096000000000000 +``` +{{% /flex-content %}} +{{< /flex >}} + +--- + +### Use alternate numeric formats +If your CSV data contains numeric values that use a non-default fraction separator (`.`) +or contain group separators, [define your numeric format](/v2.0/reference/syntax/annotated-csv/extended/#double) +in the `double`, `long`, and `unsignedLong` datatype annotations. + +{{% note %}} +If your **numeric format separators** include a comma (`,`), wrap the column annotation in double +quotes (`""`) to prevent the comma from being parsed as column separator or delimiter. +You can also [define a custom column separator](##################). +{{% /note %}} + +{{< tabs-wrapper >}} +{{% tabs %}} +[Floats](#) +[Integers](#) +[Uintegers](#) +{{% /tabs %}} +{{% tab-content %}} +{{< flex >}} +{{% flex-content %}} +##### CSV with non-default float values +``` +#datatype measurement,"double:.,",dateTime:RFC3339 +m,lbs,time +example,"1,280.7",2020-01-01T00:00:00Z +example,"1,352.5",2020-01-02T00:00:00Z +example,"1,862.8",2020-01-03T00:00:00Z +example,"2,014.9",2020-01-04T00:00:00Z +``` +{{% /flex-content %}} +{{% flex-content %}} +##### Resulting line protocol +``` +example lbs=1280.7 1577836800000000000 +example lbs=1352.5 1577923200000000000 +example lbs=1862.8 1578009600000000000 +example lbs=2014.9 1578096000000000000 +``` +{{% /flex-content %}} +{{< /flex >}} +{{% /tab-content %}} + +{{% tab-content %}} +{{< flex >}} +{{% flex-content %}} +##### CSV with non-default integer values +``` +#datatype measurement,"long:.,",dateTime:RFC3339 +m,lbs,time +example,"1,280.0",2020-01-01T00:00:00Z +example,"1,352.0",2020-01-02T00:00:00Z +example,"1,862.0",2020-01-03T00:00:00Z +example,"2,014.9",2020-01-04T00:00:00Z +``` +{{% /flex-content %}} +{{% flex-content %}} +##### Resulting line protocol +``` +example lbs=1280i 1577836800000000000 +example lbs=1352i 1577923200000000000 +example lbs=1862i 1578009600000000000 +example lbs=2014i 1578096000000000000 +``` +{{% /flex-content %}} +{{< /flex >}} +{{% /tab-content %}} + +{{% tab-content %}} +{{< flex >}} +{{% flex-content %}} +##### CSV with non-default uinteger values +``` +#datatype measurement,"unsignedLong:.,",dateTime:RFC3339 +m,lbs,time +example,"1,280.0",2020-01-01T00:00:00Z +example,"1,352.0",2020-01-02T00:00:00Z +example,"1,862.0",2020-01-03T00:00:00Z +example,"2,014.9",2020-01-04T00:00:00Z +``` +{{% /flex-content %}} +{{% flex-content %}} +##### Resulting line protocol +``` +example lbs=1280u 1577836800000000000 +example lbs=1352u 1577923200000000000 +example lbs=1862u 1578009600000000000 +example lbs=2014u 1578096000000000000 +``` +{{% /flex-content %}} +{{< /flex >}} +{{% /tab-content %}} +{{< /tabs-wrapper >}} + +--- + +### Use alternate boolean format +Line protocol supports only [specific boolean values](/v2.0/reference/syntax/line-protocol/#boolean). +If your CSV data contains boolean values that line protocol does not support, +[define your boolean format](/v2.0/reference/syntax/annotated-csv/extended/#boolean) +in the `boolean` datatype annotation. + +{{< flex >}} +{{% flex-content %}} +##### CSV with non-default boolean values +``` +sep=; +#datatype measurement,"boolean:y,Y,1:n,N,0",dateTime:RFC3339 +m,lbs,time +example,"1,280.7",2020-01-01T00:00:00Z +example,"1,352.5",2020-01-02T00:00:00Z +example,"1,862.8",2020-01-03T00:00:00Z +example,"2,014.9",2020-01-04T00:00:00Z +``` +{{% /flex-content %}} +{{% flex-content %}} +##### Resulting line protocol +``` +example lbs=1280.7 1577836800000000000 +example lbs=1352.5 1577923200000000000 +example lbs=1862.8 1578009600000000000 +example lbs=2014.9 1578096000000000000 +``` +{{% /flex-content %}} +{{< /flex >}} + +--- + +### Use different timestamp formats +The `influx write` command automatically detects **RFC3339** and **number** formatted +timestamps when converting CSV to line protocol. +If using a different timestamp format, [define your timestamp format](/v2.0/reference/syntax/annotated-csv/extended/#datetime) +in the `dateTime` datatype annotation. + +{{< flex >}} +{{% flex-content %}} +##### CSV with non-default timestamps +``` +#datatype measurement,dateTime:2006-01-02,field +m,time,lbs +example,2020-01-01,1280.7 +example,2020-01-02,1352.5 +example,2020-01-03,1862.8 +example,2020-01-04,2014.9 +``` +{{% /flex-content %}} +{{% flex-content %}} +##### Resulting line protocol +``` +example lbs=1280.7 1577836800000000000 +example lbs=1352.5 1577923200000000000 +example lbs=1862.8 1578009600000000000 +example lbs=2014.9 1578096000000000000 +``` +{{% /flex-content %}} +{{< /flex >}} From 04c8523d3c45b1b84c7a3bfded19b1bdc1109bc6 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Wed, 27 May 2020 10:42:31 -0600 Subject: [PATCH 4/6] updated ext csv docs to address PR feedback --- .../syntax/annotated-csv/extended.md | 28 +++---- content/v2.0/write-data/csv.md | 75 +++++++++++++++---- 2 files changed, 77 insertions(+), 26 deletions(-) diff --git a/content/v2.0/reference/syntax/annotated-csv/extended.md b/content/v2.0/reference/syntax/annotated-csv/extended.md index 489b889c0..89b97e857 100644 --- a/content/v2.0/reference/syntax/annotated-csv/extended.md +++ b/content/v2.0/reference/syntax/annotated-csv/extended.md @@ -20,6 +20,8 @@ related: **Extended annotated CSV** provides additional annotations and options that specify how CSV data should be converted to [line protocol](/v2.0/reference/syntax/line-protocol/) and written to InfluxDB. +InfluxDB uses the [`csv2lp` library](https://github.com/influxdata/influxdb/tree/master/pkg/csv2lp) +to convert CSV into line protocol. Extended annotated CSV supports all [Annotated CSV](/v2.0/reference/syntax/annotated-csv/) annotations. @@ -112,7 +114,7 @@ use the [`--precision` flag](/v2.0/reference/cli/influx/write/#flags) with the ##### Custom timestamp formats To specify a custom timestamp format, use timestamp formats as described in the [Go time package](https://golang.org/pkg/time). -For example: `2020-01-01`. +For example: `2020-01-02`. #### field Indicates the column is a **field** and auto-detects the field type. @@ -164,7 +166,7 @@ Append **float separators** to the `double` datatype annotation with a colon (`: For example: ``` -#datatype "fieldName|double:.," +#datatype "double:.," ``` {{% note %}} @@ -200,7 +202,7 @@ Append **integer separators** to the `long` datatype annotation with a colon (`: For example: ``` -#datatype "fieldName|long:.," +#datatype "long:.," ``` {{% note %}} @@ -236,7 +238,7 @@ Append **uinteger separators** to the `long` datatype annotation with a colon (` For example: ``` -#datatype "fieldName|usignedLong:.," +#datatype "usignedLong:.," ``` {{% note %}} @@ -266,7 +268,7 @@ Append the **boolean format** to the `boolean` datatype annotation with a colon For example: ``` -#datatype "fieldName|boolean:y,Y:n,N" +#datatype "boolean:y,Y:n,N" ``` {{% note %}} @@ -319,7 +321,7 @@ Use the `±HHmm` format to specify the timezone offset relative to UTC. ## Define custom column separator If columns are delimited using a character other than a comma, use the `sep` -keyword to define a custom separator for your CSV file. +keyword to define a custom separator **in the first line of your CSV file**. ``` sep=; @@ -336,23 +338,23 @@ header row using the following syntax: ##### Example annotation shorthand ``` -m|measurement|general,location|tag,temp|double,pm|long,time|dateTime:RFC3339 +m|measurement,location|tag|Hong Kong,temp|double,pm|long|0,time|dateTime:RFC3339 weather,San Francisco,51.9,38,2020-01-01T00:00:00Z -weather,New York,18.2,15,2020-01-01T00:00:00Z -,Hong Kong,53.6,171,2020-01-01T00:00:00Z +weather,New York,18.2,,2020-01-01T00:00:00Z +weather,,53.6,171,2020-01-01T00:00:00Z ``` ##### The shorthand explained -- The `m` column represents the **measurement** and has a default value of `general`. -- The `location` column is a **tag** with no default value. +- The `m` column represents the **measurement** and has no default value. +- The `location` column is a **tag** with the default value, `Hong Kong`. - The `temp` column is a **field** with **float** (`double`) values and no default value. -- The `pm` column is a **field** with **integer** (`long`) values and no default value. +- The `pm` column is a **field** with **integer** (`long`) values and a default of `0`. - The `time` column represents the **timestamp**, uses the **RFC3339** timestamp format, and has no default value. ##### Resulting line protocol ``` weather,location=San\ Francisco temp=51.9,pm=38i 1577836800000000000 -weather,location=New\ York temp=18.2,pm=18i 1577836800000000000 +weather,location=New\ York temp=18.2,pm=0i 1577836800000000000 general,location=Hong\ Kong temp=53.6,pm=171i 1577836800000000000 ``` diff --git a/content/v2.0/write-data/csv.md b/content/v2.0/write-data/csv.md index 55817263d..90a52401f 100644 --- a/content/v2.0/write-data/csv.md +++ b/content/v2.0/write-data/csv.md @@ -123,9 +123,52 @@ birds,loc=Detroit sighted=135 1590969600000000000 {{% /flex-content %}} {{< /flex >}} +#### Use files to inject headers +The `influx write` command supports importing multiple files in a single command. +Include annotations and header rows in their own file and import them with the write command. +Files are read in the order in which they're provided. + +```sh +influx write -b example-bucket \ + -f path/to/headers.csv \ + -f path/to/example.csv +``` + +{{< flex >}} +{{% flex-content %}} +##### headers.csv +``` +#constant measurement,birds +#datatype dataTime:2006-01-02,long,tag +``` +{{% /flex-content %}} +{{% flex-content %}} +##### example.csv +``` +date,sighted,loc +2020-01-01,12,Boise +2020-06-01,78,Boise +2020-01-01,54,Seattle +2020-06-01,112,Seattle +2020-01-01,9,Detroit +2020-06-01,135,Detroit +``` +{{% /flex-content %}} +{{< /flex >}} + +##### Resulting line protocol +``` +birds,loc=Boise sighted=12 1577836800000000000 +birds,loc=Boise sighted=78 1590969600000000000 +birds,loc=Seattle sighted=54 1577836800000000000 +birds,loc=Seattle sighted=112 1590969600000000000 +birds,loc=Detroit sighted=9 1577836800000000000 +birds,loc=Detroit sighted=135 1590969600000000000 +``` + ## Skip annotation headers -Some CSV data may include annotations that conflict with annotations necessary to -write CSV data to InfluxDB. +Some CSV data may include header rows that conflict with or lack the annotations +necessary to write CSV data to InfluxDB. Use the `--skipHeader` flag to specify the **number of rows to skip** at the beginning of the CSV data. @@ -135,6 +178,9 @@ influx write -b example-bucket \ --skipHeader=2 ``` +You can then [inject new header rows](#inject-annotation-headers) to rename columns +and provide the necessary annotations. + ## Process input as CSV The `influx write` command automatically processes files with the `.csv` extension as CSV files. If your CSV file uses a different extension, use the `--format` flat to explicitly @@ -256,11 +302,14 @@ To replace an existing column header row with annotation shorthand: 1. Use the `--skipHeader` flag to ignore the existing column header row. 2. Use the `--header` flag to inject a new column header row that uses annotation shorthand. - +{{% note %}} +`--skipHeader` is the same as `--skipHeader=1`. +{{% /note %}} + ```sh influx write -b example-bucket \ -f example.csv \ - --skipHeader=1 + --skipHeader --header="m|measurement,count|long|0,time|dateTime:RFC3339" ``` @@ -395,20 +444,20 @@ in the `boolean` datatype annotation. ``` sep=; #datatype measurement,"boolean:y,Y,1:n,N,0",dateTime:RFC3339 -m,lbs,time -example,"1,280.7",2020-01-01T00:00:00Z -example,"1,352.5",2020-01-02T00:00:00Z -example,"1,862.8",2020-01-03T00:00:00Z -example,"2,014.9",2020-01-04T00:00:00Z +m,verified,time +example,y,2020-01-01T00:00:00Z +example,n,2020-01-02T00:00:00Z +example,1,2020-01-03T00:00:00Z +example,N,2020-01-04T00:00:00Z ``` {{% /flex-content %}} {{% flex-content %}} ##### Resulting line protocol ``` -example lbs=1280.7 1577836800000000000 -example lbs=1352.5 1577923200000000000 -example lbs=1862.8 1578009600000000000 -example lbs=2014.9 1578096000000000000 +example verified=true 1577836800000000000 +example verified=false 1577923200000000000 +example verified=true 1578009600000000000 +example verified=false 1578096000000000000 ``` {{% /flex-content %}} {{< /flex >}} From ace6faf9b23655bf429426adc4c784122fcd7666 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Wed, 27 May 2020 13:57:39 -0600 Subject: [PATCH 5/6] added new influx write cli flags --- .../v2.0/reference/cli/influx/write/_index.md | 28 ++++++++++++------- .../v2.0/reference/cli/influx/write/dryrun.md | 25 ++++++++++------- content/v2.0/write-data/csv.md | 5 ++-- 3 files changed, 35 insertions(+), 23 deletions(-) diff --git a/content/v2.0/reference/cli/influx/write/_index.md b/content/v2.0/reference/cli/influx/write/_index.md index f593b05fb..cc24d5bf7 100644 --- a/content/v2.0/reference/cli/influx/write/_index.md +++ b/content/v2.0/reference/cli/influx/write/_index.md @@ -9,6 +9,9 @@ menu: parent: influx weight: 101 v2.0/tags: [write] +related: + - /v2.0/write-data/ + - /v2.0/write-data/csv/ --- The `influx write` command writes data to InfluxDB via stdin or from a specified file. @@ -27,15 +30,20 @@ influx write [command] | [dryrun](/v2.0/reference/cli/influx/write/dryrun) | Write to stdout instead of InfluxDB | ## Flags -| Flag | Description | Input type | {{< cli/mapped >}} | -|:---- |:----------- |:----------:|:------------------ | -| `-b`, `--bucket` | Bucket name | string | `INFLUX_BUCKET_NAME` | -| `--bucket-id` | Bucket ID | string | `INFLUX_BUCKET_ID` | -| `-f`, `--file` | File to import | string | | -| `--format` | Input format (`lp` or `csv`, default `lp`) | string | | -| `-h`, `--help` | Help for the `dryrun` command | | | -| `-o`, `--org` | Organization name | string | `INFLUX_ORG` | -| `--org-id` | Organization ID | string | `INFLUX_ORG_ID` | -| `-p`, `--precision` | Precision of the timestamps (default `ns`) | string | `INFLUX_PRECISION` | +| Flag | Description | Input type | {{< cli/mapped >}} | +|:---- |:----------- |:----------:|:------------------ | +| `-b`, `--bucket` | Bucket name | string | `INFLUX_BUCKET_NAME` | +| `--bucket-id` | Bucket ID | string | `INFLUX_BUCKET_ID` | +| `--debug` | Output errors to stderr | | | +| `--encoding` | Character encoding of input (default `UTF-8`) | string | | +| `-f`, `--file` | File to import | string | | +| `--format` | Input format (`lp` or `csv`, default `lp`) | string | | +| `--header` | Prepend header line to CSV input data | string | | +| `-h`, `--help` | Help for the `dryrun` command | | | +| `-o`, `--org` | Organization name | string | `INFLUX_ORG` | +| `--org-id` | Organization ID | string | `INFLUX_ORG_ID` | +| `-p`, `--precision` | Precision of the timestamps (default `ns`) | string | `INFLUX_PRECISION` | +| `--skipHeader` | Skip first n rows of input data | integer | | +| `--skipRowOnError` | Output CSV errors to stderr, but continue processing | | | {{% cli/influx-global-flags %}} diff --git a/content/v2.0/reference/cli/influx/write/dryrun.md b/content/v2.0/reference/cli/influx/write/dryrun.md index 728c4f89e..e613369d8 100644 --- a/content/v2.0/reference/cli/influx/write/dryrun.md +++ b/content/v2.0/reference/cli/influx/write/dryrun.md @@ -24,15 +24,20 @@ influx write dryrun [flags] ``` ## Flags -| Flag | Description | Input type | {{< cli/mapped >}} | -|:---- |:----------- |:----------:|:------------------ | -| `-b`, `--bucket` | Bucket name | string | `INFLUX_BUCKET_NAME` | -| `--bucket-id` | Bucket ID | string | `INFLUX_BUCKET_ID` | -| `-f`, `--file` | File to import | string | | -| `--format` | Input format (`lp` or `csv`, defaults `lp`) | string | | -| `-h`, `--help` | Help for the `dryrun` command | | | -| `-o`, `--org` | Organization name | string | `INFLUX_ORG` | -| `--org-id` | Organization ID | string | `INFLUX_ORG_ID` | -| `-p`, `--precision` | Precision of the timestamps (default `ns`) | string | `INFLUX_PRECISION` | +| Flag | Description | Input type | {{< cli/mapped >}} | +|:---- |:----------- |:----------:|:------------------ | +| `-b`, `--bucket` | Bucket name | string | `INFLUX_BUCKET_NAME` | +| `--bucket-id` | Bucket ID | string | `INFLUX_BUCKET_ID` | +| `--debug` | Output errors to stderr | | | +| `--encoding` | Character encoding of input (default `UTF-8`) | string | | +| `-f`, `--file` | File to import | string | | +| `--format` | Input format (`lp` or `csv`, default `lp`) | string | | +| `--header` | Prepend header line to CSV input data | string | | +| `-h`, `--help` | Help for the `dryrun` command | | | +| `-o`, `--org` | Organization name | string | `INFLUX_ORG` | +| `--org-id` | Organization ID | string | `INFLUX_ORG_ID` | +| `-p`, `--precision` | Precision of the timestamps (default `ns`) | string | `INFLUX_PRECISION` | +| `--skipHeader` | Skip first n rows of input data | integer | | +| `--skipRowOnError` | Output CSV errors to stderr, but continue processing | | | {{% cli/influx-global-flags %}} diff --git a/content/v2.0/write-data/csv.md b/content/v2.0/write-data/csv.md index 90a52401f..1ecbdaab5 100644 --- a/content/v2.0/write-data/csv.md +++ b/content/v2.0/write-data/csv.md @@ -58,9 +58,8 @@ mem,host=host2 used_percent=73.77 1577836820000000000 ``` {{% note %}} -To test the CSV to line protocol conversion process, include the `--dryrun` flag -with the `influx write` command to print the resulting line protocol to stdout -rather than write to InfluxDB. +To test the CSV to line protocol conversion process, use the `influx write dryrun` +command to print the resulting line protocol to stdout rather than write to InfluxDB. {{% /note %}} ## CSV Annotations From 31afd7c196041808937e545239525ad702ad1fdf Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Thu, 28 May 2020 11:23:29 -0600 Subject: [PATCH 6/6] updated write csv docs with PR feedback --- .../syntax/annotated-csv/extended.md | 60 ++++++++++--------- content/v2.0/write-data/csv.md | 4 +- 2 files changed, 35 insertions(+), 29 deletions(-) diff --git a/content/v2.0/reference/syntax/annotated-csv/extended.md b/content/v2.0/reference/syntax/annotated-csv/extended.md index 89b97e857..9fe12efbd 100644 --- a/content/v2.0/reference/syntax/annotated-csv/extended.md +++ b/content/v2.0/reference/syntax/annotated-csv/extended.md @@ -1,9 +1,9 @@ --- title: Extended annotated CSV description: > - Use the [`influx write` command](/v2.0/reference/cli/influx/write/) to write CSV data - to InfluxDB. Include annotations with the CSV data determine how the data translates - into [line protocol](/v2.0/reference/syntax/line-protocol/). + Extended annotated CSV provides additional annotations and options that specify + how CSV data should be converted to [line protocol](/v2.0/reference/syntax/line-protocol/) + and written to InfluxDB. menu: v2_0_ref: name: Extended annotated CSV @@ -27,7 +27,7 @@ annotations. {{% warn %}} The Flux [`csv.from` function](/v2.0/reference/flux/stdlib/csv/from/) only supports -**Annotated CSV**, not **Extended annotated CSV**. +**annotated CSV**, not **extended annotated CSV**. {{% /warn %}} To write data to InfluxDB, line protocol must include the following: @@ -37,8 +37,7 @@ To write data to InfluxDB, line protocol must include the following: - [timestamp](/v2.0/reference/syntax/line-protocol/#timestamp) _(Optional but recommended)_ - [tag set](/v2.0/reference/syntax/line-protocol/#tag-set) _(Optional)_ -Extended annotated CSV annotations identify which element of line protocol each -column represents and how the data should be converted. +Extended CSV annotations identify the element of line protocol a column represents. ## CSV Annotations Extended annotated CSV extends and adds the following annotations: @@ -48,10 +47,10 @@ Extended annotated CSV extends and adds the following annotations: - [timezone](#timezone) ### datatype -Use the `#datatype` annotation to specify which [line protocol element](/v2.0/reference/syntax/line-protocol/#elements-of-line-protocol) -each column represents. +Use the `#datatype` annotation to specify the [line protocol element](/v2.0/reference/syntax/line-protocol/#elements-of-line-protocol) +a column represents. To explicitly define a column as a **field** of a specific data type, use the field -type in the annotation. +type in the annotation (for example: `string`, `double`, `long`, etc.). | Data type | Resulting line protocol | |:---------- |:----------------------- | @@ -76,9 +75,9 @@ The **column value** is the **tag value**. #### dateTime Indicates the column is the **timestamp**. -`time` is as an alias for `dateTime`. +`time` is an alias for `dateTime`. If the [timestamp format](#supported-timestamp-formats) includes a time zone, -the parsed timestamp respects the time zone. +the parsed timestamp includes the time zone offset. By default, all timestamps are UTC. You can also use the [`#timezone` annotation](#timezone) to adjust timestamps to a specific time zone. @@ -106,9 +105,9 @@ Append the timestamp format to the `dateTime` datatype with (`:`). | **number** | Unix timestamp | `1577836800000000000` | {{% note %}} -If using the `number` timestamp format and timestamps are **not nanosecond Unix timestamps**, -use the [`--precision` flag](/v2.0/reference/cli/influx/write/#flags) with the -`influx write` command to specify the timestamp precision. +If using the `number` timestamp format and timestamps are **not in nanoseconds**, +use the [`influx write --precision` flag](/v2.0/reference/cli/influx/write/#flags) +to specify the [timestamp precision](/v2.0/reference/glossary/#precision). {{% /note %}} ##### Custom timestamp formats @@ -117,10 +116,17 @@ To specify a custom timestamp format, use timestamp formats as described in the For example: `2020-01-02`. #### field -Indicates the column is a **field** and auto-detects the field type. +Indicates the column is a **field**. The **column label** is the **field key**. The **column value** is the **field value**. +{{% note %}} +With the `field` datatype, field values are copies **as-is** to line protocol. +For information about line protocol values and how they are written to InfluxDB, +see [Line protocol data types and formats](/v2.0/reference/syntax/line-protocol/#data-types-and-format). +We generally recommend specifying the [field type](#field-types) in annotations. +{{% /note %}} + #### ignored The column is ignored and not written to InfluxDB. @@ -145,9 +151,9 @@ fraction from the whole number. If column values include or use other separators, such as commas (`,`) to visually separate large numbers into groups, specify the following **float separators**: -- **fraction separator**: Separates the fraction from the whole number -- **ignored separator**: Visually separates the whole number into groups but should - be ignored when parsing the float value. +- **fraction separator**: Separates the fraction from the whole number. +- **ignored separator**: Visually separates the whole number into groups but ignores + the separator when parsing the float value. Use the following syntax to specify **float separators**: @@ -171,7 +177,7 @@ For example: {{% note %}} If your **float separators** include a comma (`,`), wrap the column annotation in double -quotes (`""`) to prevent the comma from being parsed as column separator or delimiter. +quotes (`""`) to prevent the comma from being parsed as a column separator or delimiter. You can also [define a custom column separator](#define-custom-column-separator). {{% /note %}} @@ -182,8 +188,8 @@ the following **integer separators**: - **fraction separator**: Separates the fraction from the whole number. _**Integer values are truncated at the fraction separator when converted to line protocol.**_ -- **ignored separator**: Visually separates the whole number into groups but should - be ignored when parsing the integer value. +- **ignored separator**: Visually separates the whole number into groups but ignores + the separator when parsing the integer value. Use the following syntax to specify **integer separators**: @@ -207,7 +213,7 @@ For example: {{% note %}} If your **integer separators** include a comma (`,`), wrap the column annotation in double -quotes (`""`) to prevent the comma from being parsed as column separator or delimiter. +quotes (`""`) to prevent the comma from being parsed as a column separator or delimiter. You can also [define a custom column separator](#define-custom-column-separator). {{% /note %}} @@ -218,8 +224,8 @@ the following **uinteger separators**: - **fraction separator**: Separates the fraction from the whole number. _**Uinteger values are truncated at the fraction separator when converted to line protocol.**_ -- **ignored separator**: Visually separates the whole number into groups but should - be ignored when parsing the uinteger value. +- **ignored separator**: Visually separates the whole number into groups but ignores + the separator when parsing the uinteger value. Use the following syntax to specify **uinteger separators**: @@ -243,7 +249,7 @@ For example: {{% note %}} If your **uinteger separators** include a comma (`,`), wrap the column annotation in double -quotes (`""`) to prevent the comma from being parsed as column separator or delimiter. +quotes (`""`) to prevent the comma from being parsed as a column separator or delimiter. You can also [define a custom column separator](#define-custom-column-separator). {{% /note %}} @@ -273,7 +279,7 @@ For example: {{% note %}} If your **boolean format** contains commas (`,`), wrap the column annotation in double -quotes (`""`) to prevent the commas from being parsed as column separators or delimiters. +quotes (`""`) to prevent the comma from being parsed as a column separator or delimiter. You can also [define a custom column separator](#define-custom-column-separator). {{% /note %}} @@ -356,5 +362,5 @@ weather,,53.6,171,2020-01-01T00:00:00Z ``` weather,location=San\ Francisco temp=51.9,pm=38i 1577836800000000000 weather,location=New\ York temp=18.2,pm=0i 1577836800000000000 -general,location=Hong\ Kong temp=53.6,pm=171i 1577836800000000000 +weather,location=Hong\ Kong temp=53.6,pm=171i 1577836800000000000 ``` diff --git a/content/v2.0/write-data/csv.md b/content/v2.0/write-data/csv.md index 1ecbdaab5..9d0f2e51a 100644 --- a/content/v2.0/write-data/csv.md +++ b/content/v2.0/write-data/csv.md @@ -2,7 +2,7 @@ title: Write CSV data to InfluxDB description: > Use the [`influx write` command](/v2.0/reference/cli/influx/write/) to write CSV data - to InfluxDB. Include annotations with the CSV data determine how the data translates + to InfluxDB. Include annotations with the CSV data to determine how the data translates into [line protocol](/v2.0/reference/syntax/line-protocol/). menu: v2_0: @@ -343,7 +343,7 @@ in the `double`, `long`, and `unsignedLong` datatype annotations. {{% note %}} If your **numeric format separators** include a comma (`,`), wrap the column annotation in double -quotes (`""`) to prevent the comma from being parsed as column separator or delimiter. +quotes (`""`) to prevent the comma from being parsed as a column separator or delimiter. You can also [define a custom column separator](##################). {{% /note %}}