From a1c23310f55528c4577512e65328e3b7860f0d39 Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Fri, 7 May 2021 17:26:51 -0500 Subject: [PATCH 1/3] fix: replace incorrect page URLs. --- content/influxdb/v2.0/write-data/developer-tools/csv.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/influxdb/v2.0/write-data/developer-tools/csv.md b/content/influxdb/v2.0/write-data/developer-tools/csv.md index 0aad1fe58..80bb0126b 100644 --- a/content/influxdb/v2.0/write-data/developer-tools/csv.md +++ b/content/influxdb/v2.0/write-data/developer-tools/csv.md @@ -1,7 +1,7 @@ --- title: Write CSV data to InfluxDB description: > - Write CSV data with the [`influx write` command](#influx-write-command) or Flux. + Write CSV data with the [`influx write` command](/influxdb/cloud/reference/cli/influx/write/) or Flux. Include annotations with the CSV data to determine how the data translates into [line protocol](/influxdb/v2.0/reference/syntax/line-protocol/). menu: @@ -23,7 +23,7 @@ Write CSV data with the following methods: ### Flux Use the [csv.from()](/influxdb/v2.0/reference/flux/stdlib/csv/from/) and [to()](/influxdb/v2.0/reference/flux/stdlib/built-in/outputs/to/) Flux functions to write an annotated CSV to the bucket of your choice. - + {{< youtube wPKZ9i0DulQ >}} The experimental [csv.from()](/influxdb/v2.0/reference/flux/stdlib/csv/from/) function lets you write CSV from a URL. @@ -421,7 +421,7 @@ in the `double`, `long`, and `unsignedLong` datatype annotations. {{% note %}} If your **numeric format separators** include a comma (`,`), wrap the column annotation in double quotes (`""`) to prevent the comma from being parsed as a column separator or delimiter. -You can also [define a custom column separator](##################). +You can also [define a custom column separator](/influxdb/v2.0/reference/syntax/annotated-csv/extended/#define-custom-column-separator). {{% /note %}} {{< tabs-wrapper >}} From 4d34629f70b18566e5ca9a7cdf5a7cb03d847916 Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Fri, 7 May 2021 17:31:25 -0500 Subject: [PATCH 2/3] Revert "fix: update reference path in description (#2490)" This reverts commit 1d6cb41ba6db8199e218f09ec0e39e3ab0706b76. --- content/influxdb/v2.0/write-data/developer-tools/csv.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/influxdb/v2.0/write-data/developer-tools/csv.md b/content/influxdb/v2.0/write-data/developer-tools/csv.md index 7ffc1cf08..20871be12 100644 --- a/content/influxdb/v2.0/write-data/developer-tools/csv.md +++ b/content/influxdb/v2.0/write-data/developer-tools/csv.md @@ -1,7 +1,7 @@ --- title: Write CSV data to InfluxDB description: > - Write CSV data with the [`influx write` command](/influxdb/cloud/reference/cli/influx/write/) or Flux. + Write CSV data with the [`influx write` command](#influx-write-command) or Flux. Include annotations with the CSV data to determine how the data translates into [line protocol](/influxdb/v2.0/reference/syntax/line-protocol/). menu: From 6d0a44c75e81085597d0e0ddb76bdbc419a59440 Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Fri, 7 May 2021 17:31:34 -0500 Subject: [PATCH 3/3] Revert "fix: replaced bad anchor ref (#2490)" This reverts commit 316c6b2573bd7116ff5f178764db89a6f4f14b0a. --- .../v2.0/write-data/developer-tools/csv.md | 100 +----------------- 1 file changed, 5 insertions(+), 95 deletions(-) diff --git a/content/influxdb/v2.0/write-data/developer-tools/csv.md b/content/influxdb/v2.0/write-data/developer-tools/csv.md index 20871be12..0aad1fe58 100644 --- a/content/influxdb/v2.0/write-data/developer-tools/csv.md +++ b/content/influxdb/v2.0/write-data/developer-tools/csv.md @@ -16,26 +16,24 @@ related: - /influxdb/v2.0/reference/syntax/annotated-csv/ - /influxdb/v2.0/reference/cli/influx/write/ --- - Write CSV data with the following methods: - - [Flux](#flux) - [influx write command](#influx-write-command) ### Flux Use the [csv.from()](/influxdb/v2.0/reference/flux/stdlib/csv/from/) and [to()](/influxdb/v2.0/reference/flux/stdlib/built-in/outputs/to/) Flux functions to write an annotated CSV to the bucket of your choice. - + {{< youtube wPKZ9i0DulQ >}} The experimental [csv.from()](/influxdb/v2.0/reference/flux/stdlib/csv/from/) function lets you write CSV from a URL. The example below writes [NOAA water sample data](/influxdb/v2.0/reference/sample-data/#noaa-water-sample-data) to an example `noaa` bucket in an example organization: ```js -import "experimental/csv"; +import "experimental/csv" -csv.from((url: "https://influx-testdata.s3.amazonaws.com/noaa.csv")) - |> to((bucket: "noaa"), (org: "example-org")); +csv.from(url: "https://influx-testdata.s3.amazonaws.com/noaa.csv") + |> to(bucket: "noaa", org: "example-org") ``` ### influx write command @@ -47,7 +45,6 @@ Include annotations in the CSV file or inject them using the `--header` flag of the `influx write` command. ##### On this page - - [CSV Annotations](#csv-annotations) - [Inject annotation headers](#inject-annotation-headers) - [Skip annotation headers](#skip-annotation-headers) @@ -57,13 +54,11 @@ the `influx write` command. - [Advanced examples](#advanced-examples) ##### Example write command - ```sh influx write -b example-bucket -f path/to/example.csv ``` ##### example.csv - ``` #datatype measurement,tag,double,dateTime:RFC3339 m,host,used_percent,time @@ -76,7 +71,6 @@ mem,host2,73.77,2020-01-01T00:00:20Z ``` ##### Resulting line protocol - ``` mem,host=host1 used_percent=64.23 1577836800000000000 mem,host=host2 used_percent=72.01 1577836800000000000 @@ -92,7 +86,6 @@ command to print the resulting line protocol to stdout rather than write to Infl {{% /note %}} {{% note %}} - ##### "too many open files" errors When attempting to write large amounts of CSV data into InfluxDB, you might see an error like the following: @@ -106,11 +99,9 @@ To fix this error, run the following command to increase the number of open file ``` ulimit -n 10000 ``` - {{% /note %}} ## CSV Annotations - Use **CSV annotations** to specify which element of line protocol each CSV column represents and how to format the data. CSV annotations are rows at the beginning of a CSV file that describe column properties. @@ -129,12 +120,10 @@ To write data to InfluxDB, data must include the following: Use CSV annotations to specify which of these elements each column represents. ## Write raw query results back to InfluxDB - Flux returns query results in [annotated CSV](/influxdb/v2.0/reference/syntax/annotated-csv/). These results include all annotations necessary to write the data back to InfluxDB. ## Inject annotation headers - If the CSV data you want to write to InfluxDB does not contain the annotations required to properly convert the data to line protocol, use the `--header` flag to inject annotation rows into the CSV data. @@ -148,9 +137,7 @@ influx write -b example-bucket \ {{< flex >}} {{% flex-content %}} - ##### example.csv - ``` date,sighted,loc 2020-01-01,12,Boise @@ -160,12 +147,9 @@ date,sighted,loc 2020-01-01,9,Detroit 2020-06-01,135,Detroit ``` - {{% /flex-content %}} {{% flex-content %}} - ##### Resulting line protocol - ``` birds,loc=Boise sighted=12i 1577836800000000000 birds,loc=Boise sighted=78i 1590969600000000000 @@ -174,12 +158,10 @@ birds,loc=Seattle sighted=112i 1590969600000000000 birds,loc=Detroit sighted=9i 1577836800000000000 birds,loc=Detroit sighted=135i 1590969600000000000 ``` - {{% /flex-content %}} {{< /flex >}} #### Use files to inject headers - The `influx write` command supports importing multiple files in a single command. Include annotations and header rows in their own file and import them with the write command. Files are read in the order in which they're provided. @@ -192,19 +174,14 @@ influx write -b example-bucket \ {{< flex >}} {{% flex-content %}} - ##### headers.csv - ``` #constant measurement,birds #datatype dateTime:2006-01-02,long,tag ``` - {{% /flex-content %}} {{% flex-content %}} - ##### example.csv - ``` date,sighted,loc 2020-01-01,12,Boise @@ -214,12 +191,10 @@ date,sighted,loc 2020-01-01,9,Detroit 2020-06-01,135,Detroit ``` - {{% /flex-content %}} {{< /flex >}} ##### Resulting line protocol - ``` birds,loc=Boise sighted=12i 1577836800000000000 birds,loc=Boise sighted=78i 1590969600000000000 @@ -230,7 +205,6 @@ birds,loc=Detroit sighted=135i 1590969600000000000 ``` ## Skip annotation headers - Some CSV data may include header rows that conflict with or lack the annotations necessary to write CSV data to InfluxDB. Use the `--skipHeader` flag to specify the **number of rows to skip** at the @@ -246,7 +220,6 @@ You can then [inject new header rows](#inject-annotation-headers) to rename colu and provide the necessary annotations. ## Process input as CSV - The `influx write` command automatically processes files with the `.csv` extension as CSV files. If your CSV file uses a different extension, use the `--format` flat to explicitly declare the format of the input file. @@ -263,7 +236,6 @@ include the `.csv` extension or you declare the `csv`. {{% /note %}} ## Specify CSV character encoding - The `influx write` command assumes CSV files contain UTF-8 encoded characters. If your CSV data uses different character encoding, specify the encoding with the `--encoding`. @@ -275,7 +247,6 @@ influx write -b example-bucket \ ``` ## Skip rows with errors - If a row in your CSV data is missing an [element required to write to InfluxDB](/influxdb/v2.0/reference/syntax/line-protocol/#elements-of-line-protocol) or data is incorrectly formatted, when processing the row, the `influx write` command @@ -312,15 +283,12 @@ cpu,1.1 --- ### Define constants - Use the Extended annotated CSV [`#constant` annotation](/influxdb/v2.0/reference/syntax/annotated-csv/extended/#constant) to add a column and value to each row in the CSV data. {{< flex >}} {{% flex-content %}} - ##### CSV with constants - ``` #constant measurement,example #constant tag,source,csv @@ -331,34 +299,27 @@ count,time 9,2020-01-03T00:00:00Z 18,2020-01-04T00:00:00Z ``` - {{% /flex-content %}} {{% flex-content %}} - ##### Resulting line protocol - ``` example,source=csv count=1 1577836800000000000 example,source=csv count=4 1577923200000000000 example,source=csv count=9 1578009600000000000 example,source=csv count=18 1578096000000000000 ``` - {{% /flex-content %}} {{< /flex >}} --- ### Annotation shorthand - Extended annotated CSV supports [annotation shorthand](/influxdb/v2.0/reference/syntax/annotated-csv/extended/#annotation-shorthand), which lets you define the **column label**, **datatype**, and **default value** in the column header. {{< flex >}} {{% flex-content %}} - ##### CSV with annotation shorthand - ``` m|measurement,count|long|0,time|dateTime:RFC3339 example,1,2020-01-01T00:00:00Z @@ -366,24 +327,19 @@ example,4,2020-01-02T00:00:00Z example,,2020-01-03T00:00:00Z example,18,2020-01-04T00:00:00Z ``` - {{% /flex-content %}} {{% flex-content %}} - ##### Resulting line protocol - ``` example count=1 1577836800000000000 example count=4 1577923200000000000 example count=0 1578009600000000000 example count=18 1578096000000000000 ``` - {{% /flex-content %}} {{< /flex >}} #### Replace column header with annotation shorthand - It's possible to replace the column header row in a CSV file with annotation shorthand without modifying the CSV file. This lets you define column data types and default values while writing to InfluxDB. @@ -406,9 +362,7 @@ influx write -b example-bucket \ {{< flex >}} {{% flex-content %}} - ##### Unmodified example.csv - ``` m,count,time example,1,2020-01-01T00:00:00Z @@ -416,34 +370,27 @@ example,4,2020-01-02T00:00:00Z example,,2020-01-03T00:00:00Z example,18,2020-01-04T00:00:00Z ``` - {{% /flex-content %}} {{% flex-content %}} - ##### Resulting line protocol - ``` example count=1i 1577836800000000000 example count=4i 1577923200000000000 example count=0i 1578009600000000000 example count=18i 1578096000000000000 ``` - {{% /flex-content %}} {{< /flex >}} --- ### Ignore columns - Use the Extended annotated CSV [`#datatype ignored` annotation](/influxdb/v2.0/reference/syntax/annotated-csv/extended/#ignored) to ignore columns when writing CSV data to InfluxDB. {{< flex >}} {{% flex-content %}} - ##### CSV data with ignored column - ``` #datatype measurement,long,time,ignored m,count,time,foo @@ -452,26 +399,21 @@ example,4,2020-01-02T00:00:00Z,bar example,9,2020-01-03T00:00:00Z,baz example,18,2020-01-04T00:00:00Z,baz ``` - {{% /flex-content %}} {{% flex-content %}} - ##### Resulting line protocol - ``` m count=1i 1577836800000000000 m count=4i 1577923200000000000 m count=9i 1578009600000000000 m count=18i 1578096000000000000 ``` - {{% /flex-content %}} {{< /flex >}} --- ### Use alternate numeric formats - If your CSV data contains numeric values that use a non-default fraction separator (`.`) or contain group separators, [define your numeric format](/influxdb/v2.0/reference/syntax/annotated-csv/extended/#double) in the `double`, `long`, and `unsignedLong` datatype annotations. @@ -479,7 +421,7 @@ in the `double`, `long`, and `unsignedLong` datatype annotations. {{% note %}} If your **numeric format separators** include a comma (`,`), wrap the column annotation in double quotes (`""`) to prevent the comma from being parsed as a column separator or delimiter. -You can also [define a custom column separator](/influxdb/v2.0/reference/syntax/annotated-csv/extended/#define-custom-column-separator). +You can also [define a custom column separator](##################). {{% /note %}} {{< tabs-wrapper >}} @@ -491,9 +433,7 @@ You can also [define a custom column separator](/influxdb/v2.0/reference/syntax/ {{% tab-content %}} {{< flex >}} {{% flex-content %}} - ##### CSV with non-default float values - ``` #datatype measurement,"double:.,",dateTime:RFC3339 m,lbs,time @@ -502,19 +442,15 @@ example,"1,352.5",2020-01-02T00:00:00Z example,"1,862.8",2020-01-03T00:00:00Z example,"2,014.9",2020-01-04T00:00:00Z ``` - {{% /flex-content %}} {{% flex-content %}} - ##### Resulting line protocol - ``` example lbs=1280.7 1577836800000000000 example lbs=1352.5 1577923200000000000 example lbs=1862.8 1578009600000000000 example lbs=2014.9 1578096000000000000 ``` - {{% /flex-content %}} {{< /flex >}} {{% /tab-content %}} @@ -522,9 +458,7 @@ example lbs=2014.9 1578096000000000000 {{% tab-content %}} {{< flex >}} {{% flex-content %}} - ##### CSV with non-default integer values - ``` #datatype measurement,"long:.,",dateTime:RFC3339 m,lbs,time @@ -533,19 +467,15 @@ example,"1,352.0",2020-01-02T00:00:00Z example,"1,862.0",2020-01-03T00:00:00Z example,"2,014.9",2020-01-04T00:00:00Z ``` - {{% /flex-content %}} {{% flex-content %}} - ##### Resulting line protocol - ``` example lbs=1280i 1577836800000000000 example lbs=1352i 1577923200000000000 example lbs=1862i 1578009600000000000 example lbs=2014i 1578096000000000000 ``` - {{% /flex-content %}} {{< /flex >}} {{% /tab-content %}} @@ -553,9 +483,7 @@ example lbs=2014i 1578096000000000000 {{% tab-content %}} {{< flex >}} {{% flex-content %}} - ##### CSV with non-default uinteger values - ``` #datatype measurement,"unsignedLong:.,",dateTime:RFC3339 m,lbs,time @@ -564,19 +492,15 @@ example,"1,352.0",2020-01-02T00:00:00Z example,"1,862.0",2020-01-03T00:00:00Z example,"2,014.9",2020-01-04T00:00:00Z ``` - {{% /flex-content %}} {{% flex-content %}} - ##### Resulting line protocol - ``` example lbs=1280u 1577836800000000000 example lbs=1352u 1577923200000000000 example lbs=1862u 1578009600000000000 example lbs=2014u 1578096000000000000 ``` - {{% /flex-content %}} {{< /flex >}} {{% /tab-content %}} @@ -585,7 +509,6 @@ example lbs=2014u 1578096000000000000 --- ### Use alternate boolean format - Line protocol supports only [specific boolean values](/influxdb/v2.0/reference/syntax/line-protocol/#boolean). If your CSV data contains boolean values that line protocol does not support, [define your boolean format](/influxdb/v2.0/reference/syntax/annotated-csv/extended/#boolean) @@ -593,9 +516,7 @@ in the `boolean` datatype annotation. {{< flex >}} {{% flex-content %}} - ##### CSV with non-default boolean values - ``` sep=; #datatype measurement,"boolean:y,Y,1:n,N,0",dateTime:RFC3339 @@ -605,26 +526,21 @@ example,n,2020-01-02T00:00:00Z example,1,2020-01-03T00:00:00Z example,N,2020-01-04T00:00:00Z ``` - {{% /flex-content %}} {{% flex-content %}} - ##### Resulting line protocol - ``` example verified=true 1577836800000000000 example verified=false 1577923200000000000 example verified=true 1578009600000000000 example verified=false 1578096000000000000 ``` - {{% /flex-content %}} {{< /flex >}} --- ### Use different timestamp formats - The `influx write` command automatically detects **RFC3339** and **number** formatted timestamps when converting CSV to line protocol. If using a different timestamp format, [define your timestamp format](/influxdb/v2.0/reference/syntax/annotated-csv/extended/#datetime) @@ -632,9 +548,7 @@ in the `dateTime` datatype annotation. {{< flex >}} {{% flex-content %}} - ##### CSV with non-default timestamps - ``` #datatype measurement,dateTime:2006-01-02,field m,time,lbs @@ -643,18 +557,14 @@ example,2020-01-02,1352.5 example,2020-01-03,1862.8 example,2020-01-04,2014.9 ``` - {{% /flex-content %}} {{% flex-content %}} - ##### Resulting line protocol - ``` example lbs=1280.7 1577836800000000000 example lbs=1352.5 1577923200000000000 example lbs=1862.8 1578009600000000000 example lbs=2014.9 1578096000000000000 ``` - {{% /flex-content %}} {{< /flex >}}