diff --git a/content/influxdb/v2.0/write-data/developer-tools/csv.md b/content/influxdb/v2.0/write-data/developer-tools/csv.md index 0aad1fe58..20871be12 100644 --- a/content/influxdb/v2.0/write-data/developer-tools/csv.md +++ b/content/influxdb/v2.0/write-data/developer-tools/csv.md @@ -16,24 +16,26 @@ related: - /influxdb/v2.0/reference/syntax/annotated-csv/ - /influxdb/v2.0/reference/cli/influx/write/ --- + Write CSV data with the following methods: + - [Flux](#flux) - [influx write command](#influx-write-command) ### Flux Use the [csv.from()](/influxdb/v2.0/reference/flux/stdlib/csv/from/) and [to()](/influxdb/v2.0/reference/flux/stdlib/built-in/outputs/to/) Flux functions to write an annotated CSV to the bucket of your choice. - + {{< youtube wPKZ9i0DulQ >}} The experimental [csv.from()](/influxdb/v2.0/reference/flux/stdlib/csv/from/) function lets you write CSV from a URL. The example below writes [NOAA water sample data](/influxdb/v2.0/reference/sample-data/#noaa-water-sample-data) to an example `noaa` bucket in an example organization: ```js -import "experimental/csv" +import "experimental/csv"; -csv.from(url: "https://influx-testdata.s3.amazonaws.com/noaa.csv") - |> to(bucket: "noaa", org: "example-org") +csv.from((url: "https://influx-testdata.s3.amazonaws.com/noaa.csv")) + |> to((bucket: "noaa"), (org: "example-org")); ``` ### influx write command @@ -45,6 +47,7 @@ Include annotations in the CSV file or inject them using the `--header` flag of the `influx write` command. ##### On this page + - [CSV Annotations](#csv-annotations) - [Inject annotation headers](#inject-annotation-headers) - [Skip annotation headers](#skip-annotation-headers) @@ -54,11 +57,13 @@ the `influx write` command. - [Advanced examples](#advanced-examples) ##### Example write command + ```sh influx write -b example-bucket -f path/to/example.csv ``` ##### example.csv + ``` #datatype measurement,tag,double,dateTime:RFC3339 m,host,used_percent,time @@ -71,6 +76,7 @@ mem,host2,73.77,2020-01-01T00:00:20Z ``` ##### Resulting line protocol + ``` mem,host=host1 used_percent=64.23 1577836800000000000 mem,host=host2 used_percent=72.01 1577836800000000000 @@ -86,6 +92,7 @@ command to print the resulting line protocol to stdout rather than write to Infl {{% /note %}} {{% note %}} + ##### "too many open files" errors When attempting to write large amounts of CSV data into InfluxDB, you might see an error like the following: @@ -99,9 +106,11 @@ To fix this error, run the following command to increase the number of open file ``` ulimit -n 10000 ``` + {{% /note %}} ## CSV Annotations + Use **CSV annotations** to specify which element of line protocol each CSV column represents and how to format the data. CSV annotations are rows at the beginning of a CSV file that describe column properties. @@ -120,10 +129,12 @@ To write data to InfluxDB, data must include the following: Use CSV annotations to specify which of these elements each column represents. ## Write raw query results back to InfluxDB + Flux returns query results in [annotated CSV](/influxdb/v2.0/reference/syntax/annotated-csv/). These results include all annotations necessary to write the data back to InfluxDB. ## Inject annotation headers + If the CSV data you want to write to InfluxDB does not contain the annotations required to properly convert the data to line protocol, use the `--header` flag to inject annotation rows into the CSV data. @@ -137,7 +148,9 @@ influx write -b example-bucket \ {{< flex >}} {{% flex-content %}} + ##### example.csv + ``` date,sighted,loc 2020-01-01,12,Boise @@ -147,9 +160,12 @@ date,sighted,loc 2020-01-01,9,Detroit 2020-06-01,135,Detroit ``` + {{% /flex-content %}} {{% flex-content %}} + ##### Resulting line protocol + ``` birds,loc=Boise sighted=12i 1577836800000000000 birds,loc=Boise sighted=78i 1590969600000000000 @@ -158,10 +174,12 @@ birds,loc=Seattle sighted=112i 1590969600000000000 birds,loc=Detroit sighted=9i 1577836800000000000 birds,loc=Detroit sighted=135i 1590969600000000000 ``` + {{% /flex-content %}} {{< /flex >}} #### Use files to inject headers + The `influx write` command supports importing multiple files in a single command. Include annotations and header rows in their own file and import them with the write command. Files are read in the order in which they're provided. @@ -174,14 +192,19 @@ influx write -b example-bucket \ {{< flex >}} {{% flex-content %}} + ##### headers.csv + ``` #constant measurement,birds #datatype dateTime:2006-01-02,long,tag ``` + {{% /flex-content %}} {{% flex-content %}} + ##### example.csv + ``` date,sighted,loc 2020-01-01,12,Boise @@ -191,10 +214,12 @@ date,sighted,loc 2020-01-01,9,Detroit 2020-06-01,135,Detroit ``` + {{% /flex-content %}} {{< /flex >}} ##### Resulting line protocol + ``` birds,loc=Boise sighted=12i 1577836800000000000 birds,loc=Boise sighted=78i 1590969600000000000 @@ -205,6 +230,7 @@ birds,loc=Detroit sighted=135i 1590969600000000000 ``` ## Skip annotation headers + Some CSV data may include header rows that conflict with or lack the annotations necessary to write CSV data to InfluxDB. Use the `--skipHeader` flag to specify the **number of rows to skip** at the @@ -220,6 +246,7 @@ You can then [inject new header rows](#inject-annotation-headers) to rename colu and provide the necessary annotations. ## Process input as CSV + The `influx write` command automatically processes files with the `.csv` extension as CSV files. If your CSV file uses a different extension, use the `--format` flat to explicitly declare the format of the input file. @@ -236,6 +263,7 @@ include the `.csv` extension or you declare the `csv`. {{% /note %}} ## Specify CSV character encoding + The `influx write` command assumes CSV files contain UTF-8 encoded characters. If your CSV data uses different character encoding, specify the encoding with the `--encoding`. @@ -247,6 +275,7 @@ influx write -b example-bucket \ ``` ## Skip rows with errors + If a row in your CSV data is missing an [element required to write to InfluxDB](/influxdb/v2.0/reference/syntax/line-protocol/#elements-of-line-protocol) or data is incorrectly formatted, when processing the row, the `influx write` command @@ -283,12 +312,15 @@ cpu,1.1 --- ### Define constants + Use the Extended annotated CSV [`#constant` annotation](/influxdb/v2.0/reference/syntax/annotated-csv/extended/#constant) to add a column and value to each row in the CSV data. {{< flex >}} {{% flex-content %}} + ##### CSV with constants + ``` #constant measurement,example #constant tag,source,csv @@ -299,27 +331,34 @@ count,time 9,2020-01-03T00:00:00Z 18,2020-01-04T00:00:00Z ``` + {{% /flex-content %}} {{% flex-content %}} + ##### Resulting line protocol + ``` example,source=csv count=1 1577836800000000000 example,source=csv count=4 1577923200000000000 example,source=csv count=9 1578009600000000000 example,source=csv count=18 1578096000000000000 ``` + {{% /flex-content %}} {{< /flex >}} --- ### Annotation shorthand + Extended annotated CSV supports [annotation shorthand](/influxdb/v2.0/reference/syntax/annotated-csv/extended/#annotation-shorthand), which lets you define the **column label**, **datatype**, and **default value** in the column header. {{< flex >}} {{% flex-content %}} + ##### CSV with annotation shorthand + ``` m|measurement,count|long|0,time|dateTime:RFC3339 example,1,2020-01-01T00:00:00Z @@ -327,19 +366,24 @@ example,4,2020-01-02T00:00:00Z example,,2020-01-03T00:00:00Z example,18,2020-01-04T00:00:00Z ``` + {{% /flex-content %}} {{% flex-content %}} + ##### Resulting line protocol + ``` example count=1 1577836800000000000 example count=4 1577923200000000000 example count=0 1578009600000000000 example count=18 1578096000000000000 ``` + {{% /flex-content %}} {{< /flex >}} #### Replace column header with annotation shorthand + It's possible to replace the column header row in a CSV file with annotation shorthand without modifying the CSV file. This lets you define column data types and default values while writing to InfluxDB. @@ -362,7 +406,9 @@ influx write -b example-bucket \ {{< flex >}} {{% flex-content %}} + ##### Unmodified example.csv + ``` m,count,time example,1,2020-01-01T00:00:00Z @@ -370,27 +416,34 @@ example,4,2020-01-02T00:00:00Z example,,2020-01-03T00:00:00Z example,18,2020-01-04T00:00:00Z ``` + {{% /flex-content %}} {{% flex-content %}} + ##### Resulting line protocol + ``` example count=1i 1577836800000000000 example count=4i 1577923200000000000 example count=0i 1578009600000000000 example count=18i 1578096000000000000 ``` + {{% /flex-content %}} {{< /flex >}} --- ### Ignore columns + Use the Extended annotated CSV [`#datatype ignored` annotation](/influxdb/v2.0/reference/syntax/annotated-csv/extended/#ignored) to ignore columns when writing CSV data to InfluxDB. {{< flex >}} {{% flex-content %}} + ##### CSV data with ignored column + ``` #datatype measurement,long,time,ignored m,count,time,foo @@ -399,21 +452,26 @@ example,4,2020-01-02T00:00:00Z,bar example,9,2020-01-03T00:00:00Z,baz example,18,2020-01-04T00:00:00Z,baz ``` + {{% /flex-content %}} {{% flex-content %}} + ##### Resulting line protocol + ``` m count=1i 1577836800000000000 m count=4i 1577923200000000000 m count=9i 1578009600000000000 m count=18i 1578096000000000000 ``` + {{% /flex-content %}} {{< /flex >}} --- ### Use alternate numeric formats + If your CSV data contains numeric values that use a non-default fraction separator (`.`) or contain group separators, [define your numeric format](/influxdb/v2.0/reference/syntax/annotated-csv/extended/#double) in the `double`, `long`, and `unsignedLong` datatype annotations. @@ -421,7 +479,7 @@ in the `double`, `long`, and `unsignedLong` datatype annotations. {{% note %}} If your **numeric format separators** include a comma (`,`), wrap the column annotation in double quotes (`""`) to prevent the comma from being parsed as a column separator or delimiter. -You can also [define a custom column separator](##################). +You can also [define a custom column separator](/influxdb/v2.0/reference/syntax/annotated-csv/extended/#define-custom-column-separator). {{% /note %}} {{< tabs-wrapper >}} @@ -433,7 +491,9 @@ You can also [define a custom column separator](##################). {{% tab-content %}} {{< flex >}} {{% flex-content %}} + ##### CSV with non-default float values + ``` #datatype measurement,"double:.,",dateTime:RFC3339 m,lbs,time @@ -442,15 +502,19 @@ example,"1,352.5",2020-01-02T00:00:00Z example,"1,862.8",2020-01-03T00:00:00Z example,"2,014.9",2020-01-04T00:00:00Z ``` + {{% /flex-content %}} {{% flex-content %}} + ##### Resulting line protocol + ``` example lbs=1280.7 1577836800000000000 example lbs=1352.5 1577923200000000000 example lbs=1862.8 1578009600000000000 example lbs=2014.9 1578096000000000000 ``` + {{% /flex-content %}} {{< /flex >}} {{% /tab-content %}} @@ -458,7 +522,9 @@ example lbs=2014.9 1578096000000000000 {{% tab-content %}} {{< flex >}} {{% flex-content %}} + ##### CSV with non-default integer values + ``` #datatype measurement,"long:.,",dateTime:RFC3339 m,lbs,time @@ -467,15 +533,19 @@ example,"1,352.0",2020-01-02T00:00:00Z example,"1,862.0",2020-01-03T00:00:00Z example,"2,014.9",2020-01-04T00:00:00Z ``` + {{% /flex-content %}} {{% flex-content %}} + ##### Resulting line protocol + ``` example lbs=1280i 1577836800000000000 example lbs=1352i 1577923200000000000 example lbs=1862i 1578009600000000000 example lbs=2014i 1578096000000000000 ``` + {{% /flex-content %}} {{< /flex >}} {{% /tab-content %}} @@ -483,7 +553,9 @@ example lbs=2014i 1578096000000000000 {{% tab-content %}} {{< flex >}} {{% flex-content %}} + ##### CSV with non-default uinteger values + ``` #datatype measurement,"unsignedLong:.,",dateTime:RFC3339 m,lbs,time @@ -492,15 +564,19 @@ example,"1,352.0",2020-01-02T00:00:00Z example,"1,862.0",2020-01-03T00:00:00Z example,"2,014.9",2020-01-04T00:00:00Z ``` + {{% /flex-content %}} {{% flex-content %}} + ##### Resulting line protocol + ``` example lbs=1280u 1577836800000000000 example lbs=1352u 1577923200000000000 example lbs=1862u 1578009600000000000 example lbs=2014u 1578096000000000000 ``` + {{% /flex-content %}} {{< /flex >}} {{% /tab-content %}} @@ -509,6 +585,7 @@ example lbs=2014u 1578096000000000000 --- ### Use alternate boolean format + Line protocol supports only [specific boolean values](/influxdb/v2.0/reference/syntax/line-protocol/#boolean). If your CSV data contains boolean values that line protocol does not support, [define your boolean format](/influxdb/v2.0/reference/syntax/annotated-csv/extended/#boolean) @@ -516,7 +593,9 @@ in the `boolean` datatype annotation. {{< flex >}} {{% flex-content %}} + ##### CSV with non-default boolean values + ``` sep=; #datatype measurement,"boolean:y,Y,1:n,N,0",dateTime:RFC3339 @@ -526,21 +605,26 @@ example,n,2020-01-02T00:00:00Z example,1,2020-01-03T00:00:00Z example,N,2020-01-04T00:00:00Z ``` + {{% /flex-content %}} {{% flex-content %}} + ##### Resulting line protocol + ``` example verified=true 1577836800000000000 example verified=false 1577923200000000000 example verified=true 1578009600000000000 example verified=false 1578096000000000000 ``` + {{% /flex-content %}} {{< /flex >}} --- ### Use different timestamp formats + The `influx write` command automatically detects **RFC3339** and **number** formatted timestamps when converting CSV to line protocol. If using a different timestamp format, [define your timestamp format](/influxdb/v2.0/reference/syntax/annotated-csv/extended/#datetime) @@ -548,7 +632,9 @@ in the `dateTime` datatype annotation. {{< flex >}} {{% flex-content %}} + ##### CSV with non-default timestamps + ``` #datatype measurement,dateTime:2006-01-02,field m,time,lbs @@ -557,14 +643,18 @@ example,2020-01-02,1352.5 example,2020-01-03,1862.8 example,2020-01-04,2014.9 ``` + {{% /flex-content %}} {{% flex-content %}} + ##### Resulting line protocol + ``` example lbs=1280.7 1577836800000000000 example lbs=1352.5 1577923200000000000 example lbs=1862.8 1578009600000000000 example lbs=2014.9 1578096000000000000 ``` + {{% /flex-content %}} {{< /flex >}}