From 04c8523d3c45b1b84c7a3bfded19b1bdc1109bc6 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Wed, 27 May 2020 10:42:31 -0600 Subject: [PATCH] updated ext csv docs to address PR feedback --- .../syntax/annotated-csv/extended.md | 28 +++---- content/v2.0/write-data/csv.md | 75 +++++++++++++++---- 2 files changed, 77 insertions(+), 26 deletions(-) diff --git a/content/v2.0/reference/syntax/annotated-csv/extended.md b/content/v2.0/reference/syntax/annotated-csv/extended.md index 489b889c0..89b97e857 100644 --- a/content/v2.0/reference/syntax/annotated-csv/extended.md +++ b/content/v2.0/reference/syntax/annotated-csv/extended.md @@ -20,6 +20,8 @@ related: **Extended annotated CSV** provides additional annotations and options that specify how CSV data should be converted to [line protocol](/v2.0/reference/syntax/line-protocol/) and written to InfluxDB. +InfluxDB uses the [`csv2lp` library](https://github.com/influxdata/influxdb/tree/master/pkg/csv2lp) +to convert CSV into line protocol. Extended annotated CSV supports all [Annotated CSV](/v2.0/reference/syntax/annotated-csv/) annotations. @@ -112,7 +114,7 @@ use the [`--precision` flag](/v2.0/reference/cli/influx/write/#flags) with the ##### Custom timestamp formats To specify a custom timestamp format, use timestamp formats as described in the [Go time package](https://golang.org/pkg/time). -For example: `2020-01-01`. +For example: `2020-01-02`. #### field Indicates the column is a **field** and auto-detects the field type. @@ -164,7 +166,7 @@ Append **float separators** to the `double` datatype annotation with a colon (`: For example: ``` -#datatype "fieldName|double:.," +#datatype "double:.," ``` {{% note %}} @@ -200,7 +202,7 @@ Append **integer separators** to the `long` datatype annotation with a colon (`: For example: ``` -#datatype "fieldName|long:.," +#datatype "long:.," ``` {{% note %}} @@ -236,7 +238,7 @@ Append **uinteger separators** to the `long` datatype annotation with a colon (` For example: ``` -#datatype "fieldName|usignedLong:.," +#datatype "usignedLong:.," ``` {{% note %}} @@ -266,7 +268,7 @@ Append the **boolean format** to the `boolean` datatype annotation with a colon For example: ``` -#datatype "fieldName|boolean:y,Y:n,N" +#datatype "boolean:y,Y:n,N" ``` {{% note %}} @@ -319,7 +321,7 @@ Use the `±HHmm` format to specify the timezone offset relative to UTC. ## Define custom column separator If columns are delimited using a character other than a comma, use the `sep` -keyword to define a custom separator for your CSV file. +keyword to define a custom separator **in the first line of your CSV file**. ``` sep=; @@ -336,23 +338,23 @@ header row using the following syntax: ##### Example annotation shorthand ``` -m|measurement|general,location|tag,temp|double,pm|long,time|dateTime:RFC3339 +m|measurement,location|tag|Hong Kong,temp|double,pm|long|0,time|dateTime:RFC3339 weather,San Francisco,51.9,38,2020-01-01T00:00:00Z -weather,New York,18.2,15,2020-01-01T00:00:00Z -,Hong Kong,53.6,171,2020-01-01T00:00:00Z +weather,New York,18.2,,2020-01-01T00:00:00Z +weather,,53.6,171,2020-01-01T00:00:00Z ``` ##### The shorthand explained -- The `m` column represents the **measurement** and has a default value of `general`. -- The `location` column is a **tag** with no default value. +- The `m` column represents the **measurement** and has no default value. +- The `location` column is a **tag** with the default value, `Hong Kong`. - The `temp` column is a **field** with **float** (`double`) values and no default value. -- The `pm` column is a **field** with **integer** (`long`) values and no default value. +- The `pm` column is a **field** with **integer** (`long`) values and a default of `0`. - The `time` column represents the **timestamp**, uses the **RFC3339** timestamp format, and has no default value. ##### Resulting line protocol ``` weather,location=San\ Francisco temp=51.9,pm=38i 1577836800000000000 -weather,location=New\ York temp=18.2,pm=18i 1577836800000000000 +weather,location=New\ York temp=18.2,pm=0i 1577836800000000000 general,location=Hong\ Kong temp=53.6,pm=171i 1577836800000000000 ``` diff --git a/content/v2.0/write-data/csv.md b/content/v2.0/write-data/csv.md index 55817263d..90a52401f 100644 --- a/content/v2.0/write-data/csv.md +++ b/content/v2.0/write-data/csv.md @@ -123,9 +123,52 @@ birds,loc=Detroit sighted=135 1590969600000000000 {{% /flex-content %}} {{< /flex >}} +#### Use files to inject headers +The `influx write` command supports importing multiple files in a single command. +Include annotations and header rows in their own file and import them with the write command. +Files are read in the order in which they're provided. + +```sh +influx write -b example-bucket \ + -f path/to/headers.csv \ + -f path/to/example.csv +``` + +{{< flex >}} +{{% flex-content %}} +##### headers.csv +``` +#constant measurement,birds +#datatype dataTime:2006-01-02,long,tag +``` +{{% /flex-content %}} +{{% flex-content %}} +##### example.csv +``` +date,sighted,loc +2020-01-01,12,Boise +2020-06-01,78,Boise +2020-01-01,54,Seattle +2020-06-01,112,Seattle +2020-01-01,9,Detroit +2020-06-01,135,Detroit +``` +{{% /flex-content %}} +{{< /flex >}} + +##### Resulting line protocol +``` +birds,loc=Boise sighted=12 1577836800000000000 +birds,loc=Boise sighted=78 1590969600000000000 +birds,loc=Seattle sighted=54 1577836800000000000 +birds,loc=Seattle sighted=112 1590969600000000000 +birds,loc=Detroit sighted=9 1577836800000000000 +birds,loc=Detroit sighted=135 1590969600000000000 +``` + ## Skip annotation headers -Some CSV data may include annotations that conflict with annotations necessary to -write CSV data to InfluxDB. +Some CSV data may include header rows that conflict with or lack the annotations +necessary to write CSV data to InfluxDB. Use the `--skipHeader` flag to specify the **number of rows to skip** at the beginning of the CSV data. @@ -135,6 +178,9 @@ influx write -b example-bucket \ --skipHeader=2 ``` +You can then [inject new header rows](#inject-annotation-headers) to rename columns +and provide the necessary annotations. + ## Process input as CSV The `influx write` command automatically processes files with the `.csv` extension as CSV files. If your CSV file uses a different extension, use the `--format` flat to explicitly @@ -256,11 +302,14 @@ To replace an existing column header row with annotation shorthand: 1. Use the `--skipHeader` flag to ignore the existing column header row. 2. Use the `--header` flag to inject a new column header row that uses annotation shorthand. - +{{% note %}} +`--skipHeader` is the same as `--skipHeader=1`. +{{% /note %}} + ```sh influx write -b example-bucket \ -f example.csv \ - --skipHeader=1 + --skipHeader --header="m|measurement,count|long|0,time|dateTime:RFC3339" ``` @@ -395,20 +444,20 @@ in the `boolean` datatype annotation. ``` sep=; #datatype measurement,"boolean:y,Y,1:n,N,0",dateTime:RFC3339 -m,lbs,time -example,"1,280.7",2020-01-01T00:00:00Z -example,"1,352.5",2020-01-02T00:00:00Z -example,"1,862.8",2020-01-03T00:00:00Z -example,"2,014.9",2020-01-04T00:00:00Z +m,verified,time +example,y,2020-01-01T00:00:00Z +example,n,2020-01-02T00:00:00Z +example,1,2020-01-03T00:00:00Z +example,N,2020-01-04T00:00:00Z ``` {{% /flex-content %}} {{% flex-content %}} ##### Resulting line protocol ``` -example lbs=1280.7 1577836800000000000 -example lbs=1352.5 1577923200000000000 -example lbs=1862.8 1578009600000000000 -example lbs=2014.9 1578096000000000000 +example verified=true 1577836800000000000 +example verified=false 1577923200000000000 +example verified=true 1578009600000000000 +example verified=false 1578096000000000000 ``` {{% /flex-content %}} {{< /flex >}}