From 991261f0165aa895f3b33eba2cae814e9cb34ada Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Thu, 12 Mar 2020 14:42:44 -0600 Subject: [PATCH 1/5] WIP geo guide --- content/v2.0/query-data/flux/geo.md | 75 +++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 content/v2.0/query-data/flux/geo.md diff --git a/content/v2.0/query-data/flux/geo.md b/content/v2.0/query-data/flux/geo.md new file mode 100644 index 000000000..e5a5a68df --- /dev/null +++ b/content/v2.0/query-data/flux/geo.md @@ -0,0 +1,75 @@ +--- +title: Work with geo-temporal data +description: > + Use the Flux Geo package to filter geo-temporal data and group by geographic location or track. +menu: + v2_0: + name: Geo-temporal data + parent: Query with Flux +weight: 220 +--- + +Use the [Flux Geo package](/v2.0/reference/flux/stdlib/experimental/geo) to +filter geo-temporal data and group by geographic location or track. +Import the `experimental/geo` package. + +```js +import "experimental/geo" +``` + +{{% warn %}} +The Geo package is experimental and subject to change at any time. +By using it, you agree to the [risks of experimental functions](/v2.0/reference/flux/stdlib/experimental/#use-experimental-functions-at-your-own-risk). +{{% /warn %}} + +### Primary Geo functions +The following functions provide the core functionality of the Geo package. +Other functions in the package exists primarily to support these functions: + +- [`geo.s2CellIDToken()`](/v2.0/reference/flux/stdlib/experimental/geo/s2cellidtoken/) +- [`geo.filterRows()`](/v2.0/reference/flux/stdlib/experimental/geo/filterrows/) +- [`geo.asTracks()`](/v2.0/reference/flux/stdlib/experimental/geo/astracks/) +- [`geo.groupByArea()`](/v2.0/reference/flux/stdlib/experimental/geo/groupbyarea/) + +### Shape your data for Geo functions +The Geo package requires the following data schema: + +- a **`lat` field** field containing the **latitude in decimal degrees** (WGS 84) +- a **`lon` field** field containing the **longitude in decimal degrees** (WGS 84) +- an **s2_cell_id** tag. + - Telegraf s2_geo processor (ships with Telegraf 1.14) + - Language-specific S2 Libraries + - `geo.s2CellIDToken()` to add the cell ID at query time. + +## Define a region +- box +- circle +- polygon + +## Filter geo-temporal data by region +- `geo.filterRows()` +- Strict vs non-strict filtering + +## Group geo-temporal data + +### Group data by area +- `geo.groupByArea()` + +### Group data into tracks or routes +- `geo.groupByTrack()` + +### Helper functions +```js +import "geo" + +shapeGeoData = (tables=<-, latField, lonField, level, correlationKey=["_time"]), + tables + |> map(fn: (r) => ({ r with _field: if r._field == latFeild then "lat" else if r._field == lonField then "lon" else r._field })) + |> geo.toRows(correlationKey: correlationKey) + |> map(fn: (r) => ({ r with s2_cell_id: geo.s2CellIDToken( point: {lat: r.lat, lon: r.lon}, level: level)})) + +from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "example-measurement") + |> shapeGeoData(latField: "latitude", lonField: "longitude", level: 10) +``` From 160d0b51db961d731f742ba2a9aed63f055230e5 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Fri, 20 Mar 2020 09:09:05 -0600 Subject: [PATCH 2/5] separated geo guide into a set of guides --- content/v2.0/query-data/flux/_index.md | 2 +- content/v2.0/query-data/flux/geo.md | 75 ----------- content/v2.0/query-data/flux/geo/_index.md | 68 ++++++++++ .../query-data/flux/geo/filter-by-region.md | 118 ++++++++++++++++++ .../query-data/flux/geo/group-geo-data.md | 15 +++ .../query-data/flux/geo/shape-geo-data.md | 107 ++++++++++++++++ 6 files changed, 309 insertions(+), 76 deletions(-) delete mode 100644 content/v2.0/query-data/flux/geo.md create mode 100644 content/v2.0/query-data/flux/geo/_index.md create mode 100644 content/v2.0/query-data/flux/geo/filter-by-region.md create mode 100644 content/v2.0/query-data/flux/geo/group-geo-data.md create mode 100644 content/v2.0/query-data/flux/geo/shape-geo-data.md diff --git a/content/v2.0/query-data/flux/_index.md b/content/v2.0/query-data/flux/_index.md index f9dc8dd20..a10bf37f5 100644 --- a/content/v2.0/query-data/flux/_index.md +++ b/content/v2.0/query-data/flux/_index.md @@ -33,4 +33,4 @@ data = from(bucket: "example-bucket") --- -{{< children >}} +{{< children pages="all" >}} diff --git a/content/v2.0/query-data/flux/geo.md b/content/v2.0/query-data/flux/geo.md deleted file mode 100644 index e5a5a68df..000000000 --- a/content/v2.0/query-data/flux/geo.md +++ /dev/null @@ -1,75 +0,0 @@ ---- -title: Work with geo-temporal data -description: > - Use the Flux Geo package to filter geo-temporal data and group by geographic location or track. -menu: - v2_0: - name: Geo-temporal data - parent: Query with Flux -weight: 220 ---- - -Use the [Flux Geo package](/v2.0/reference/flux/stdlib/experimental/geo) to -filter geo-temporal data and group by geographic location or track. -Import the `experimental/geo` package. - -```js -import "experimental/geo" -``` - -{{% warn %}} -The Geo package is experimental and subject to change at any time. -By using it, you agree to the [risks of experimental functions](/v2.0/reference/flux/stdlib/experimental/#use-experimental-functions-at-your-own-risk). -{{% /warn %}} - -### Primary Geo functions -The following functions provide the core functionality of the Geo package. -Other functions in the package exists primarily to support these functions: - -- [`geo.s2CellIDToken()`](/v2.0/reference/flux/stdlib/experimental/geo/s2cellidtoken/) -- [`geo.filterRows()`](/v2.0/reference/flux/stdlib/experimental/geo/filterrows/) -- [`geo.asTracks()`](/v2.0/reference/flux/stdlib/experimental/geo/astracks/) -- [`geo.groupByArea()`](/v2.0/reference/flux/stdlib/experimental/geo/groupbyarea/) - -### Shape your data for Geo functions -The Geo package requires the following data schema: - -- a **`lat` field** field containing the **latitude in decimal degrees** (WGS 84) -- a **`lon` field** field containing the **longitude in decimal degrees** (WGS 84) -- an **s2_cell_id** tag. - - Telegraf s2_geo processor (ships with Telegraf 1.14) - - Language-specific S2 Libraries - - `geo.s2CellIDToken()` to add the cell ID at query time. - -## Define a region -- box -- circle -- polygon - -## Filter geo-temporal data by region -- `geo.filterRows()` -- Strict vs non-strict filtering - -## Group geo-temporal data - -### Group data by area -- `geo.groupByArea()` - -### Group data into tracks or routes -- `geo.groupByTrack()` - -### Helper functions -```js -import "geo" - -shapeGeoData = (tables=<-, latField, lonField, level, correlationKey=["_time"]), - tables - |> map(fn: (r) => ({ r with _field: if r._field == latFeild then "lat" else if r._field == lonField then "lon" else r._field })) - |> geo.toRows(correlationKey: correlationKey) - |> map(fn: (r) => ({ r with s2_cell_id: geo.s2CellIDToken( point: {lat: r.lat, lon: r.lon}, level: level)})) - -from(bucket: "example-bucket") - |> range(start: -1h) - |> filter(fn: (r) => r._measurement == "example-measurement") - |> shapeGeoData(latField: "latitude", lonField: "longitude", level: 10) -``` diff --git a/content/v2.0/query-data/flux/geo/_index.md b/content/v2.0/query-data/flux/geo/_index.md new file mode 100644 index 000000000..8b6e3a597 --- /dev/null +++ b/content/v2.0/query-data/flux/geo/_index.md @@ -0,0 +1,68 @@ +--- +title: Work with geo-temporal data +description: > + Use the Flux Geo package to filter geo-temporal data and group by geographic location or track. +menu: + v2_0: + name: Geo-temporal data + parent: Query with Flux +weight: 220 +--- + +Use the [Flux Geo package](/v2.0/reference/flux/stdlib/experimental/geo) to +filter geo-temporal data and group by geographic location or track. +Import the `experimental/geo` package. + +```js +import "experimental/geo" +``` + +{{% warn %}} +The Geo package is experimental and subject to change at any time. +By using it, you agree to the [risks of experimental functions](/v2.0/reference/flux/stdlib/experimental/#use-experimental-functions-at-your-own-risk). +{{% /warn %}} + +{{< children >}} + +## Sample data +Many of the examples in this section use a `sampleGeoData` variable that represents +a sample set of geo-temporal data. +The [Bird Migration Sample Data](https://github.com/influxdata/influxdb2-sample-data/tree/master/bird-migration-data) +available on GitHub provides sample geo-temporal data that meets the +[requirements of the Flux Geo package](/v2.0/reference/flux/stdlib/experimental/geo/#geo-schema-requirements). + +### Load annotated CSV sample data +Use the [experimental `csv.from()` function](/v2.0/reference/flux/stdlib/experimental/csv/from/) +to load the sample bird migration annotated CSV data from GitHub: + +```js +import `experimental/csv` + +sampleGeoData = csv.from( + url: "https://github.com/influxdata/influxdb2-sample-data/blob/master/bird-migration-data/bird-migration.csv" +) +``` + +{{% note %}} +This downloads all the sample data each time you execute the query **(~1.3 MB)**. +If bandwidth is a concern, use the [`to()` function](/v2.0/reference/flux/stdlib/built-in/outputs/to/) +to write the data to a bucket, and then query the bucket with [`from()`](/v2.0/reference/flux/stdlib/built-in/inputs/from/). +{{% /note %}} + +### Write sample data to InfluxDB with line protocol +Use `curl` and the `influx write` command to write bird migration line protocol to InfluxDB. +Replace `example-bucket` with your destination bucket: + +```sh +curl https://raw.githubusercontent.com/influxdata/influxdb2-sample-data/master/bird-migration-data/bird-migration.line --output ./tmp-data +influx write -b example-bucket @./tmp-data +rm -f ./tmp-data +``` + +Use Flux to query the bird migration data and assign it to the `sampleGeoData` variable: + +```js +sampleGeoData = from(bucket: "example-bucket") + |> range(start: 2019-01-01T00:00:00Z, stop: 2019-12-31T23:59:59Z) + |> filter(fn: (r) => r._measurement == "migration") +``` diff --git a/content/v2.0/query-data/flux/geo/filter-by-region.md b/content/v2.0/query-data/flux/geo/filter-by-region.md new file mode 100644 index 000000000..b9ef1debf --- /dev/null +++ b/content/v2.0/query-data/flux/geo/filter-by-region.md @@ -0,0 +1,118 @@ +--- +title: Filter geo-temporal data by region +description: > + Use the `geo.filterRows` function to filter geo-temporal data by box-shaped, circular, or polygonal geographic regions. +menu: + v2_0: + name: Filter by region + parent: Geo-temporal data +weight: 302 +--- + +Use the [`geo.fitlerRows` function](/v2.0/reference/flux/stdlib/geo/filterrows/) +to filter geo-temporal data by geographic region. +Define a box-shaped, circular, or polygonal region to filter by. +Enable or disable strict filtering. + +- [Define geographic regions](#define-geographic-regions) +- [Strict and non-strict filtering](#strict-and-non-strict-filtering) + +The following example uses the [sample bird migration data](/v2.0/query-data/flux/geo/#sample-data) +and queries data points **within 200km of Cairo, Egypt**: + +```js +import "experimental/geo" + +sampleGeoData + |> geo.filterRows( + region: {lat: 30.04, lon: 31.23, radius: 200.0}, + strict: true + ) +``` + +## Define geographic regions +Many functions in the Geo package filter data based on geographic region. +Define geographic regions using the following shapes: + +- [box](#box) +- [circle](#circle) +- [polygon](#polygon) + +### box +Define a box-shaped region by specifying an object containing the following properties: + +- **minLat:** minimum latitude in decimal degrees (WGS 84) _(Float)_ +- **maxLat:** maximum latitude in decimal degrees (WGS 84) _(Float)_ +- **minLon:** minimum longitude in decimal degrees (WGS 84) _(Float)_ +- **maxLon:** maximum longitude in decimal degrees (WGS 84) _(Float)_ + +##### Example box-shaped region +```js +{ + minLat: 40.51757813, + maxLat: 40.86914063, + minLon: -73.65234375, + maxLon: -72.94921875 +} +``` + +### circle +Define a circular region by specifying an object containing the following properties: + +- **lat**: latitude of the circle center in decimal degrees (WGS 84) _(Float)_ +- **lon**: longitude of the circle center in decimal degrees (WGS 84) _(Float)_ +- **radius**: radius of the circle in kilometers (km) _(Float)_ + +##### Example circular region +```js +{ + lat: 40.69335938, + lon: -73.30078125, + radius: 20.0 +} +``` + +### polygon +Define a custom polygon region using an object containing the following properties: + +- **points**: points that define the custom polygon _(Array of objects)_ + + Define each point with an object containing the following properties: + + - **lat**: latitude in decimal degrees (WGS 84) _(Float)_ + - **lon**: longitude in decimal degrees (WGS 84) _(Float)_ + +##### Example polygonal region +```js +{ + points: [ + {lat: 40.671659, lon: -73.936631}, + {lat: 40.706543, lon: -73.749177}, + {lat: 40.791333, lon: -73.880327} + ] +} +``` + +## Strict and non-strict filtering +In most cases, the specified geographic region does not perfectly align with S2 grid cells. + +- **Non-strict filtering** returns points that may be outside of the specified region but + inside S2 grid cells partially covered by the region. +- **Strict filtering** returns only points inside the specified region. + +_Strict filtering is less performant, but more accurate than non-strict filtering._ + + S2 grid cell + Filter region + Returned point + +{{< flex >}} +{{% flex-content %}} +**Strict filtering** +{{< svg "/static/svgs/geo-strict.svg" >}} +{{% /flex-content %}} +{{% flex-content %}} +**Non-strict filtering** +{{< svg "/static/svgs/geo-non-strict.svg" >}} +{{% /flex-content %}} +{{< /flex >}} diff --git a/content/v2.0/query-data/flux/geo/group-geo-data.md b/content/v2.0/query-data/flux/geo/group-geo-data.md new file mode 100644 index 000000000..700a44f47 --- /dev/null +++ b/content/v2.0/query-data/flux/geo/group-geo-data.md @@ -0,0 +1,15 @@ +--- +title: Group geo-temporal data +description: > + placeholder +menu: + v2_0: + parent: Geo-temporal data +weight: 302 +--- + +### Group data by area +- `geo.groupByArea()` + +### Group data into tracks or routes +- `geo.groupByTrack()` diff --git a/content/v2.0/query-data/flux/geo/shape-geo-data.md b/content/v2.0/query-data/flux/geo/shape-geo-data.md new file mode 100644 index 000000000..790e9e125 --- /dev/null +++ b/content/v2.0/query-data/flux/geo/shape-geo-data.md @@ -0,0 +1,107 @@ +--- +title: Shape data to work with the Geo package +description: > + Functions in the Flux Geo package require **lat** and **lon** fields and an **s2_cell_id** tag. + Rename latitude and longitude fields and generate S2 cell ID tokens. +menu: + v2_0: + name: Shape geo-temporal data + parent: Geo-temporal data +weight: 301 +--- + +Functions in the Geo package require the following data schema: + +- an **s2_cell_id** tag containing the [S2 Cell ID](https://s2geometry.io/devguide/s2cell_hierarchy.html#s2cellid-numbering) + **as a token** +- a **`lat` field** field containing the **latitude in decimal degrees** (WGS 84) +- a **`lon` field** field containing the **longitude in decimal degrees** (WGS 84) + + +- [Generate S2 cell ID tokens](#generate-s2-cell-id-tokens) +- [Rename latitude and longitude fields](#rename-latitude-and-longitude-fields) + + +## Generate S2 cell ID tokens +The Geo package uses the [S2 Geometry Library](https://s2geometry.io/) which +represents geographic coordinates on a three-dimensional sphere. +The sphere is divided into [cells](https://s2geometry.io/devguide/s2cell_hierarchy), +each with a unique 64-bit identifier (S2 cell ID). +Grid and S2 cell ID accuracy are defined by a [level](https://s2geometry.io/resources/s2cell_statistics). + +{{% note %}} +For faster filtering, use higher S2 Cell ID levels. +But know that that higher levels increase [series cardinality](/v2.0/reference/glossary/#series-cardinality). +{{% /note %}} + +The Geo package requires S2 cell IDs as tokens. +Use one of the following options to generate and add S2 cell IDs to your geo-temporal data: + +- [Use the Telegraf S2 Geo processor](#use-the-telegraf-s2-geo-processor) +- [Use language-specific S2 libraries](#use-language-specific-s2-libraries) +- [Add S2 cell ID token with Flux](#add-s2-cell-id-token-with-flux) + +### Use the Telegraf S2 Geo processor +The [Telegraf S2 Geo (`s2geo`) processor](https://github.com/influxdata/telegraf/tree/master/plugins/processors/s2geo) +uses `lat` and `lon` field values to generate S2 cell ID tokens at a specified `cell_level`. + +To have Telegraf use existing latitude +and longitude values to generate S2 cell ID tokens and assign them to the `s2_cell_id` +tag in each line of line protocol, add the following to your `telegraf.conf`: + +```toml +[[processors.s2geo]] + ## The name of the lat and lon fields containing WGS-84 latitude and + ## longitude in decimal degrees. + lat_field = "lat" + lon_field = "lon" + + ## New tag to create + tag_key = "s2_cell_id" + + ## Cell level (see https://s2geometry.io/resources/s2cell_statistics.html) + cell_level = 9 +``` + +### Use language-specific S2 libraries +Many programming languages offer S2 Libraries with methods for generating S2 cell ID tokens. +Use latitude and longitude with the `s2.CellID.ToToken` endpoint of the S2 Geometry +Library to generate `s2_cell_id` tags. For example: + +- **Go:** [s2.CellID.ToToken()](https://godoc.org/github.com/golang/geo/s2#CellID.ToToken) +- **Python:** [s2sphere.CellId.to_token()](https://s2sphere.readthedocs.io/en/latest/api.html#s2sphere.CellId) +- **JavaScript:** [s2.cellid.toToken()](https://github.com/mapbox/node-s2/blob/master/API.md#cellidtotoken---string) + +### Add S2 cell ID tokens with Flux +Use the [`geo.s2CellIDToken()` function](/v2.0/reference/flux/stdlib/experimental/geo/s2cellidtoken/) +with existing longitude (`lon`) and latitude (`lat`) field values to generate and add the S2 cell ID token. +First, use the [`geo.toRows()` function](/v2.0/reference/flux/stdlib/experimental/geo/torows/) +to pivot **lat** and **lon** fields into row-wise sets: + +```js +import "experimental/geo" + +from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "example-measurement") + |> geo.toRows() + |> map(fn: (r) => ({ r with + s2_cell_id: geo.s2CellIDToken(point: {lon: r.lon, lat: r.lat}, level: 10) + })) +``` + +## Rename latitude and longitude fields +Use [`map()`](/v2.0/reference/flux/stdlib/built-in/transformations/map/) to rename +existing latitude and longitude fields keys using other names. + +```js +from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "example-measurement") + |> map(fn: (r) => ({ r with + _field: + if r._field == "existingLatitudeField" then "lat" + else if r._field == "existingLongitudeField" then "lon" + else r._field + })) +``` From db423159ddf350c748bee3288b11f63ee7ad9147 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Fri, 20 Mar 2020 10:22:36 -0600 Subject: [PATCH 3/5] added content for the geo group by functions --- .../query-data/flux/geo/group-geo-data.md | 53 +++++++++++++++++-- 1 file changed, 50 insertions(+), 3 deletions(-) diff --git a/content/v2.0/query-data/flux/geo/group-geo-data.md b/content/v2.0/query-data/flux/geo/group-geo-data.md index 700a44f47..caa1df440 100644 --- a/content/v2.0/query-data/flux/geo/group-geo-data.md +++ b/content/v2.0/query-data/flux/geo/group-geo-data.md @@ -1,15 +1,62 @@ --- title: Group geo-temporal data description: > - placeholder + Use the `geo.groupByArea()` and `geo.groupByTrack()` functions to group geo-temporal + data points by area or by tracks or routes. menu: v2_0: parent: Geo-temporal data weight: 302 --- +Use the `geo.groupByArea()` and `geo.asTracks()` functions to group geo-temporal +data points by area or by tracks or routes. + +- [Group data by area](#group-data-by-area) +- [Group data into tracks or routes](#group-data-into-tracks-or-routes) + ### Group data by area -- `geo.groupByArea()` +Use the [`geo.groupByArea()` function](/v2.0/reference/flux/stdlib/experimental/geo/groupbyarea/) +to group geo-temporal data points by geographic area. +Areas are determined by [S2 grid cells](https://s2geometry.io/devguide/s2cell_hierarchy.html#s2cellid-numbering) + +- Use the `newColumn` parameter to specify a new column in which to store the unique area identifier for each point. +- Use the `level` parameter to specify the [S2 cell level](https://s2geometry.io/resources/s2cell_statistics) + to use when calculating geographic areas. + +The following example uses the [sample bird migration data](/v2.0/query-data/flux/geo/#sample-data) +to query data points within 200km of Cairo, Egypt and group them by geographic area: + +```js +import "experimental/geo" + +sampleGeoData + |> geo.filterRows(region: {lat: 30.04, lon: 31.23, radius: 200.0}) + |> geo.groupByArea( + newColumn: "geoArea", + level: 5 + ) +``` ### Group data into tracks or routes -- `geo.groupByTrack()` +Use [`geo.asTracks()` function](/v2.0/reference/flux/stdlib/experimental/geo/astracks/) +to group data points into tracks or routes and order them by time or other columns. +Data must contain a unique identifier for each track. For example: `id` or `tid`. + +- Use the `groupBy` parameter to specify columns that uniquely identify each track or route. +- Use the `sortBy` parameter to specify which columns to sort by. Default is `["_time"]`. + +The following example uses the [sample bird migration data](/v2.0/query-data/flux/geo/#sample-data) +to query data points within 200km of Cairo, Egypt and group them into routes unique +to each bird: + +```js +import "experimental/geo" + +sampleGeoData + |> geo.filterRows(region: {lat: 30.04, lon: 31.23, radius: 200.0}) + |> geo.asTracks( + groupBy: ["id"], + sortBy: ["_time"] + ) +``` From 058268168096a019f143089648b2bd04e2612908 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Fri, 20 Mar 2020 11:01:37 -0600 Subject: [PATCH 4/5] updated children shortcode to show code examples when provided in the frontmatter --- CONTRIBUTING.md | 27 ++++++++--- .../query-data/flux/geo/filter-by-region.md | 10 ++++ .../query-data/flux/geo/group-geo-data.md | 8 ++++ .../query-data/flux/geo/shape-geo-data.md | 48 ++++++++++++------- layouts/shortcodes/children.html | 3 ++ 5 files changed, 72 insertions(+), 24 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b5d1fe948..d744780ce 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -75,6 +75,7 @@ related: # Creates links to specific internal and external content at the bottom external_url: # Used in children shortcode type="list" for page links that are external list_image: # Image included with article descriptions in children type="articles" shortcode list_note: # Used in children shortcode type="list" to add a small note next to listed links +list_code_example: # Code example included with article descriptions in children type="articles" shortcode ``` #### Title usage @@ -406,17 +407,29 @@ The following list types are available: - **list:** lists children article links in an unordered list. - **functions:** a special use-case designed for listing Flux functions. +#### Include a code example with a child summary +Use the `list_code_example` frontmatter to provide a code example with an article +in an articles list. + +~~~yaml +list_code_example: | + ```sh + This is a code example + ``` +~~~ + #### Children frontmatter Each children list `type` uses [frontmatter properties](#page-frontmatter) when generating the list of articles. The following table shows which children types use which frontmatter properties: -| Frontmatter | articles | list | functions | -|:----------- |:--------:|:----:|:---------:| -| `list_title` | ✓ | ✓ | ✓ | -| `description` | ✓ | | | -| `external_url` | ✓ | ✓ | | -| `list_image` | ✓ | | | -| `list_note` | | ✓ | | +| Frontmatter | articles | list | functions | +|:----------- |:--------:|:----:|:---------:| +| `list_title` | ✓ | ✓ | ✓ | +| `description` | ✓ | | | +| `external_url` | ✓ | ✓ | | +| `list_image` | ✓ | | | +| `list_note` | | ✓ | | +| `list_code_example` | ✓ | | | ### Inline icons The `icon` shortcode allows you to inject icons in paragraph text. diff --git a/content/v2.0/query-data/flux/geo/filter-by-region.md b/content/v2.0/query-data/flux/geo/filter-by-region.md index b9ef1debf..e788a6f54 100644 --- a/content/v2.0/query-data/flux/geo/filter-by-region.md +++ b/content/v2.0/query-data/flux/geo/filter-by-region.md @@ -7,6 +7,16 @@ menu: name: Filter by region parent: Geo-temporal data weight: 302 +list_code_example: | + ```js + import "experimental/geo" + + sampleGeoData + |> geo.filterRows( + region: {lat: 30.04, lon: 31.23, radius: 200.0}, + strict: true + ) + ``` --- Use the [`geo.fitlerRows` function](/v2.0/reference/flux/stdlib/geo/filterrows/) diff --git a/content/v2.0/query-data/flux/geo/group-geo-data.md b/content/v2.0/query-data/flux/geo/group-geo-data.md index caa1df440..a0519387b 100644 --- a/content/v2.0/query-data/flux/geo/group-geo-data.md +++ b/content/v2.0/query-data/flux/geo/group-geo-data.md @@ -7,6 +7,14 @@ menu: v2_0: parent: Geo-temporal data weight: 302 +list_code_example: | + ```js + import "experimental/geo" + + sampleGeoData + |> geo.groupByArea(newColumn: "geoArea", level: 5) + |> geo.asTracks(groupBy: ["id"],sortBy: ["_time"]) + ``` --- Use the `geo.groupByArea()` and `geo.asTracks()` functions to group geo-temporal diff --git a/content/v2.0/query-data/flux/geo/shape-geo-data.md b/content/v2.0/query-data/flux/geo/shape-geo-data.md index 790e9e125..9a688928e 100644 --- a/content/v2.0/query-data/flux/geo/shape-geo-data.md +++ b/content/v2.0/query-data/flux/geo/shape-geo-data.md @@ -8,6 +8,21 @@ menu: name: Shape geo-temporal data parent: Geo-temporal data weight: 301 +list_code_example: | + ```js + import "experimental/geo" + + sampleGeoData + |> map(fn: (r) => ({ r with + _field: + if r._field == "latitude" then "lat" + else if r._field == "longitude" then "lon" + else r._field + })) + |> map(fn: (r) => ({ r with + s2_cell_id: geo.s2CellIDToken(point: {lon: r.lon, lat: r.lat}, level: 10) + })) + ``` --- Functions in the Geo package require the following data schema: @@ -18,9 +33,24 @@ Functions in the Geo package require the following data schema: - a **`lon` field** field containing the **longitude in decimal degrees** (WGS 84) -- [Generate S2 cell ID tokens](#generate-s2-cell-id-tokens) - [Rename latitude and longitude fields](#rename-latitude-and-longitude-fields) +- [Generate S2 cell ID tokens](#generate-s2-cell-id-tokens) +## Rename latitude and longitude fields +Use [`map()`](/v2.0/reference/flux/stdlib/built-in/transformations/map/) to rename +existing latitude and longitude fields keys using other names. + +```js +from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "example-measurement") + |> map(fn: (r) => ({ r with + _field: + if r._field == "existingLatitudeField" then "lat" + else if r._field == "existingLongitudeField" then "lon" + else r._field + })) +``` ## Generate S2 cell ID tokens The Geo package uses the [S2 Geometry Library](https://s2geometry.io/) which @@ -89,19 +119,3 @@ from(bucket: "example-bucket") s2_cell_id: geo.s2CellIDToken(point: {lon: r.lon, lat: r.lat}, level: 10) })) ``` - -## Rename latitude and longitude fields -Use [`map()`](/v2.0/reference/flux/stdlib/built-in/transformations/map/) to rename -existing latitude and longitude fields keys using other names. - -```js -from(bucket: "example-bucket") - |> range(start: -1h) - |> filter(fn: (r) => r._measurement == "example-measurement") - |> map(fn: (r) => ({ r with - _field: - if r._field == "existingLatitudeField" then "lat" - else if r._field == "existingLongitudeField" then "lon" - else r._field - })) -``` diff --git a/layouts/shortcodes/children.html b/layouts/shortcodes/children.html index 360d44bc9..f24ca8051 100644 --- a/layouts/shortcodes/children.html +++ b/layouts/shortcodes/children.html @@ -35,6 +35,9 @@ {{ end }} {{ end }} + {{ if .Params.list_code_example }} + {{ .Params.list_code_example | markdownify }} + {{ end }} {{ end }} {{ else if (eq $type "functions") }} From eb218e89939e7ac5a6193fb76e1400e895be5a18 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Mon, 23 Mar 2020 14:14:04 -0600 Subject: [PATCH 5/5] updated geo guides to address PR feedback --- content/v2.0/query-data/flux/geo/_index.md | 24 +++++++++---- .../query-data/flux/geo/filter-by-region.md | 17 +++++----- .../query-data/flux/geo/group-geo-data.md | 20 +++++------ .../query-data/flux/geo/shape-geo-data.md | 34 +++++++++---------- .../flux/stdlib/experimental/geo/_index.md | 5 ++- layouts/shortcodes/children.html | 2 +- 6 files changed, 56 insertions(+), 46 deletions(-) diff --git a/content/v2.0/query-data/flux/geo/_index.md b/content/v2.0/query-data/flux/geo/_index.md index 8b6e3a597..3cc45fae4 100644 --- a/content/v2.0/query-data/flux/geo/_index.md +++ b/content/v2.0/query-data/flux/geo/_index.md @@ -11,19 +11,31 @@ weight: 220 Use the [Flux Geo package](/v2.0/reference/flux/stdlib/experimental/geo) to filter geo-temporal data and group by geographic location or track. -Import the `experimental/geo` package. - -```js -import "experimental/geo" -``` {{% warn %}} The Geo package is experimental and subject to change at any time. By using it, you agree to the [risks of experimental functions](/v2.0/reference/flux/stdlib/experimental/#use-experimental-functions-at-your-own-risk). {{% /warn %}} +**To work with geo-temporal data:** + +1. Import the `experimental/geo` package. + + ```js + import "experimental/geo" + ``` + +2. Load geo-temporal data. _See below for [sample geo-temporal data](#sample-data)._ +3. Do one or more of the following: + + - [Shape data to work with the Geo package](#shape-data-to-work-with-the-geo-package) + - [Filter data by region](#filter-geo-temporal-data-by-region) (using strict or non-strict filters) + - [Group data by area or by track](#group-geo-temporal-data) + {{< children >}} +--- + ## Sample data Many of the examples in this section use a `sampleGeoData` variable that represents a sample set of geo-temporal data. @@ -44,7 +56,7 @@ sampleGeoData = csv.from( ``` {{% note %}} -This downloads all the sample data each time you execute the query **(~1.3 MB)**. +`csv.from(url: ...)` downloads sample data each time you execute the query **(~1.3 MB)**. If bandwidth is a concern, use the [`to()` function](/v2.0/reference/flux/stdlib/built-in/outputs/to/) to write the data to a bucket, and then query the bucket with [`from()`](/v2.0/reference/flux/stdlib/built-in/inputs/from/). {{% /note %}} diff --git a/content/v2.0/query-data/flux/geo/filter-by-region.md b/content/v2.0/query-data/flux/geo/filter-by-region.md index e788a6f54..cffde7f5a 100644 --- a/content/v2.0/query-data/flux/geo/filter-by-region.md +++ b/content/v2.0/query-data/flux/geo/filter-by-region.md @@ -19,13 +19,11 @@ list_code_example: | ``` --- -Use the [`geo.fitlerRows` function](/v2.0/reference/flux/stdlib/geo/filterrows/) -to filter geo-temporal data by geographic region. -Define a box-shaped, circular, or polygonal region to filter by. -Enable or disable strict filtering. +Use the [`geo.filterRows` function](/v2.0/reference/flux/stdlib/geo/filterrows/) +to filter geo-temporal data by geographic region: -- [Define geographic regions](#define-geographic-regions) -- [Strict and non-strict filtering](#strict-and-non-strict-filtering) +1. [Define a geographic region](#define-geographic-regions) +2. [Use strict or non-strict filtering](#strict-and-non-strict-filtering) The following example uses the [sample bird migration data](/v2.0/query-data/flux/geo/#sample-data) and queries data points **within 200km of Cairo, Egypt**: @@ -40,9 +38,9 @@ sampleGeoData ) ``` -## Define geographic regions +## Define a geographic region Many functions in the Geo package filter data based on geographic region. -Define geographic regions using the following shapes: +Define a geographic region using one of the the following shapes: - [box](#box) - [circle](#circle) @@ -83,7 +81,8 @@ Define a circular region by specifying an object containing the following proper ``` ### polygon -Define a custom polygon region using an object containing the following properties: +Define a polygonal region with an object containing the latitude and longitude for +each point in the polygon: - **points**: points that define the custom polygon _(Array of objects)_ diff --git a/content/v2.0/query-data/flux/geo/group-geo-data.md b/content/v2.0/query-data/flux/geo/group-geo-data.md index a0519387b..8a4d64c3d 100644 --- a/content/v2.0/query-data/flux/geo/group-geo-data.md +++ b/content/v2.0/query-data/flux/geo/group-geo-data.md @@ -1,8 +1,8 @@ --- title: Group geo-temporal data description: > - Use the `geo.groupByArea()` and `geo.groupByTrack()` functions to group geo-temporal - data points by area or by tracks or routes. + Use the `geo.groupByArea()` to group geo-temporal data by area and `geo.asTracks()` + to group data into tracks or routes. menu: v2_0: parent: Geo-temporal data @@ -17,8 +17,8 @@ list_code_example: | ``` --- -Use the `geo.groupByArea()` and `geo.asTracks()` functions to group geo-temporal -data points by area or by tracks or routes. +Use the `geo.groupByArea()` to group geo-temporal data by area and `geo.asTracks()` +to group data into tracks or routes. - [Group data by area](#group-data-by-area) - [Group data into tracks or routes](#group-data-into-tracks-or-routes) @@ -28,9 +28,9 @@ Use the [`geo.groupByArea()` function](/v2.0/reference/flux/stdlib/experimental/ to group geo-temporal data points by geographic area. Areas are determined by [S2 grid cells](https://s2geometry.io/devguide/s2cell_hierarchy.html#s2cellid-numbering) -- Use the `newColumn` parameter to specify a new column in which to store the unique area identifier for each point. -- Use the `level` parameter to specify the [S2 cell level](https://s2geometry.io/resources/s2cell_statistics) - to use when calculating geographic areas. +- Specify a new column to store the unique area identifier for each point with the `newColumn` parameter. +- Specify the [S2 cell level](https://s2geometry.io/resources/s2cell_statistics) + to use when calculating geographic areas with the `level` parameter. The following example uses the [sample bird migration data](/v2.0/query-data/flux/geo/#sample-data) to query data points within 200km of Cairo, Egypt and group them by geographic area: @@ -46,13 +46,13 @@ sampleGeoData ) ``` -### Group data into tracks or routes +### Group data by track or route Use [`geo.asTracks()` function](/v2.0/reference/flux/stdlib/experimental/geo/astracks/) to group data points into tracks or routes and order them by time or other columns. Data must contain a unique identifier for each track. For example: `id` or `tid`. -- Use the `groupBy` parameter to specify columns that uniquely identify each track or route. -- Use the `sortBy` parameter to specify which columns to sort by. Default is `["_time"]`. +- Specify columns that uniquely identify each track or route with the `groupBy` parameter. +- Specify which columns to sort by with the `sortBy` parameter. Default is `["_time"]`. The following example uses the [sample bird migration data](/v2.0/query-data/flux/geo/#sample-data) to query data points within 200km of Cairo, Egypt and group them into routes unique diff --git a/content/v2.0/query-data/flux/geo/shape-geo-data.md b/content/v2.0/query-data/flux/geo/shape-geo-data.md index 9a688928e..a93b81e42 100644 --- a/content/v2.0/query-data/flux/geo/shape-geo-data.md +++ b/content/v2.0/query-data/flux/geo/shape-geo-data.md @@ -38,7 +38,7 @@ Functions in the Geo package require the following data schema: ## Rename latitude and longitude fields Use [`map()`](/v2.0/reference/flux/stdlib/built-in/transformations/map/) to rename -existing latitude and longitude fields keys using other names. +existing latitude and longitude fields using other names. ```js from(bucket: "example-bucket") @@ -53,31 +53,29 @@ from(bucket: "example-bucket") ``` ## Generate S2 cell ID tokens -The Geo package uses the [S2 Geometry Library](https://s2geometry.io/) which -represents geographic coordinates on a three-dimensional sphere. +The Geo package uses the [S2 Geometry Library](https://s2geometry.io/) to represent +geographic coordinates on a three-dimensional sphere. The sphere is divided into [cells](https://s2geometry.io/devguide/s2cell_hierarchy), each with a unique 64-bit identifier (S2 cell ID). Grid and S2 cell ID accuracy are defined by a [level](https://s2geometry.io/resources/s2cell_statistics). {{% note %}} -For faster filtering, use higher S2 Cell ID levels. -But know that that higher levels increase [series cardinality](/v2.0/reference/glossary/#series-cardinality). +To filter more quickly, use higher S2 Cell ID levels, +but know that that higher levels increase [series cardinality](/v2.0/reference/glossary/#series-cardinality). {{% /note %}} The Geo package requires S2 cell IDs as tokens. -Use one of the following options to generate and add S2 cell IDs to your geo-temporal data: +To generate add S2 cell IDs tokens to your data, use one of the following options: -- [Use the Telegraf S2 Geo processor](#use-the-telegraf-s2-geo-processor) -- [Use language-specific S2 libraries](#use-language-specific-s2-libraries) -- [Add S2 cell ID token with Flux](#add-s2-cell-id-token-with-flux) +- [Generate S2 cell ID tokens with Telegraf](#generate-s2-cell-id-tokens-with-telegraf) +- [Generate S2 cell ID tokens language-specific libraries](#generate-s2-cell-id-tokens-language-specific-libraries) +- [Generate S2 cell ID tokens with Flux](#generate-s2-cell-id-tokens-with-flux) -### Use the Telegraf S2 Geo processor -The [Telegraf S2 Geo (`s2geo`) processor](https://github.com/influxdata/telegraf/tree/master/plugins/processors/s2geo) -uses `lat` and `lon` field values to generate S2 cell ID tokens at a specified `cell_level`. +### Generate S2 cell ID tokens with Telegraf +Enable the [Telegraf S2 Geo (`s2geo`) processor](https://github.com/influxdata/telegraf/tree/master/plugins/processors/s2geo) +to generate S2 cell ID tokens at a specified `cell_level` using `lat` and `lon` field values. -To have Telegraf use existing latitude -and longitude values to generate S2 cell ID tokens and assign them to the `s2_cell_id` -tag in each line of line protocol, add the following to your `telegraf.conf`: +Add the `processors.s2geo` configuration to your Telegraf configuration file (`telegraf.conf`): ```toml [[processors.s2geo]] @@ -93,7 +91,9 @@ tag in each line of line protocol, add the following to your `telegraf.conf`: cell_level = 9 ``` -### Use language-specific S2 libraries +Telegraf stores the S2 cell ID token in the `s2_cell_id` tag. + +### Generate S2 cell ID tokens language-specific libraries Many programming languages offer S2 Libraries with methods for generating S2 cell ID tokens. Use latitude and longitude with the `s2.CellID.ToToken` endpoint of the S2 Geometry Library to generate `s2_cell_id` tags. For example: @@ -102,7 +102,7 @@ Library to generate `s2_cell_id` tags. For example: - **Python:** [s2sphere.CellId.to_token()](https://s2sphere.readthedocs.io/en/latest/api.html#s2sphere.CellId) - **JavaScript:** [s2.cellid.toToken()](https://github.com/mapbox/node-s2/blob/master/API.md#cellidtotoken---string) -### Add S2 cell ID tokens with Flux +### Generate S2 cell ID tokens with Flux Use the [`geo.s2CellIDToken()` function](/v2.0/reference/flux/stdlib/experimental/geo/s2cellidtoken/) with existing longitude (`lon`) and latitude (`lat`) field values to generate and add the S2 cell ID token. First, use the [`geo.toRows()` function](/v2.0/reference/flux/stdlib/experimental/geo/torows/) diff --git a/content/v2.0/reference/flux/stdlib/experimental/geo/_index.md b/content/v2.0/reference/flux/stdlib/experimental/geo/_index.md index 514761f14..46d77d612 100644 --- a/content/v2.0/reference/flux/stdlib/experimental/geo/_index.md +++ b/content/v2.0/reference/flux/stdlib/experimental/geo/_index.md @@ -48,9 +48,8 @@ Geometry Library to generate `s2_cell_id` tags. Specify your [S2 Cell ID level](https://s2geometry.io/resources/s2cell_statistics.html). {{% note %}} -For faster filtering, use higher S2 Cell ID levels. -But know that that higher levels increase -[series cardinality](/v2.0/reference/glossary/#series-cardinality). +To filter more quickly, use higher S2 Cell ID levels, +but know that that higher levels increase [series cardinality](/v2.0/reference/glossary/#series-cardinality). {{% /note %}} Language-specific implementations of the S2 Geometry Library provide methods for diff --git a/layouts/shortcodes/children.html b/layouts/shortcodes/children.html index f24ca8051..fedd86036 100644 --- a/layouts/shortcodes/children.html +++ b/layouts/shortcodes/children.html @@ -18,7 +18,7 @@ {{ $title := cond ( isset .Params "list_title" ) .Params.list_title .Title }} {{ $url := cond ( isset .Params "external_url" ) .Params.external_url .RelPermalink }} {{ $target := cond ( isset .Params "external_url" ) "_blank" "" }} -

{{ $title }}

+

{{ $title }}

{{- if .Description }}{{- .Description | markdownify -}} {{ else }}{{- .Summary | markdownify -}}