diff --git a/.circleci/config.yml b/.circleci/config.yml index 8acf131d1..c0841eee3 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -6,6 +6,9 @@ jobs: environment: HUGO_VERSION: "0.81.0" S3DEPLOY_VERSION: "2.3.5" + # From https://github.com/bep/s3deploy/releases + S3DEPLOY_VERSION_HASH: "95de91ed207ba32abd0df71f9681c1ede952f8358f3510b980b02550254c941a" + steps: - checkout - restore_cache: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 76a649621..cb475d703 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -260,7 +260,7 @@ Find more info [here][{{< enterprise-link >}}] ``` ### InfluxDB Cloud Content -For sections content that relate specifically to InfluxDB Cloud, use the `{{% cloud %}}` shortcode. +For sections of content that relate specifically to InfluxDB Cloud, use the `{{% cloud %}}` shortcode. ```md {{% cloud %}} @@ -333,6 +333,18 @@ current product. Easier to maintain being you update the version number in the ` {{< latest-patch >}} ``` +### API endpoint +Use the `{{< api-endpoint >}}` shortcode to generate a code block that contains +a colored request method and a specified API endpoint. +Provide the following arguments: + +- **method**: HTTP request method (get, post, patch, put, or delete) +- **endpoint**: API endpoint + +```md +{{< api-endpoint method="get" endpoint="/api/v2/tasks">}} +``` + ### Tabbed Content Shortcodes are available for creating "tabbed" content (content that is changed by a users' selection). Ther following three must be used: @@ -437,6 +449,13 @@ you can customize the text by passing a string argument with the shortcode. **Output:** This is required +If using other named arguments like `key` or `color`, use the `text` argument to +customize the text of the required message. + +```md +{{< req text="Required if ..." color="blue" type="key" >}} +``` + #### Required elements in a list When identifying required elements in a list, use `{{< req type="key" >}}` to generate a "* Required" key before the list. For required elements in the list, include @@ -450,6 +469,18 @@ a "* Required" key before the list. For required elements in the list, include - **This element is NOT required** ``` +#### Change color of required text +Use the `color` argument to change the color of required text. +The following colors are available: + +- blue +- green +- magenta + +```md +{{< req color="magenta" text="This is required" >}} +``` + ### Keybinds Use the `{{< keybind >}}` shortcode to include OS-specific keybindings/hotkeys. The following parameters are available: @@ -489,7 +520,7 @@ flowchart TB Use the `{{< filesystem-diagram >}}` shortcode to create a styled file system diagram using a Markdown unordered list. -##### Example filestsytem diagram shortcode +##### Example filesystem diagram shortcode ```md {{< filesystem-diagram >}} - Dir1/ @@ -637,6 +668,80 @@ list_code_example: | ``` ~~~ +#### Organize and include native code examples +To include text from a file in `/assets/text/`, use the +`{{< get-assets-text >}}` shortcode and provide the relative path and filename. + +This is useful for maintaining and referencing sample code variants in their + native file formats. + +1. Store code examples in their native formats at `/assets/text/`. + ```md + /assets/text/example1/example.js + /assets/text/example1/example.py + ``` + +2. Include the files, e.g. in code tabs +````md + {{% code-tabs-wrapper %}} + {{% code-tabs %}} + [Javascript](#js) + [Python](#py) + {{% /code-tabs %}} + {{% code-tab-content %}} + ```js + {{< get-assets-text "example1/example.js" >}} + ``` + {{% /code-tab-content %}} + {{% code-tab-content %}} + ```py + {{< get-assets-text "example1/example.py" >}} + ``` + {{% /code-tab-content %}} + {{% /code-tabs-wrapper %}} +```` + +#### Include specific files from the same directory +To include the text from one file in another file in the same +directory, use the `{{< get-leaf-text >}}` shortcode. +The directory that contains both files must be a +Hugo [*Leaf Bundle*](https://gohugo.io/content-management/page-bundles/#leaf-bundles), +a directory that doesn't have any child directories. + +In the following example, `api` is a leaf bundle. `content` isn't. + +```md +content +| +|--- api + | query.pdmc + | query.sh + | _index.md +``` + +##### query.pdmc +```md +# Query examples +``` + +##### query.sh +```md +curl https://localhost:8086/query +``` + +To include `query.sh` and `query.pdmc` in `api/_index.md`, use the following code: +````md +{{< get-leaf-text "query.pdmc" >}} + +# Curl example +```sh +{{< get-leaf-text "query.sh" >}} +``` +```` + +Avoid using the following file extensions when naming included text files since Hugo interprets these as markup languages: +`.ad`, `.adoc`, `.asciidoc`, `.htm`, `.html`, `.markdown`, `.md`, `.mdown`, `.mmark`, `.pandoc`, `.pdc`, `.org`, or `.rst`. + #### Reference a query example in children To include a query example with the children in your list, update `data/query_examples.yml` with the example code, input, and output, and use the `list_query_example` diff --git a/api-docs/yarn.lock b/api-docs/yarn.lock old mode 100755 new mode 100644 index cd1d43f81..b2b281efe --- a/api-docs/yarn.lock +++ b/api-docs/yarn.lock @@ -403,15 +403,6 @@ classnames@^2.2.6: resolved "https://registry.yarnpkg.com/classnames/-/classnames-2.3.1.tgz#dfcfa3891e306ec1dad105d0e88f4417b8535e8e" integrity sha512-OlQdbZ7gLfGarSqxesMesDa5uz7KFbID8Kpq/SxIoNGDqY8lSYs0D+hhtBXhcdB3rcbXArFr7vlHheLk1voeNA== -clipboard@^2.0.0: - version "2.0.8" - resolved "https://registry.yarnpkg.com/clipboard/-/clipboard-2.0.8.tgz#ffc6c103dd2967a83005f3f61976aa4655a4cdba" - integrity sha512-Y6WO0unAIQp5bLmk1zdThRhgJt/x3ks6f30s3oE3H1mgIEU33XyQjEf8gsf6DxC7NPX8Y1SsNWjUjL/ywLnnbQ== - dependencies: - good-listener "^1.2.2" - select "^1.1.2" - tiny-emitter "^2.0.0" - cliui@^6.0.0: version "6.0.0" resolved "https://registry.yarnpkg.com/cliui/-/cliui-6.0.0.tgz#511d702c0c4e41ca156d7d0e96021f23e13225b1" @@ -568,11 +559,6 @@ decko@^1.2.0: resolved "https://registry.yarnpkg.com/decko/-/decko-1.2.0.tgz#fd43c735e967b8013306884a56fbe665996b6817" integrity sha1-/UPHNelnuAEzBohKVvvmZZlraBc= -delegate@^3.1.2: - version "3.2.0" - resolved "https://registry.yarnpkg.com/delegate/-/delegate-3.2.0.tgz#b66b71c3158522e8ab5744f720d8ca0c2af59166" - integrity sha512-IofjkYBZaZivn0V8nnsMJGBr4jVLxHDheKSW88PyxS5QC4Vo9ZbZVvhzlSxY87fVq3STR6r+4cGepyHkcWOQSw== - des.js@^1.0.0: version "1.0.1" resolved "https://registry.yarnpkg.com/des.js/-/des.js-1.0.1.tgz#5382142e1bdc53f85d86d53e5f4aa7deb91e0843" @@ -718,13 +704,6 @@ globals@^11.1.0: resolved "https://registry.yarnpkg.com/globals/-/globals-11.12.0.tgz#ab8795338868a0babd8525758018c2a7eb95c42e" integrity sha512-WOBp/EEGUiIsJSp7wcv/y6MO+lV9UoncWqxuFfm8eBwzWNgyfBd6Gz+IeKQ9jCmyhoH99g15M3T+QaVHFjizVA== -good-listener@^1.2.2: - version "1.2.2" - resolved "https://registry.yarnpkg.com/good-listener/-/good-listener-1.2.2.tgz#d53b30cdf9313dffb7dc9a0d477096aa6d145c50" - integrity sha1-1TswzfkxPf+33JoNR3CWqm0UXFA= - dependencies: - delegate "^3.1.2" - grapheme-splitter@^1.0.4: version "1.0.4" resolved "https://registry.yarnpkg.com/grapheme-splitter/-/grapheme-splitter-1.0.4.tgz#9cf3a665c6247479896834af35cf1dbb4400767e" @@ -1208,11 +1187,9 @@ postcss-value-parser@^4.0.2: integrity sha512-97DXOFbQJhk71ne5/Mt6cOu6yxsSfM0QGQyl0L25Gca4yGWEGJaig7l7gbCX623VqTBNGLRLaVUCnNkcedlRSQ== prismjs@^1.20.0: - version "1.23.0" - resolved "https://registry.yarnpkg.com/prismjs/-/prismjs-1.23.0.tgz#d3b3967f7d72440690497652a9d40ff046067f33" - integrity sha512-c29LVsqOaLbBHuIbsTxaKENh1N2EQBOHaWv7gkHN4dgRbxSREqDnDbtFJYdpPauS4YCplMSNCABQ6Eeor69bAA== - optionalDependencies: - clipboard "^2.0.0" + version "1.24.0" + resolved "https://registry.yarnpkg.com/prismjs/-/prismjs-1.24.0.tgz#0409c30068a6c52c89ef7f1089b3ca4de56be2ac" + integrity sha512-SqV5GRsNqnzCL8k5dfAjCNhUrF3pR0A9lTDSCUZeh/LIshheXJEaP0hwLz2t4XHivd2J/v2HR+gRnigzeKe3cQ== process-nextick-args@~2.0.0: version "2.0.1" @@ -1440,11 +1417,6 @@ scheduler@^0.19.1: loose-envify "^1.1.0" object-assign "^4.1.1" -select@^1.1.2: - version "1.1.2" - resolved "https://registry.yarnpkg.com/select/-/select-1.1.2.tgz#0e7350acdec80b1108528786ec1d4418d11b396d" - integrity sha1-DnNQrN7ICxEIUoeG7B1EGNEbOW0= - set-blocking@^2.0.0: version "2.0.0" resolved "https://registry.yarnpkg.com/set-blocking/-/set-blocking-2.0.0.tgz#045f9782d011ae9a6803ddd382b24392b3d890f7" @@ -1633,11 +1605,6 @@ timers-browserify@^2.0.4: dependencies: setimmediate "^1.0.4" -tiny-emitter@^2.0.0: - version "2.1.0" - resolved "https://registry.yarnpkg.com/tiny-emitter/-/tiny-emitter-2.1.0.tgz#1d1a56edfc51c43e863cbb5382a72330e3555423" - integrity sha512-NB6Dk1A9xgQPMoGqC5CVXn123gWyte215ONT5Pp5a0yt4nlEoO1ZWeCwpncaekPHXO60i47ihFnZPiRPjRMq4Q== - to-arraybuffer@^1.0.0: version "1.0.1" resolved "https://registry.yarnpkg.com/to-arraybuffer/-/to-arraybuffer-1.0.1.tgz#7d229b1fcc637e466ca081180836a7aabff83f43" diff --git a/assets/js/influxdb-url.js b/assets/js/influxdb-url.js index f1394a3aa..119ac7572 100644 --- a/assets/js/influxdb-url.js +++ b/assets/js/influxdb-url.js @@ -128,14 +128,46 @@ Cookies.set('influx-docs-api-lib', selectedApiLib) // Iterate through code blocks and update InfluxDB urls // Requires objects with cloud and oss keys and url values function updateUrls(prevUrls, newUrls) { - var preference = getPreference() + var prevUrlsParsed = { + cloud: {}, + oss: {} + } + var newUrlsParsed = { + cloud: {}, + oss: {} + } + + Object.keys(prevUrls).forEach(function(k) { + try { + prevUrlsParsed[k] = new URL(prevUrls[k]) + } catch { + prevUrlsParsed[k] = { host: prevUrls[k] } + } + }) + + Object.keys(newUrls).forEach(function(k) { + try { + newUrlsParsed[k] = new URL(newUrls[k]) + } catch { + newUrlsParsed[k] = { host: newUrls[k] } + } + }) + + /** + * Match and replace host with host + * then replace URL with URL. + **/ var cloudReplacements = [ - { replace: prevUrls.cloud, with: newUrls.cloud}, - { replace: prevUrls.oss, with: newUrls.cloud } + { replace: prevUrlsParsed.cloud.host, with: newUrlsParsed.cloud.host }, + { replace: prevUrlsParsed.oss.host, with: newUrlsParsed.cloud.host }, + { replace: prevUrls.cloud, with: newUrls.cloud }, + { replace: prevUrls.oss, with: newUrls.cloud }, ] var ossReplacements = [ + { replace: prevUrlsParsed.cloud.host, with: newUrlsParsed.cloud.host }, + { replace: prevUrlsParsed.oss.host, with: newUrlsParsed.oss.host }, { replace: prevUrls.cloud, with: newUrls.cloud}, { replace: prevUrls.oss, with: newUrls.oss } ] diff --git a/assets/styles/layouts/_api-overrides.scss b/assets/styles/layouts/_api-overrides.scss index 031e000b5..e9a65eb0a 100644 --- a/assets/styles/layouts/_api-overrides.scss +++ b/assets/styles/layouts/_api-overrides.scss @@ -59,7 +59,7 @@ $bold: 700; color: $b-pool; } &:before { - content: '\e918'; + content: "\e919"; font-family: 'icomoon'; margin-right: .65rem; } @@ -73,7 +73,7 @@ $bold: 700; border-radius: 4.5px; transition: all .2s; &:before { - content: "\e933"; + content: "\e934"; display: inline-block; font-size: .95rem; margin-right: .5rem; diff --git a/assets/styles/layouts/_article.scss b/assets/styles/layouts/_article.scss index 94f311b6c..4c158cd73 100644 --- a/assets/styles/layouts/_article.scss +++ b/assets/styles/layouts/_article.scss @@ -147,6 +147,10 @@ font-size: .9rem; font-weight: $medium; } + + &.blue {color: $b-dodger;} + &.green {color: $gr-viridian;} + &.magenta {color: $p-comet;} } h2,h3,h4,h5,h6 { diff --git a/assets/styles/layouts/_sidebar.scss b/assets/styles/layouts/_sidebar.scss index 57040ec4a..0dd2914e4 100644 --- a/assets/styles/layouts/_sidebar.scss +++ b/assets/styles/layouts/_sidebar.scss @@ -9,7 +9,7 @@ position: relative; flex-grow: 1; &:after { - content: '\e905'; + content: "\e905"; display: block; font-family: 'icomoon'; position: absolute; diff --git a/assets/styles/layouts/_top-nav.scss b/assets/styles/layouts/_top-nav.scss index 3a5468b9c..1ffd527d9 100644 --- a/assets/styles/layouts/_top-nav.scss +++ b/assets/styles/layouts/_top-nav.scss @@ -75,7 +75,7 @@ } &:after { - content: '\e917'; + content: "\e918"; font-family: 'icomoon'; position: absolute; top: .45rem; diff --git a/assets/styles/layouts/_url-selector.scss b/assets/styles/layouts/_url-selector.scss index 7f94dc026..67ba6b738 100644 --- a/assets/styles/layouts/_url-selector.scss +++ b/assets/styles/layouts/_url-selector.scss @@ -329,7 +329,7 @@ label:after { border-radius: 0 0 $radius $radius; &:before { - content: "\e923"; + content: "\e924"; display: inline-block; margin-right: .35rem; font-family: "icomoon"; diff --git a/assets/styles/layouts/_v1-overrides.scss b/assets/styles/layouts/_v1-overrides.scss index 12d255d9c..e07611d2d 100644 --- a/assets/styles/layouts/_v1-overrides.scss +++ b/assets/styles/layouts/_v1-overrides.scss @@ -1,7 +1,7 @@ body.v1, body.platform{ .article .article--content { blockquote { - padding: 1.65rem 2rem .1rem 2rem; + padding: 1.65rem 2rem; margin: 1rem 0 2rem; border-width: 0 0 0 4px; border-style: solid; diff --git a/assets/styles/layouts/article/_buttons.scss b/assets/styles/layouts/article/_buttons.scss index 2743420d0..f9a3917ad 100644 --- a/assets/styles/layouts/article/_buttons.scss +++ b/assets/styles/layouts/article/_buttons.scss @@ -35,7 +35,7 @@ a.btn { } &.download:before { - content: "\e91c"; + content: "\e91d"; font-family: "icomoon"; margin-right: .5rem; font-size: 1.1rem; diff --git a/assets/styles/layouts/article/_children.scss b/assets/styles/layouts/article/_children.scss index d9d3a4171..b1df3ec77 100644 --- a/assets/styles/layouts/article/_children.scss +++ b/assets/styles/layouts/article/_children.scss @@ -5,7 +5,7 @@ margin-top: -.5rem; a a:after { - content: "\e919"; + content: "\e91a"; font-family: "icomoon"; color: rgba($article-heading, .35); vertical-align: bottom; diff --git a/assets/styles/layouts/article/_code.scss b/assets/styles/layouts/article/_code.scss index 6dad04b30..11b5f6e57 100644 --- a/assets/styles/layouts/article/_code.scss +++ b/assets/styles/layouts/article/_code.scss @@ -80,8 +80,11 @@ pre .api { font-weight: bold; font-size: .9rem; - &.get { background: $gr-rainforest; } + &.get { background: $gr-viridian; } &.post { background: $b-ocean; } + &.patch { background: $y-topaz; } + &.delete { background: $r-ruby; } + &.put {background: $br-pulsar; } } //////////////////////////////////////////////////////////////////////////////// diff --git a/assets/styles/layouts/article/_feedback.scss b/assets/styles/layouts/article/_feedback.scss index ef427c848..792c32c98 100644 --- a/assets/styles/layouts/article/_feedback.scss +++ b/assets/styles/layouts/article/_feedback.scss @@ -74,12 +74,12 @@ } &.edit:before { - content: "\e92e"; + content: "\e92f"; font-size: .75rem; vertical-align: top; } &.issue:before { - content: "\e933"; + content: "\e934"; font-size: .95rem; } } diff --git a/assets/styles/layouts/article/_lists.scss b/assets/styles/layouts/article/_lists.scss index 62335622e..10a2d3345 100644 --- a/assets/styles/layouts/article/_lists.scss +++ b/assets/styles/layouts/article/_lists.scss @@ -55,4 +55,9 @@ li { } .list-note { font-size: .85rem } + + h4,h5,h6 { + margin-top: 1em; + padding-top: 0; + } } diff --git a/assets/styles/layouts/article/_tables.scss b/assets/styles/layouts/article/_tables.scss index 0ecb0f483..43d203c06 100644 --- a/assets/styles/layouts/article/_tables.scss +++ b/assets/styles/layouts/article/_tables.scss @@ -47,6 +47,8 @@ table { } } + + img { margin-bottom: 0; } } #flags:not(.no-shorthand), #global-flags { diff --git a/assets/styles/styles-default.scss b/assets/styles/styles-default.scss index 513530306..36e7f6798 100644 --- a/assets/styles/styles-default.scss +++ b/assets/styles/styles-default.scss @@ -1,4 +1,4 @@ -// InfluxData Docs Default Theme (Light) +// InfluxData Docs Default Theme (Light) // Import Tools @import "tools/icomoon", diff --git a/assets/styles/tools/_icomoon.scss b/assets/styles/tools/_icomoon.scss index b6d4624ef..c871239ff 100644 --- a/assets/styles/tools/_icomoon.scss +++ b/assets/styles/tools/_icomoon.scss @@ -1,10 +1,10 @@ @font-face { font-family: 'icomoon'; - src: url('fonts/icomoon.eot?a22byr'); - src: url('fonts/icomoon.eot?a22byr#iefix') format('embedded-opentype'), - url('fonts/icomoon.ttf?a22byr') format('truetype'), - url('fonts/icomoon.woff?a22byr') format('woff'), - url('fonts/icomoon.svg?a22byr#icomoon') format('svg'); + src: url('fonts/icomoon.eot?itn2ph'); + src: url('fonts/icomoon.eot?itn2ph#iefix') format('embedded-opentype'), + url('fonts/icomoon.ttf?itn2ph') format('truetype'), + url('fonts/icomoon.woff?itn2ph') format('woff'), + url('fonts/icomoon.svg?itn2ph#icomoon') format('svg'); font-weight: normal; font-style: normal; font-display: block; @@ -25,11 +25,8 @@ -moz-osx-font-smoothing: grayscale; } -.icon-crown:before { - content: "\e934"; -} -.icon-book-pencil:before { - content: "\e965"; +.icon-bar-chart:before { + content: "\e913"; } .icon-influx-logo:before { content: "\e900"; @@ -89,107 +86,110 @@ content: "\e912"; } .icon-heart1:before { - content: "\e913"; -} -.icon-settings:before { content: "\e914"; } -.icon-zoom-in:before { +.icon-settings:before { content: "\e915"; } -.icon-zoom-out:before { +.icon-zoom-in:before { content: "\e916"; } -.icon-chevron-down:before { +.icon-zoom-out:before { content: "\e917"; } -.icon-chevron-left:before { +.icon-chevron-down:before { content: "\e918"; } -.icon-chevron-right:before { +.icon-chevron-left:before { content: "\e919"; } -.icon-chevron-up:before { +.icon-chevron-right:before { content: "\e91a"; } -.icon-menu:before { +.icon-chevron-up:before { content: "\e91b"; } -.icon-download:before { +.icon-menu:before { content: "\e91c"; } -.icon-minus:before { +.icon-download:before { content: "\e91d"; } -.icon-plus:before { +.icon-minus:before { content: "\e91e"; } -.icon-add-cell:before { +.icon-plus:before { content: "\e91f"; } -.icon-alert:before { +.icon-add-cell:before { content: "\e920"; } -.icon-calendar:before { +.icon-alert:before { content: "\e921"; } -.icon-checkmark:before { +.icon-calendar:before { content: "\e922"; } -.icon-cog-thick:before { +.icon-checkmark:before { content: "\e923"; } -.icon-dashboards:before { +.icon-cog-thick:before { content: "\e924"; } -.icon-data-explorer:before { +.icon-dashboards:before { content: "\e925"; } -.icon-ui-download:before { +.icon-data-explorer:before { content: "\e926"; } -.icon-duplicate:before { +.icon-ui-download:before { content: "\e927"; } -.icon-export:before { +.icon-duplicate:before { content: "\e928"; } -.icon-fullscreen:before { +.icon-export:before { content: "\e929"; } -.icon-influx-icon:before { +.icon-fullscreen:before { content: "\e92a"; } -.icon-note:before { +.icon-influx-icon:before { content: "\e92b"; } -.icon-organizations:before { +.icon-note:before { content: "\e92c"; } -.icon-pause:before { +.icon-organizations:before { content: "\e92d"; } -.icon-pencil:before { +.icon-pause:before { content: "\e92e"; } -.icon-play:before { +.icon-pencil:before { content: "\e92f"; } -.icon-ui-plus:before { +.icon-play:before { content: "\e930"; } -.icon-refresh:before { +.icon-ui-plus:before { content: "\e931"; } -.icon-remove:before { +.icon-refresh:before { content: "\e932"; } -.icon-alert-circle:before { +.icon-remove:before { content: "\e933"; } -.icon-trash:before { +.icon-alert-circle:before { + content: "\e934"; +} +.icon-crown:before { content: "\e935"; } +.icon-trash:before { + content: "\e936"; +} .icon-triangle:before { content: "\e937"; } @@ -232,6 +232,9 @@ .icon-eye-open:before { content: "\e957"; } +.icon-book-pencil:before { + content: "\e965"; +} .icon-heart:before { content: "\e9da"; } diff --git a/assets/text/api/v1-compat/auth/cloud/basic-auth.js b/assets/text/api/v1-compat/auth/cloud/basic-auth.js new file mode 100644 index 000000000..c500efe54 --- /dev/null +++ b/assets/text/api/v1-compat/auth/cloud/basic-auth.js @@ -0,0 +1,36 @@ +/** + * Use an InfluxDB Cloud username and token + * with Basic Authentication + * to query the InfluxDB 1.x compatibility API + */ + +const https = require('https'); +const querystring = require('querystring'); + +function queryWithUsername() { + const queryparams = { + db: 'mydb', + q: 'SELECT * FROM cpu_usage', + }; + + const options = { + host: 'localhost:8086', + path: '/query?' + querystring.stringify(queryparams), + auth: 'exampleuser@influxdata.com:YourAuthToken', + headers: { + 'Content-type': 'application/json' + }, + }; + + const request = https.get(options, (response) => { + let rawData = ''; + response.on('data', () => { + response.on('data', (chunk) => { rawData += chunk; }); + }) + response.on('end', () => { + console.log(rawData); + }) + }); + + request.end(); +} diff --git a/assets/text/api/v1-compat/auth/cloud/basic-auth.sh b/assets/text/api/v1-compat/auth/cloud/basic-auth.sh new file mode 100644 index 000000000..53d133634 --- /dev/null +++ b/assets/text/api/v1-compat/auth/cloud/basic-auth.sh @@ -0,0 +1,16 @@ +####################################### +# Use an InfluxDB 1.x compatible username +# and password with Basic Authentication +# to query the InfluxDB 1.x compatibility API +####################################### +# Use default retention policy +####################################### +# Use the --user option with `--user :` syntax +# or the `--user ` interactive syntax to ensure your credentials are +# encoded in the header. +####################################### + +curl --get "http://localhost:8086/query" \ + --user "OneDotXUsername":"YourAuthToken" \ + --data-urlencode "db=mydb" \ + --data-urlencode "q=SELECT * FROM cpu_usage" diff --git a/assets/text/api/v1-compat/auth/oss/basic-auth.js b/assets/text/api/v1-compat/auth/oss/basic-auth.js new file mode 100644 index 000000000..cde4d9191 --- /dev/null +++ b/assets/text/api/v1-compat/auth/oss/basic-auth.js @@ -0,0 +1,37 @@ +/** + * Use an InfluxDB 1.x compatible username and password + * to query the InfluxDB 1.x compatibility API + * + * Use Basic authentication + */ + +const https = require('https'); +const querystring = require('querystring'); + +function queryWithUsername() { + const queryparams = { + db: 'mydb', + q: 'SELECT * FROM cpu_usage', + }; + + const options = { + host: 'localhost:8086', + path: '/query?' + querystring.stringify(queryparams), + auth: 'OneDotXUsername:yourPasswordOrToken', + headers: { + 'Content-type': 'application/json' + }, + }; + + const request = https.get(options, (response) => { + let rawData = ''; + response.on('data', () => { + response.on('data', (chunk) => { rawData += chunk; }); + }) + response.on('end', () => { + console.log(rawData); + }) + }); + + request.end(); +} diff --git a/assets/text/api/v1-compat/auth/oss/basic-auth.sh b/assets/text/api/v1-compat/auth/oss/basic-auth.sh new file mode 100644 index 000000000..d431ec1da --- /dev/null +++ b/assets/text/api/v1-compat/auth/oss/basic-auth.sh @@ -0,0 +1,16 @@ +####################################### +# Use an InfluxDB 1.x compatible username +# and password with Basic Authentication +# to query the InfluxDB 1.x compatibility API +####################################### +# Use default retention policy +####################################### +# Use the --user option with `--user :` syntax +# or the `--user ` interactive syntax to ensure your credentials are +# encoded in the header. +####################################### + +curl --get "http://localhost:8086/query" \ + --user "OneDotXUsername":"yourPasswordOrToken" \ + --data-urlencode "db=mydb" \ + --data-urlencode "q=SELECT * FROM cpu_usage" diff --git a/assets/text/api/v1-compat/auth/oss/querystring-auth.js b/assets/text/api/v1-compat/auth/oss/querystring-auth.js new file mode 100644 index 000000000..b0ac8c405 --- /dev/null +++ b/assets/text/api/v1-compat/auth/oss/querystring-auth.js @@ -0,0 +1,38 @@ +/** + * Use an InfluxDB 1.x compatible username and password + * to query the InfluxDB 1.x compatibility API + * + * Use authentication query parameters: + * ?u=&p= + * + * Use default retention policy. + */ + +const https = require('https'); +const querystring = require('querystring'); + +function queryWithToken() { + const queryparams = { + db: 'mydb', + q: 'SELECT * FROM cpu_usage', + u: 'OneDotXUsername', + p: 'yourPasswordOrToken' + }; + + const options = { + host: 'localhost:8086', + path: "/query?" + querystring.stringify(queryparams) + }; + + const request = https.get(options, (response) => { + let rawData = ''; + response.on('data', () => { + response.on('data', (chunk) => { rawData += chunk; }); + }) + response.on('end', () => { + console.log(rawData); + }) + }); + + request.end(); +} diff --git a/assets/text/api/v1-compat/auth/oss/querystring-auth.sh b/assets/text/api/v1-compat/auth/oss/querystring-auth.sh new file mode 100644 index 000000000..a59f76f63 --- /dev/null +++ b/assets/text/api/v1-compat/auth/oss/querystring-auth.sh @@ -0,0 +1,14 @@ +####################################### +# Use an InfluxDB 1.x compatible username and password +# to query the InfluxDB 1.x compatibility API +####################################### +# Use authentication query parameters: +# ?u=&p= +# Use default retention policy. +####################################### + +curl --get "http://localhost:8086/query" \ + --data-urlencode "u=OneDotXUsername" \ + --data-urlencode "p=yourPasswordOrToken" \ + --data-urlencode "db=mydb" \ + --data-urlencode "q=SELECT * FROM cpu_usage" diff --git a/assets/text/api/v1-compat/auth/oss/token-auth.js b/assets/text/api/v1-compat/auth/oss/token-auth.js new file mode 100644 index 000000000..0a9bdd22f --- /dev/null +++ b/assets/text/api/v1-compat/auth/oss/token-auth.js @@ -0,0 +1,35 @@ +/** + * Use a token in the Authorization header + * to query the InfluxDB 1.x compatibility API + */ + +const https = require('https'); +const querystring = require('querystring'); + +function queryWithToken() { + const queryparams = { + db: 'mydb', + q: 'SELECT * FROM cpu_usage', + }; + + const options = { + host: 'localhost:8086', + path: "/query?" + querystring.stringify(queryparams), + headers: { + 'Authorization': 'Token YourAuthToken', + 'Content-type': 'application/json' + }, + }; + + const request = https.get(options, (response) => { + let rawData = ''; + response.on('data', () => { + response.on('data', (chunk) => { rawData += chunk; }); + }) + response.on('end', () => { + console.log(rawData); + }) + }); + + request.end(); +} diff --git a/assets/text/api/v1-compat/auth/oss/token-auth.sh b/assets/text/api/v1-compat/auth/oss/token-auth.sh new file mode 100644 index 000000000..1fe60eb0e --- /dev/null +++ b/assets/text/api/v1-compat/auth/oss/token-auth.sh @@ -0,0 +1,10 @@ +####################################### +# Use a token in the Authorization header +# to query the InfluxDB 1.x compatibility API +####################################### + +curl --get "http://localhost:8086" \ + --header "Authorization: Token YourAuthToken" \ + --header 'Content-type: application/json' \ + --data-urlencode "db=mydb" \ + --data-urlencode "q=SELECT * FROM cpu_usage" diff --git a/assets/text/bucket-schema/bucket-schema-columns.sh b/assets/text/bucket-schema/bucket-schema-columns.sh new file mode 100644 index 000000000..905e87330 --- /dev/null +++ b/assets/text/bucket-schema/bucket-schema-columns.sh @@ -0,0 +1,6 @@ +[ +{"name": "time", "type": "timestamp"}, +{"name": "host", "type": "tag"}, +{"name": "usage_user", "type": "field", "dataType": "float"}, +{"name": "usage_system", "type": "field", "dataType": "float"} +] diff --git a/assets/text/bucket-schema/bucket-schema-type.sh b/assets/text/bucket-schema/bucket-schema-type.sh new file mode 100644 index 000000000..02a07166c --- /dev/null +++ b/assets/text/bucket-schema/bucket-schema-type.sh @@ -0,0 +1,3 @@ +influx bucket create \ + --name my_schema_bucket \ + --schema-type explicit diff --git a/assets/text/bucket-schema/sensor.ndjson b/assets/text/bucket-schema/sensor.ndjson new file mode 100644 index 000000000..91d6fb474 --- /dev/null +++ b/assets/text/bucket-schema/sensor.ndjson @@ -0,0 +1,5 @@ +{"name": "time", "type": "timestamp"} +{"name": "service", "type": "tag"} +{"name": "sensor", "type": "tag"} +{"name": "temperature", "type": "field", "dataType": "float"} +{"name": "humidity", "type": "field", "dataType": "float"} diff --git a/assets/text/bucket-schema/usage-cpu.json b/assets/text/bucket-schema/usage-cpu.json new file mode 100644 index 000000000..6b9c48f81 --- /dev/null +++ b/assets/text/bucket-schema/usage-cpu.json @@ -0,0 +1,7 @@ +[ +{"name": "time", "type": "timestamp"}, +{"name": "service", "type": "tag"}, +{"name": "host", "type": "tag"}, +{"name": "usage_user", "type": "field", "dataType": "float"}, +{"name": "usage_system", "type": "field", "dataType": "float"} +] diff --git a/assets/text/bucket-schema/usage-resources.csv b/assets/text/bucket-schema/usage-resources.csv new file mode 100644 index 000000000..2827fd62d --- /dev/null +++ b/assets/text/bucket-schema/usage-resources.csv @@ -0,0 +1,6 @@ +name,type,data_type +time,timestamp, +host,tag, +service,tag, +fsRead,field,float +fsWrite,field,float diff --git a/content/chronograf/v1.8/guides/dashboard-template-variables.md b/content/chronograf/v1.8/guides/dashboard-template-variables.md index f590e0829..ab3aa182a 100644 --- a/content/chronograf/v1.8/guides/dashboard-template-variables.md +++ b/content/chronograf/v1.8/guides/dashboard-template-variables.md @@ -1,7 +1,8 @@ --- title: Use dashboard template variables description: > - Dashboard variables let you to alter specific components of cells' queries without having to edit the queries, making it easy to interact with your dashboard cells and explore your data. + Chronograf dashboard template variables let you update cell queries without editing queries, + making it easy to interact with your dashboard cells and explore your data. aliases: - /chronograf/v1.8/introduction/templating/ - /chronograf/v1.8/templating/ @@ -11,10 +12,10 @@ menu: parent: Guides --- -Chronograf's dashboard template variables let you update cell queries without editing queries, making it easy to interact with your dashboard cells and explore your data. +Chronograf dashboard template variables let you update cell queries without editing queries, +making it easy to interact with your dashboard cells and explore your data. - [Use template variables](#use-template-variables) -- [Quoting template variables](#quoting-template-variables) - [Predefined template variables](#predefined-template-variables) - [Create custom template variables](#create-custom-template-variables) - [Template variable types](#template-variable-types) @@ -23,21 +24,27 @@ Chronograf's dashboard template variables let you update cell queries without ed ## Use template variables -When creating Chronograf dashboards, use template variables in cell queries and titles. +When creating Chronograf dashboards, use either [predefined template variables](#predefined-template-variables) or [custom template variables](#create-custom-template-variables) in your cell queries and titles. +After you set up variables, variables are available to select in your dashboard user interface (UI). -In the query, surround template variables names with colons (`:`) as follows: +- [Use template variables in cell queries](#use-template-variables-in-cell-queries) + - [InfluxQL](#influxql) + - [Flux](#flux) +- [Use template variables in cell titles](#use-template-variables-in-cell-titles) + +![Use template variables](/img/chronograf/1-6-template-vars-use.gif) + +### Use template variables in cell queries +Both InfluxQL and Flux support template variables. + +#### InfluxQL +In an InfluxQL query, surround template variables names with colons (`:`) as follows: ```sql SELECT :variable_name: FROM "telegraf"."autogen".:measurement: WHERE time < :dashboardTime: ``` -Use either [predefined template variables](#predefined-template-variables) -or [custom template variables](#create-custom-template-variables). -After you set up variables, variables are available to select in your dashboard user-interface (UI). - -![Using template variables](/img/chronograf/1-6-template-vars-use.gif) - -## Quoting template variables +##### Quoting template variables in InfluxQL For **predefined meta queries** such as "Field Keys" and "Tag Values", **do not add quotes** (single or double) to your queries. Chronograf will add quotes as follows: @@ -45,7 +52,7 @@ For **predefined meta queries** such as "Field Keys" and "Tag Values", **do not SELECT :variable_name: FROM "telegraf"."autogen".:measurement: WHERE time < :dashboardTime: ``` -For **custom queries**, **CSV**, or **map queries**, quote the values in the query in accordance with standard [InfluxQL](/influxdb/v1.8/query_language/) syntax as follows: +For **custom queries**, **CSV**, or **map queries**, quote the values in the query following standard [InfluxQL](/{{< latest "influxdb" "v1" >}}/query_language/) syntax: - For numerical values, **do not quote**. - For string values, choose to quote the values in the variable definition (or not). See [String examples](#string-examples) below. @@ -53,14 +60,14 @@ For **custom queries**, **CSV**, or **map queries**, quote the values in the que {{% note %}} **Tips for quoting strings:** - When using custom meta queries that return strings, typically, you quote the variable values when using them in a dashboard query, given InfluxQL results are returned without quotes. - - If you are using template variable strings in regular expression syntax (when using quotes may cause query syntax errors), the flexibility in query quoting methods is particularly useful. {{% /note %}} -#### String examples +##### String examples Add single quotes when you define template variables, or in your queries, but not both. -##### Example 1: Add single quotes in variable definition + +###### Add single quotes in variable definition If you define a custom CSV variable named `host` using single quotes: @@ -72,10 +79,10 @@ Do not include quotes in your query: ```sql SELECT mean("usage_user") AS "mean_usage_user" FROM "telegraf"."autogen"."cpu" -WHERE "host" = :host: and time > :dashboardTime +WHERE "host" = :host: and time > :dashboardTime: ``` -##### Example 2: Add single quotes in query +###### Add single quotes in query If you define a custom CSV variable named `host` without quotes: @@ -90,14 +97,52 @@ SELECT mean("usage_user") AS "mean_usage_user" FROM "telegraf"."autogen"."cpu" WHERE "host" = ':host:' and time > :dashboardTime ``` +#### Flux +To use a template variable in a Flux query, include the variable key in your query. + +{{% note %}} +Flux dashboard cell queries don't support **custom template variables**, but do +support [predefined template variables](#predefined-template-variables). +{{% /note %}} + +```js +from(bucket: "example-bucket") + |> range(start: dashboardTime, stop: dashboardTime) + |> filter(fn: (r) => r._field == "example-field") + |> aggregateWindow(every: autoInterval, fn: last) +``` + +### Use template variables in cell titles +To dynamically change the title of a dashboard cell, +use the `:variable-name:` syntax. + +For example, a variable named `field` with a value of `temp` +and a variable named `location` with a value of `San Antonio`, use the following syntax: + +``` +:temp: data for :location: +``` + +Displays as: + +{{< img-hd src= "/img/chronograf/1-9-template-var-title.png" alt="Use template variables in cell titles" />}} + ## Predefined template variables -Chronograf includes predefined template variables controlled by elements in the Chrongraf UI. -These template variables can be used in any of your cells' queries. +Chronograf includes predefined template variables controlled by elements in the Chronograf UI. +Use predefined template variables in your cell queries. -[`:dashboardTime:`](#dashboardtime) -[`:upperDashboardTime:`](#upperdashboardtime) -[`:interval:`](#interval) +InfluxQL and Flux include their own sets of predefined template variables: + +{{< tabs-wrapper >}} +{{% tabs %}} +[InfluxQL](#) +[Flux](#) +{{% /tabs %}} +{{% tab-content %}} +- [`:dashboardTime:`](#dashboardtime) +- [`:upperDashboardTime:`](#upperdashboardtime) +- [`:interval:`](#interval) ### dashboardTime The `:dashboardTime:` template variable is controlled by the "time" dropdown in your Chronograf dashboard. @@ -113,9 +158,11 @@ FROM "telegraf".."cpu" WHERE time > :dashboardTime: ``` -> In order to use the date picker to specify a particular time range in the past -> which does not include "now", the query should be constructed using `:dashboardTime:` -> as the lower limit and [`:upperDashboardTime:`](#upperdashboardtime) as the upper limit. +{{% note %}} +To use the date picker to specify a particular time range in the past +which does not include "now", construct the query using `:dashboardTime:` +as the start time and [`:upperDashboardTime:`](#upperdashboardtime) as the stop time. +{{% /note %}} ### upperDashboardTime The `:upperDashboardTime:` template variable is defined by the upper time limit specified using the date picker. @@ -144,10 +191,61 @@ FROM "telegraf".."cpu" WHERE time > :dashboardtime: GROUP BY time(:interval:) ``` +{{% /tab-content %}} +{{% tab-content %}} +- [`dashboardTime`](#dashboardtime-flux) +- [`upperDashboardTime`](#upperdashboardtime-flux) +- [`autoInterval`](#autointerval) + +### dashboardTime {id="dashboardtime-flux"} + +The `dashboardTime` template variable is controlled by the "time" dropdown in your Chronograf dashboard. + +Dashboard time selector + +If using relative time, this variable represents the time offset specified in the dropdown (-5m, -15m, -30m, etc.) and assumes time is relative to "now". +If using absolute time defined by the date picker, `dashboardTime` is populated with the start time. + +```js +from(bucket: "telegraf/autogen") + |> range(start: dashboardTime) + |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") +``` + +{{% note %}} +To use the date picker to specify a time range in the past without "now", use `dashboardTime` as the lower limit and +[`upperDashboardTime`](#upperdashboardtime-flux) as the upper limit. +{{% /note %}} + +### upperDashboardTime{id="upperdashboardtime-flux"} +The `upperDashboardTime` template variable is defined by the stop time specified using the date picker. + +Dashboard date picker + +For relative time frames, this variable inherits `now()`. For absolute time frames, this variable inherits the stop time. + +```js +from(bucket: "telegraf/autogen") + |> range(start: dashboardTime, stop: upperDashboardTime) + |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") +``` + +### autoInterval +The `autoInterval` template variable is controlled by the display width of the +dashboard cell and is calculated by the duration of time that each pixel covers. +Use the `autoInterval` variable to limit downsample data to display a maximum of one point per pixel. + +```js +from(bucket: "telegraf/autogen") + |> range(start: dashboardTime) + |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") + |> aggregateWindow(every: autoInterval, fn: mean) +``` +{{% /tab-content %}} +{{< /tabs-wrapper >}} ## Create custom template variables -Template variables are essentially an array of potential values used to populate parts of your cells' queries. Chronograf lets you create custom template variables powered by meta queries or CSV uploads that return an array of possible values. To create a template variable: @@ -167,15 +265,15 @@ and a dropdown for the variable will be included at the top of your dashboard. ## Template Variable Types Chronograf supports the following template variable types: -[Databases](#databases) -[Measurements](#measurements) -[Field Keys](#field-keys) -[Tag Keys](#tag-keys) -[Tag Values](#tag-values) -[CSV](#csv) -[Map](#map) -[Custom Meta Query](#custom-meta-query) -[Text](#text) +- [Databases](#databases) +- [Measurements](#measurements) +- [Field Keys](#field-keys) +- [Tag Keys](#tag-keys) +- [Tag Values](#tag-values) +- [CSV](#csv) +- [Map](#map) +- [Custom Meta Query](#custom-meta-query) +- [Text](#text) ### Databases Database template variables allow you to select from multiple target [databases](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#database). @@ -193,7 +291,8 @@ SELECT "purchases" FROM :databaseVar:."autogen"."customers" ``` #### Database variable use cases -Database template variables are good when visualizing multiple databases with similar or identical data structures. They allow you to quickly switch between visualizations for each of your databases. +Use database template variables when visualizing multiple databases with similar or identical data structures. +Variables let you quickly switch between visualizations for each of your databases. ### Measurements Vary the target [measurement](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#measurement). @@ -285,11 +384,14 @@ value3 value4 ``` -> Since string field values [require single quotes in InfluxQL](/{{< latest "influxdb" "v1" >}}/troubleshooting/frequently-asked-questions/#when-should-i-single-quote-and-when-should-i-double-quote-in-queries), string values should be wrapped in single quotes. +{{% note %}} +String field values [require single quotes in InfluxQL](/{{< latest "influxdb" "v1" >}}/troubleshooting/frequently-asked-questions/#when-should-i-single-quote-and-when-should-i-double-quote-in-queries), +so wrap string values in single quotes. ->```csv +```csv 'string1','string2','string3','string4' ``` +{{% /note %}} _**Example CSV variable in a cell query**_ ```sql @@ -315,14 +417,16 @@ key4,value4 Map variable dropdown -> If values are meant to be used as string field values, wrap them in single quote ([required by InfluxQL](/{{< latest "influxdb" "v1" >}}/troubleshooting/frequently-asked-questions/#when-should-i-single-quote-and-when-should-i-double-quote-in-queries)). This only pertains to values. String keys do not matter. +{{% note %}} +Wrap string field values in single quotes ([required by InfluxQL](/{{< latest "influxdb" "v1" >}}/troubleshooting/frequently-asked-questions/#when-should-i-single-quote-and-when-should-i-double-quote-in-queries)) (string field values only; string keys do not require quotes). ->```csv +```csv key1,'value1' key2,'value2' key3,'value3' key4,'value4' ``` +{{% /note %}} _**Example Map variable in a cell query**_ ```sql @@ -345,7 +449,8 @@ The customer names would populate your template variable dropdown rather than th ### Custom Meta Query Vary part of a query with a customized meta query that pulls a specific array of values from InfluxDB. -These variables let you pull a highly customized array of potential values and offer advanced functionality such as [filtering values based on other template variables](#filter-template-variables-with-other-template-variables). +Custom meta query variables let you pull a highly customized array of potential values and offer +advanced functionality such as [filtering values based on other template variables](#filter-template-variables-with-other-template-variables). Custom meta query @@ -355,7 +460,7 @@ SELECT "purchases" FROM "animals"."autogen"."customers" WHERE "customer" = :cust ``` #### Custom meta query variable use cases -Custom meta query template variables should be used any time you are pulling values from InfluxDB, but the pre-canned template variable types aren't able to return the desired list of values. +Use custom InfluxQL meta query template variables when predefined template variable types aren't able to return the values you want. ### Text Vary a part of a query with a single string of text. @@ -373,21 +478,21 @@ as well as many other parameters that control the display of graphs in your dash These names are either [predefined variables](#predefined-template-variables) or would conflict with existing URL query parameters. -`:database:` -`:measurement:` -`:dashboardTime:` -`:upperDashboardTime:` -`:interval:` -`:upper:` -`:lower:` -`:zoomedUpper:` -`:zoomedLower:` -`refreshRate:` +- `:database:` +- `:measurement:` +- `:dashboardTime:` +- `:upperDashboardTime:` +- `:interval:` +- `:upper:` +- `:lower:` +- `:zoomedUpper:` +- `:zoomedLower:` +- `:refreshRate:` ## Advanced template variable usage ### Filter template variables with other template variables -[Custom meta query template variables](#custom-meta-query) allow you to filter the array of potential variable values using other existing template variables. +[Custom meta query template variables](#influxQL-meta-query) let you filter the array of potential variable values using other existing template variables. For example, let's say you want to list all the field keys associated with a measurement, but want to be able to change the measurement: @@ -395,7 +500,7 @@ For example, let's say you want to list all the field keys associated with a mea measurementVar -2. Create a template variable named `:fieldKey:` that uses the [custom meta query](#custom-meta-query) variable type. +2. Create a template variable named `:fieldKey:` that uses the [InfluxQL meta query](#influxql-meta-query) variable type. The following meta query pulls a list of field keys based on the existing `:measurementVar:` template variable. ```sql @@ -404,7 +509,7 @@ The following meta query pulls a list of field keys based on the existing `:meas fieldKey -3. Create a new dashboard cell that uses the `:fieldKey:` and `:measurementVar` template variables in its query. +3. Create a new dashboard cell that uses the `fieldKey` and `measurementVar` template variables in its query. ```sql SELECT :fieldKey: FROM "telegraf"..:measurementVar: WHERE time > :dashboardTime: @@ -418,8 +523,8 @@ The resulting dashboard will work like this: Chronograf uses URL query parameters (also known as query string parameters) to set both display options and template variables in the URL. This makes it easy to share links to dashboards so they load in a specific state with specific template variable values selected. -URL query parameters are appeneded to the end of the URL with a question mark (`?`) indicating beginning of query parameters. -Multiple query paramemters can be chained together using an ampersand (`&`). +URL query parameters are appended to the end of the URL with a question mark (`?`) indicating the beginning of query parameters. +Chain multiple query parameters together using an ampersand (`&`). To declare a template variable or a date range as a URL query parameter, it must follow the following pattern: @@ -445,8 +550,10 @@ Name of the template variable. `variableValue` Value of the template variable. -> Whenever template variables are modified in the dashboard, the corresponding -> URL query parameters are automatically updated. +{{% note %}} +When template variables are modified in the dashboard, the corresponding +URL query parameters are automatically updated. +{{% /note %}} #### Example template variable query parameter ``` diff --git a/content/chronograf/v1.9/_index.md b/content/chronograf/v1.9/_index.md new file mode 100644 index 000000000..41c27fd5a --- /dev/null +++ b/content/chronograf/v1.9/_index.md @@ -0,0 +1,58 @@ +--- +title: Chronograf 1.9 documentation +description: > + Chronograf is InfluxData's open source web application. + Use Chronograf with the other components of the TICK stack to visualize your + monitoring data and easily create alerting and automation rules. +menu: + chronograf_1_9: + name: Chronograf v1.9 +weight: 1 +--- + +Chronograf is InfluxData's open source web application. +Use Chronograf with the other components of the [TICK stack](https://www.influxdata.com/products/) to visualize your monitoring data and easily create alerting and automation rules. + +![Chronograf Collage](/img/chronograf/1-6-chronograf-collage.png) + +## Key features + +### Infrastructure monitoring + +* View all hosts and their statuses in your infrastructure +* View the configured applications on each host +* Monitor your applications with Chronograf's [pre-created dashboards](/chronograf/v1.9/guides/using-precreated-dashboards/) + +### Alert management + +Chronograf offers a UI for [Kapacitor](https://github.com/influxdata/kapacitor), InfluxData's data processing framework for creating alerts, running ETL jobs, and detecting anomalies in your data. + +* Generate threshold, relative, and deadman alerts on your data +* Easily enable and disable existing alert rules +* View all active alerts on an alert dashboard +* Send alerts to the supported event handlers, including Slack, PagerDuty, HipChat, and [more](/chronograf/v1.9/guides/configuring-alert-endpoints/) + +### Data visualization + +* Monitor your application data with Chronograf's [pre-created dashboards](/chronograf/v1.9/guides/using-precreated-dashboards/) +* Create your own customized dashboards complete with various graph types and [template variables](/chronograf/v1.9/guides/dashboard-template-variables/) +* Investigate your data with Chronograf's data explorer and query templates + +### Database management + +* Create and delete databases and retention policies +* View currently-running queries and stop inefficient queries from overloading your system +* Create, delete, and assign permissions to users (Chronograf supports [InfluxDB OSS](/{{< latest "influxdb" "v1" >}}/administration/authentication_and_authorization/#authorization) and InfluxDB Enterprise user management) + + +### Multi-organization and multi-user support + +{{% note %}} +**Note:** To use this feature, OAuth 2.0 authentication must be configured. +Once configured, the Chronograf Admin tab on the Admin menu is visible. +For details, see [Managing Chronograf security](/chronograf/v1.9/administration/managing-security/). +{{% /note %}} + +* Create organizations and assign users to those organizations +* Restrict access to administrative functions +* Allow users to set up and maintain unique dashboards for their organizations diff --git a/content/chronograf/v1.9/about_the_project/_index.md b/content/chronograf/v1.9/about_the_project/_index.md new file mode 100644 index 000000000..312807ac7 --- /dev/null +++ b/content/chronograf/v1.9/about_the_project/_index.md @@ -0,0 +1,45 @@ +--- +title: About the Chronograf project +description: Learn about Chronograf, the user interface (UI) for InfluxDB. +menu: + chronograf_1_9: + name: About the project + weight: 10 +--- +Chronograf is the user interface component of the [InfluxData time series platform](https://www.influxdata.com/time-series-platform/). It makes the monitoring and alerting for your infrastructure easy to setup and maintain. It is simple to use and includes templates and libraries to allow you to rapidly build dashboards with realtime visualizations of your data. + +Follow the links below for more information. + +{{< children >}} + +Chronograf is released under the GNU Affero General Public License. This Free Software Foundation license is fairly new, +and differs from the more widely known and understood GPL. + +Our goal with using AGPL is to preserve the concept of copyleft with Chronograf. +With traditional GPL, copyleft was associated with the concept of distribution of software. +The problem is that nowadays, distribution of software is rare: things tend to run in the cloud. AGPL fixes this “loophole” +in GPL by saying that if you use the software over a network, you are bound by the copyleft. Other than that, +the license is virtually the same as GPL v3. + +To say this another way: if you modify the core source code of Chronograf, the goal is that you have to contribute +those modifications back to the community. + +Note however that it is NOT required that your dashboards and alerts created by using Chronograf be published. +The copyleft applies only to the source code of Chronograf itself. + +If this explanation isn't good enough for you and your use case, we dual license Chronograf under our +[standard commercial license](https://www.influxdata.com/legal/slsa/). + +[Contact sales for more information](https://www.influxdata.com/contact-sales/). + +## Third Party Software + +InfluxData products contain third party software, which means the copyrighted, patented, or otherwise legally protected +software of third parties that is incorporated in InfluxData products. + +Third party suppliers make no representation nor warranty with respect to such third party software or any portion thereof. +Third party suppliers assume no liability for any claim that might arise with respect to such third party software, +nor for a customer’s use of or inability to use the third party software. + +The [list of third party software components, including references to associated license and other materials](https://github.com/influxdata/chronograf/blob/master/LICENSE_OF_DEPENDENCIES.md), +is maintained on a version by version basis. diff --git a/content/chronograf/v1.9/about_the_project/cla.md b/content/chronograf/v1.9/about_the_project/cla.md new file mode 100644 index 000000000..6e52e85d0 --- /dev/null +++ b/content/chronograf/v1.9/about_the_project/cla.md @@ -0,0 +1,12 @@ +--- +title: InfluxData Contributor License Agreement (CLA) +description: > + Before contributing to the Chronograf project, submit the InfluxData Contributor License Agreement. +menu: + chronograf_1_9: + weight: 30 + parent: About the project + url: https://www.influxdata.com/legal/cla/ +--- + +Before you can contribute to the Chronograf project, you need to submit the [InfluxData Contributor License Agreement (CLA)](https://www.influxdata.com/legal/cla/) available on the InfluxData main site. diff --git a/content/chronograf/v1.9/about_the_project/contributing.md b/content/chronograf/v1.9/about_the_project/contributing.md new file mode 100644 index 000000000..506455300 --- /dev/null +++ b/content/chronograf/v1.9/about_the_project/contributing.md @@ -0,0 +1,12 @@ +--- +title: Contribute to Chronograf +description: Contribute to the Chronograf project. +menu: + chronograf_1_9: + name: Contribute + weight: 20 + parent: About the project + url: https://github.com/influxdata/chronograf/blob/master/CONTRIBUTING.md +--- + +See [Contributing to Chronograf](https://github.com/influxdata/chronograf/blob/master/CONTRIBUTING.md) in the Chronograf GitHub project to learn how you can contribute to the Chronograf project. diff --git a/content/chronograf/v1.9/about_the_project/licenses.md b/content/chronograf/v1.9/about_the_project/licenses.md new file mode 100644 index 000000000..c5d14d69a --- /dev/null +++ b/content/chronograf/v1.9/about_the_project/licenses.md @@ -0,0 +1,12 @@ +--- +title: Open source license for Chronograf +description: Find the open source license for Chronograf. +menu: + chronograf_1_9: + Name: Open source license + weight: 40 + parent: About the project + url: https://github.com/influxdata/chronograf/blob/master/LICENSE +--- + +The [open source license for Chronograf](https://github.com/influxdata/chronograf/blob/master/LICENSE) is available in the Chronograf GitHub project. diff --git a/content/chronograf/v1.9/about_the_project/release-notes-changelog.md b/content/chronograf/v1.9/about_the_project/release-notes-changelog.md new file mode 100644 index 000000000..c5fcc12be --- /dev/null +++ b/content/chronograf/v1.9/about_the_project/release-notes-changelog.md @@ -0,0 +1,1218 @@ +--- +title: Chronograf 1.9 release notes +description: Important changes and what's new in each version of Chronograf. +menu: + chronograf_1_9: + name: Release notes + weight: 10 + parent: About the project +--- + +## v1.9.0 [2021-06-25] + +{{% warn %}} +### Breaking Changes + +#### OAuth PKCE +Add [OAuth PKCE (RFC7636)](https://oauth.net/2/pkce/) to OAuth integrations in Chronograf. +PKCE mitigates the threat of the authorization code being intercepted during the OAuth token exchange. +Google, Azure, Octa, Auth0, Gitlab, and others already support OAuth PKCE. +Enabling PKCE should have no effect on integrations with services that don't +support it yet (such as Github or Bitbucket). +To disable PKCE, set the `OAUTH_NO_PKCE` environment variable to `=`true` or +include the `--oauth-no-pkce` flag when starting `chronograf`. +{{% /warn %}} + +### Features +- Support data migrations to ETCD over HTTPS. +- Set trusted CA certificates for ETCD connections. +- Configure new Kapacitor alert endpoints in the UI. + - [ServiceNow](/chronograf/v1.9/guides/configuring-alert-endpoints/#servicenow) + - [BigPanda](/chronograf/v1.9/guides/configuring-alert-endpoints/#bigpanda) + - [Teams](/chronograf/v1.9/guides/configuring-alert-endpoints/#teams) + - [Zenoss](/chronograf/v1.9/guides/configuring-alert-endpoints/#zenoss) +- Remove HipChat alert endpoints. +- [Show or hide the log status histogram](/chronograf/v1.9/guides/analyzing-logs/#show-or-hide-the-log-status-histogram) + in the Log Viewer. +- Add the following meta query templates in the Data Explorer: + - `SHOW FIELD KEYS` + - `SHOW SUBSCRIPTIONS` + - `SHOW QUERIES` + - `SHOW GRANTS` + - `SHOW SHARDS` + - `SHOW SHARD GROUPS` + - `EXPLAIN` + - `EXPLAIN ANALYZE` +- Flux improvements and additional functionality: + - Add predefined and custom dashboard template variables to Flux query execution. + Flux queries include a `v` record with a key value pair for each variable. + - [Define template variables with Flux](/chronograf/v1.9/guides/dashboard-template-variables/#flux-query). + - Add [Kapacitor Flux tasks](/chronograf/v1.9/guides/advanced-kapacitor/#manage-kapacitor-flux-tasks) + on the Manage Tasks page (read only). + - Provide documentation link when Flux is not enabled in InfluxDB 1.8+. + - Write to buckets when Flux mode is selected. +- Filter fields in the Query Builder. +- [Select write precision](/chronograf/v1.9/guides/write-to-influxdb/#upload-line-protocol-through-the-chronograf-ui) + when writing data through the Chronograf UI. +- Support [GitHub Enterprise in the existing GitHub OAuth integration](/chronograf/v1.9/administration/managing-security/#configure-github-authentication). +- Update the Queries page in the InfluxDB Admin section of the UI to include the following: + - By default, sort queries by execution time in descending order. + - Sort queries by execution time or database. + - Include database count in the Queries page title. + - Select the refresh interval of the Queries page. +- [Set up InfluxDB Cloud and InfluxDB OSS 2.x connections](/chronograf/v1.9/administration/config-options/#influxdb-connection-options) + with the `chronograf` CLI. +- Add [custom auto-refresh intervals](/chronograf/v1.9/administration/config-options/#--custom-auto-refresh). +- Send multiple queries to a dashboard. +- Add macOS arm64 builds. + +### Bug Fixes +- Open alert handler configuration pages with URL hash. +- Omit errors during line visualizations of meta query results. +- Delete log stream when TICKscript editor page is closed. +- Generate correct Flux property expressions. +- Repair stale database list in Log Viewer. +- Exclude `_start` and `_stop` columns in Flux query results. +- Improve server type detection in the Connection Wizard. +- Add error handling to Alert History page. +- Don't fetch tag values when a measurement doesn't contain tags. +- Filter out roles with unknown organization references. +- Detect Flux support in the Flux proxy. +- Manage execution status per individual query. +- Parse exported dashboards in a resources directory. +- Enforce unique dashboard template variable names. +- Don't modify queries passed to a Dashboard page using a query URL parameter. +- Fix unsafe React lifecycle functions. +- Improve communication with InfluxDB Enterprise. + +### Other +- Upgrade UI to TypeScript 4.2.2. +- Upgrade dependencies and use ESLint for TypeScript. +- Update dependency licenses. +- Upgrade Markdown renderer. +- Upgrade Go to 1.16. +- Upgrade build process to Python 3. + +## v1.8.10 [2020-02-08] + +### Features + +- Add the ability to set the active InfluxDB database and retention policy for InfluxQL commands. Now, in Chronograf Data Explorer, if you select a metaquery template (InfluxQL command) that requires you to specify an active database, such as `DROP MEASUREMENT`, `DROP SERIES FROM`, and `DELETE FROM`, the `USE` command is prepended to your InfluxQL command as follows: + +``` +USE "db_name"; DROP MEASUREMENT "measurement_name" +USE "db_name"; DROP SERIES FROM "measurement_name" WHERE "tag" = 'value' +USE "db_name"; DELETE FROM "measurement_name" WHERE "tag" = 'value' AND time < '2020-01-01' +``` + +- Add support for Bitbucket `emails` endpoint with generic OAuth. For more information, see [Bitbucket documentation](https://developer.atlassian.com/bitbucket/api/2/reference/resource/user/emails) and how to [configure Chronograf to authenticate with OAuth 2.0](/chronograf/v1.9/administration/managing-security/#configure-chronograf-to-authenticate-with-oauth-2-0). + +### Bug Fixes + +- Repair ARMv5 build. +- Upgrade to Axios 0.21.1. +- Stop async executions on unmounted LogsPage. +- Repair dashboard import to remap sources in variables. +- UI updates: + - Ignore databases that cannot be read. Now, the Admin page correctly displays all databases that the user has permissions to. + - Improve the Send to Dashboard feedback on the Data Explorer page. +- Log Viewer updates: + - Avoid endless networking loop. + - Show timestamp with full nanosecond precision. + +## v1.8.9.1 [2020-12-10] + +### Features +- Configure etcd with client TLS certificate. +- Support Flux in InfluxDB Cloud and InfluxDB OSS 2.x sources. +- Support Flux Schema Explorer in InfluxDB Cloud and InfluxDB OSS 2.x sources. +- Let users specify InfluxDB v2 authentication. +- Validate credentials before creating or updating InfluxDB sources. +- Use fully qualified bucket names when using Flux in the Data Explorer. +- Upgrade Go to 1.15.5. +- Upgrade Node.js to 14 LTS. + +### Bug Fixes +- Prevent briefly displaying "No Results" in dashboard cells upon refresh. +- Warn about unsupported queries when creating or editing alert rules. +- Use the `AND` logical operator with not-equal (`!=`) tag comparisons in generated TICKscript `where` filters. +- Disable InfluxDB admin page when administration is not possible + (while using InfluxDB Cloud or InfluxDB OSS 2.x sources). +- Use token authentication against InfluxDB Cloud and InfluxDB OSS 2.x sources. +- Avoid blank screen on Windows. +- Repair visual comparison with time variables (`:upperDashboardTime:` and `:dashboardTime:`). +- Repair possible millisecond differences in duration computation. +- Remove deprecated React SFC type. + +## v.1.8.8 [2020-11-04] + +## Features + +- Add the option to select a recovery action in the [OpsGenie2](/chronograf/v1.9/guides/configuring-alert-endpoints/#opsgenie2) configuration. + +## Bug Fixes + +- Ensure the alert rule name is correctly displayed in the Alert Rules and TICKscript lists. +- Resolve the issue that caused a truncated dashboard name. +- Ensure the TICKscript editor is scrollable in Firefox. +- Apply default timeouts in server connections to ensure a shared HTTP transport connection is used between Chronograf and InfluxDB or Kapacitor. +- Retain the selected time zone (local or UTC) in the range picker. +- Export CSV with a time column formatted according to the selected time zone (local or UTC). + +## v.1.8.7 [2020-10-06] + +{{% warn %}} +This release includes breaking changes: +TLS1.2 is now the default minimum required TLS version. If you have clients that require older TLS versions, use one of the following when starting Chronograf: + - The `--tls-min-version=1.1` option + - The `TLS_MIN_VERSION=1.1` environment variable +{{% /warn %}} + +## Features +- Allow to configure HTTP basic access authentication. +- Allow setting token-prefix in Alerta configuration. +- Make session inactivity duration configurable. +- Allow configuration of TLS ciphers and versions. + +## Bug Fixes +- Disable default dashboard auto-refresh. +- Fix to user migration. +- Add `isPresent` filter to rule TICKscript. +- Make vertical scrollbar visible when rows overflow in TableGraph. +- Upgrade `papaparse` to 5.3.0. +- Require well-formatted commit messages in pull request. +- Upgrade `node` to v12. + +## v1.8.6 [2020-08-27] + +### Features + +- Upgrade Dockerfile to use Alpine 3.12. + +### Bug Fixes + +- Escape tag values in Query Builder. +- Sort namespaces by database and retention policy. +- Make MySQL protoboard more useful by using derivatives for counter values. +- Add HTTP security headers. +- Resolve an issue that caused existing data to be overwritten when there were multiple results for a specific time. Now, all query results are successfully shown in the Table visualization. +- Resolve an issue that prevented boolean field and tag values from being displayed. Now, field and tag values are printed in TICKscript logs. + +## v1.8.5 [2020-07-08] + +### Bug Fixes + +- Fix public-url generic OAuth configuration issue. +- Fix crash when starting Chronograf built by Go 1.14 on Windows. +- Keep dashboard's table sorting stable on data refresh. +- Repair TICKscript editor scrolling on Firefox. +- Better parse Flux CSV results. +- Support `.Time.Unix` in alert message validation. +- Fix error when viewing Flux raw data after edit. +- Repair management of Kapacitor rules and TICKscripts. +- Avoid undefined error when dashboard is not ready yet. +- Fall back to point timestamp in log viewer. +- Add global functions and string trimming to alert message validation. +- Merge query results with unique column names. +- Avoid exiting presentation mode when zooming out. +- Avoid duplication of `csv.from` in functions list. + +## v1.8.4 [2020-05-01] + +### Bug Fixes + +- Fix misaligned tables when scrolling. + +## v1.8.3 [2020-04-23] + +### Bug Fixes + +- Fixed missing token subcommand. +- Incomplete OAuth configurations now throw errors listing missing components. +- Extend OAuth JWT timeout to match cookie lifespan. + +### Features + +- Added ability to ignore or verify custom OAuth certs. + + +## v1.8.2 [2020-04-13] + +### Features + +- Update to Flux v0.65.0. + +### Bug Fixes + +- Fix table rendering bug introduced in 1.8.1. + +## v1.8.1 [2020-04-06] + +{{% warn %}} **Warning:** Critical bug that impacted table rendering was introduced in 1.8.1. **Do not install this release**, install [v1.8.2](##v1-8-2-2020-04-13), which includes the features and bug fixes below. +{{% /warn %}} + +### Features + +- Add ability to directly authenticate single SuperAdmin user against the API. + +### Bug Fixes + +- Update table results to output formatted strings rather than as single-line values. +- Handle change to newsfeed data structure. + +## v1.8.0 [2020-02-19] + +### Features + +- Add Chronograf high availability (HA) configuration, including: + - Support for [`etcd`](https://etcd.io/) as an alternate backend store for multiple instances of Chronograf + - `chronoctl` tool to migrate from BoltDB to etcd + + If you're installing Chronograf for the first time, learn how to [create a new Chronograf HA configuration](/chronograf/v1.9/administration/create-high-availability/). + If you're upgrading Chronograf, learn how to [migrate your existing Chronograf configuration to HA](/chronograf/v1.9/administration/migrate-to-high-availability/). +- Add configuration option to [disable the Host List page](/chronograf/v1.9/administration/config-options/#host-page-disabled-h). +- Add ability to select a data source when [creating a template variable](/chronograf/v1.9/guides/dashboard-template-variables/#create-custom-template-variables). +- Add the `refresh` query parameter to set the dashboard auto-refresh interval (by default, 10000 milliseconds). Discover ways to [configure your dashboard](/chronograf/v1.9/guides/create-a-dashboard/#step-6-configure-your-dashboard). + +### Bug Fixes + +- The Log Viewer no longer crashes if an invalid severity is used (for example, `emreg` instead of `emerg`). Now, by default, the Log Viewer assigns the first color in the list on the Configure Log View page to an invalid severity. +- Resolve issue when using InfluxDB self-signed certificates that caused Chronograf to fail to check for Flux support. Now the source `insecureSkipVerify` setting is passed to the Flux client. +- Update `non_negative_derivative` calls to use 1s. +- Update InfluxData URL to use the correct Cross-Origin Resource Sharing (CORS) headers. + +### Miscellaneous + +* No longer support direct upgrade from Chronograf 1.3.x. If you're upgrading 1.3.x, first install 1.7.x, and then install 1.8. + +## v1.7.17 [2020-01-08] + +### Bug Fixes + +- Allow logging out when using OAuth. + +## v1.7.16 [2019-12-18] + +### Bug Fixes + +- Update the Flux schema explorer to correctly use meta queries. +- Restore blank y-axis labels by using a known y-axis label if available. +- Add `:upperDashboardTime:` to InfluxQL queries to support custom dashboard time ranges. +- Update the heuristic for matching Kapacitor TICKscripts when creating batch tasks. + +## v1.7.15 [2019-11-12] + +### Features + +- Pin to latest minor GO version; improve Docker build process. + +### Bug Fixes + +- Remove optional ID in create dashboard swagger. +- Resolve GitHub pagination error for users with more than 10 GitHub organizations. +- Fix HTTP 400 error that occurred when making HTTP requests on an HTTPS server. +- Upgrade to Flux v0.50.2, includes updated Flux functions. +- Fix date range picker in Data Explorer. + +## v1.7.14 [2019-08-27] + +### Bug Fixes + +- Fix Data Explorer crashing due to empty query. +- Fix styles in Kapacitor alert config page. + +## v1.7.13 [2019-08-20] + +### Bug Fixes + +- Fix scroll to row bug on table graphs. +- Wrap inline commas in quotes to distinguish from CSV delimiters. +- Fix TICKscript editor syntax coloring. +- Fix JWK check when using login_id. +- Fix dashboard typos. +- Configure Papa Parse to distinguish inline vs delimiter commas. +- Fix TICKscript editor coloring for boolean literals. +- Fix JWK signing key check. +- Fix alert rule message text template parsing. +- Fix erroneous query manipulation. +- Fix group by database for numSeries and numMeasurement queries in canned dashboards. +- Update `axios` and `lodash` dependenies with known vulnerabilities. +- Fix dashboard typos in protoboard queries. +- Fix repeating last command in Data Explore window when multiple tabs are open. + +### Features + +- Add toggle for UTC and local time. +- Add time zone selector to data explorer. +- Add time zone toggle. +- Add Login Hint and redirect to OAuth provider automatically. + +## v1.7.12 [2019-06-20] + +## Bug Fixes + +- Clarify wording of PagerDuty v1 deprecation message. +- Requesting info from an unavailable source no longer causes the page to hang. +- Create Chronograf user before CentOS installation. +- Add support for web workers in IE11. +- Properly update query time bounds when zooming in on a dashboard. +- Fix an issue where Flux responses weren't parsed correctly. + +### Features + +- Allow negative numbers for configured y-axis minimums + +## v1.7.11 [2019-04-24] + +## Bug Fixes + +- Fix fetching tag keys in Flux Builder. + +## v1.7.10 [2019-04-16] + +### Bug Fixes + +- Fix the input for line controls in visualization options. +- Fix Cell editor visualization not using CEO time range. +- Fix an issue where imports were not working in Flux scripts. + +### Features + +- Updated the UI to work with the latest Flux version. + +## v1.7.9 [2019-3-20] + +### Bug Fixes + +* Fix the input for line controls in visualization options. +* Stop scrollbars from covering text in Flux Editor. +* Insert Flux function near cursor in Flux Editor. +* Fix double-quoting of map template values. +* Fix disappearing data when scrolling a table. + +## v1.7.8 [2019-2-13] + +### Bug Fixes + +* Escape injected meta query values. +* Fix out-of-range decimal places. +* Stop raw y-axis format from getting updated to 10. +* Correct autoInterval calculations. +* Fix multiple organizations not showing configured Kapacitor connections. +* Fix the inability to edit Kapacitor info in the onboarding wizard. + +## v1.7.7 [2019-1-16] + +### Bug Fixes + +* Use JWT in Enterprise for authentication in Flux. + +## v1.7.6 [2019-1-14] + +### Bug Fixes + +* Properly set scroll to row for table graph. +* Prevent Kapacitor URLs from being overwritten in Connection Wizard. +* Fix logs intermittently appearing empty on first load. +* Prevent meta node URLs from being overwritten in Connection Wizard. +* Update functions list for Flux 0.12. + +## v1.7.5 [2018-12-17] + +### Bug Fixes + +* Update Go, Node, and Alpine versions. + +## v1.7.4 [2018-12-12] + +### UI Improvements + +* Add loading spinners while fetching protoboards. +* Add ability to skip Kapacitor step while adding a connection. +* Remove extra save options for retention policy during database creation. + +### Bug Fixes + +* Fix logs page getting stuck on scroll to top. +* Fix Flux pivot function using incorrectly named parameters. +* Fix momentary display of fallback notes while dashboard is loading. +* Fix issue displaying UUID in table cells. +* Update functions list for Flux 0.7.1. +* Fix single stat graphs decimal places when using Flux. +* Fix missing data and type in refreshing graph. +* Fix logs in loading state. +* Improve display of Flux Wizard on small screens. +* Update logs histogram data on click and new search. +* Prevent cell renaming widget from pushing other header elements offscreen. +* Fix Flux editor scrollbars. +* Use valid characters for Sensu IDs. + +## v1.7.3 [2018-11-13] + +### UI Improvements + +* Add loading spinners while fetching protoboards. + +### Bug Fixes + +* Get protoboards from multistore if not able to find from ProtoboardsPath. +* Handle basepath issue with missing slash. +* Fix the ping protoboard. +* Save fieldOptions to cells created from Data Explorer page. +* Fix grouping in canned dashboard queries. +* Update canned dashboard queries so they all use database and retention policy. +* Remove "dismiss" text from and add "X to dismiss" to wizard overlay steps. +* Update Docker, InfluxDB, and PostgreSQL protoboards. + +## v1.7.2 [2018-11-08] + +### Bug Fixes + +* Remove hardcoded database/retention period from protoboards. + +## v1.7.1 [2018-11-07] + +### Bug Fixes + +* Fix empty graph on alert rule creation page. +* Add protoboard environment variables to build scripts. +* Show manual refresh when paused. +* Update Dockerfile to include protoboards. +* Fix log columns not rendering. +* Fix scroll loading indicator not hiding in logs. + + +## v1.7.0 [2018-11-06] + +### Features + +* Add filestore-backed API for protodashboards. +* Add loading status indicator to hosts page. +* Add ability to copy expanded/untruncated log message. +* Add close button for logs popover. +* Add search attributes to log viewer. +* Add regex search for app name in log lines. +* Save log line wrap/truncate preference. +* Add button on Data Explorer to send query to dashboard cell. +* Introduce note visualization type. +* Add Dynamic Source option to Cell Editor Overlay (CEO) source selector. +* Add time selector dropdown to CEO. +* Add visualization options to the Data Explorer. +* Add Flux query editor to the Data Explorer and use same UI as CEO. +* Add ability to save a Flux query to a cell. +* Allow deep linking Flux script in Data Explorer. +* Add ability to use line graph, single stat, and table visualizations for Flux queries. +* Allow Flux dashboard cells to be exported. +* Add option to disable gzip compression. + +### UI Improvements + +* Clear logs after searching. +* Add logs page loading spinner. + +* Autofocus dashboard query editor. +* Fix query editor flickering on update. +* Remove character count limit from prefix and suffix for single stat and gauge cells. +* Add button to encourage switching visualization type to table graph when query response is not supported by line graph. +* Colorize entire single stat cell. +* Positioned cloned cells adjacent to target cell. +* Add metaquery template generator button to Data Explorer and CEO. +* Redesign CEO for reuse in other parts of application. +* Automatically scroll to the current measurement in the Data Explorer. + +### Bug Fixes + +* Render null data point values in alerts table as em dashes. +* Add validation to alert rule messages. +* Fix search results updating race condition. + +## v1.6.1 [2018-08-02] + +### Features + +* Include source IDs, links, and names in dashboard exports +* Add ability to map sources when importing dashboards + +### UI Improvements + +* Make it easier to get mouse into hover legend + +### Bug Fixes + +* Ensure text template variables reflect query parameters +* Enable using a new, blank text template variable in a query +* Ensure cells with broken queries display “No Data” +* Fix use of template variables within InfluxQL regexes +* Pressing play on log viewer goes to "now" +* Fix display of log viewer histogram when a basepath is enabled +* Fix crosshairs and hover legend display in Alert Rule visualization +* Size loading spinners based on height of their container + +## v1.6.0 [2018-07-18] + +### Features + +* Add support for template variables in cell titles. +* Add ability to export and import dashboards. +* Add ability to override template variables and time ranges via URL query. +* Add pprof routes to chronograf server. +* Add API to get/update Log Viewer UI config. +* Consume new Log Viewer config API in client to allow user to configure log viewer UI for their organization. + +### UI Improvements + +* Sort task table on Manage Alert page alphabetically. +* Redesign icons in side navigation. +* Remove Snip functionality in hover legend. +* Upgrade Data Explorer query text field with syntax highlighting and partial multi-line support. +* Truncate message preview in Alert Rules table. +* Improve performance of graph crosshairs. +* Hide dashboard cell menu until mouse over cell. +* Auto-Scale single-stat text to match cell dimensions. + +### Bug Fixes + +* Ensure cell queries use constraints from TimeSelector. +* Fix Gauge color selection bug. +* Fix erroneous icons in Date Picker widget. +* Fix allowing hyphens in basepath. +* Fix error in cell when tempVar returns no values. +* Change arrows in table columns so that ascending sort points up and descending points down. +* Fix crosshairs moving passed the edges of graphs. +* Change y-axis options to have valid defaults. +* Stop making requests for old sources after changing sources. +* Fix health check status code creating FireFox error. +* Change decimal places to enforce 2 places by default in cells. + + +## v1.5.0.1 [2018-06-04] + +### Bug Fixes + +* Fix Color Scale Dropdown + +## v1.5.0.0 [2018-05-22] + +### Features + +* Add table view as a visualization type. +* Add default retention policy field as option in source configuration for use in querying hosts from Host List page and Host pages. +* Add support for PagerDuty v2 alert endpoints in UI. +* Add support for OpsGenie v2 alert endpoints in UI. +* Add support for Kafka alert endpoint in UI to configure and create alert handlers. +* Add support for disabling Kapacitor services. +* Add support for multiple Slack alert endpoint configurations in the UI. + +### User interface improvements + +* Notify user when a dashboard cell is added, removed, or cloned. +* Fix Template Variables Control Bar to top of dashboard page. +* Remove extra click when creating dashboard cell. +* Reduce font sizes in dashboards for increased space efficiency. +* Add overlay animation to Template Variables Manager. +* Display 'No results' on cells without results. +* Disable template variables for non-editing users. +* YAxisLabels in Dashboard Graph Builder not showing until graph is redrawn. +* Ensure table views have a consistent user experience between Google Chrome and Mozilla Firefox. +* Change AutoRefresh interval to paused. +* Get cloned cell name for notification from cloned cell generator function. +* Improve load time for Host page. +* Show Kapacitor batch point info in log panel. + +### Bug fixes + +* Allow user to select TICKscript editor with mouse-click. +* Change color when value is equal to or greater than the threshold value. +* Fix base path for Kapacitor logs. +* Fix logout when using `basepath` and simplify `basepath` usage (deprecates `PREFIX_ROUTES`). +* Fix graphs in alert rule builder for queries that include `groupBy`. +* Fix auto not showing in the group by dropdown and explorer getting disconnected. +* Display y-axis label on initial graph load. +* Fix not being able to change the source in the CEO display. +* Fix only the selected template variable value getting loaded. +* Fix Generic OAuth bug for GitHub Enterprise where the principal was incorrectly being checked for email being Primary and Verified. +* Fix missing icons when using basepath. +* Limit max-width of TICKScript editor. +* Fix naming of new TICKScripts. +* Fix data explorer query error reporting regression. +* Fix Kapacitor Logs fetch regression. + +## v1.4.4.1 [2018-04-16] + +### Bug fixes + +* Snapshot all db struct types in migration files. + +## v1.4.4.0 [2018-04-13] + +### Features + +* Add support for RS256/JWKS verification, support for `id_token` parsing (as in ADFS). +* Add ability to set a color palette for Line, Stacked, Step-Plot, and Bar graphs. +* Add ability to clone dashboards. +* Change `:interval:` to represent a raw InfluxQL duration value. +* Add paginated measurements API to server. +* Data Explorer measurements can be toggled open. + +### UI improvements + +* New dashboard cells appear at bottom of layout and assume the size of the most common cell. +* Standardize delete confirmation interactions. +* Standardize Save and Cancel interactions. +* Improve cell renaming. + +### Bug fixes + +* Always save template variables on first edit. +* Query annotations at auto-refresh interval. +* Display link to configure Kapacitor on Alerts Page if no configured Kapacitor. +* Fix saving of new TICKscripts. +* Fix appearance of cell y-axis titles. +* Only add `stateChangesOnly` to new rules. +* Fix 500s when deleting organizations. +* Fixes issues with providing regexp in query. +* Ensure correct basepath prefix in URL pathname when passing InfluxQL query param to Data Explorer. +* Fix type error bug in Kapacitor Alert Config page and persist deleting of team and recipient in OpsGenieConfig. +* Fixes errors caused by switching query tabs in CEO. +* Only send threshold value to parent on blur. +* Require that emails on GitHub & Generic OAuth2 principals be verified & primary, if those fields are provided. +* Send notification when retention policy (rp) creation returns a failure. +* Show valid time in custom time range when now is selected. +* Default to zero for gauges. + +## v1.4.3.3 [2018-04-12] + +### Bug Fixes + +* Require that emails on GitHub & Generic OAuth2 principals be verified & primary if those fields are provided. + +## v1.4.3.1 [2018-04-02] + +### Bug fixes + +* Fixes template variable editing not allowing saving. +* Save template variables on first edit. +* Fix template variables not loading all values. + + +## v1.4.3.0 [2018-3-28] + +### Features + +* Add unsafe SSL to Kapacitor UI configuration +* Add server flag to grant SuperAdmin status to users authenticating from a specific Auth0 organization + +### UI Improvements + +* Redesign system notifications + +### Bug Fixes + +* Fix Heroku OAuth 2.0 provider support +* Fix error reporting in Data Explorer +* Fix Okta OAuth 2.0 provider support +* Change hover text on delete mappings confirmation button to 'Delete' +* Automatically add graph type 'line' to any graph missing a type +* Fix hanging browser on docker host dashboard +* Fix Kapacitor Rules task enabled checkboxes to only toggle exactly as clicked +* Prevent Multi-Select Dropdown in InfluxDB Admin Users and Roles tabs from losing selection state +* Fix intermittent missing fill from graphs +* Support custom time range in annotations API wrapper + +## v1.4.2.5 [2018-04-12] + +### Bug Fixes + +* Require that emails on GitHub & Generic OAuth2 principals be verified & primary if those fields are provided. + +## v1.4.2.3 [2018-03-08] + +### Bug fixes + +* Include URL in Kapacitor connection creation requests. + +## v1.4.2.1 [2018-02-28] + +### Features + +* Prevent execution of queries in cells that are not in view on the Dashboard page. +* Add an optional persistent legend which can toggle series visibility to dashboard cells. +* Allow user to annotate graphs via UI or API. + +### UI improvements + +* Add ability to set a prefix and suffix on Single Stat and Gauge cell types. +* Rename 'Create Alerts' page to 'Manage Tasks'; redesign page to improve clarity of purpose. + +### Bug fixes + +* Save only selected template variable values into dashboards for non-CSV template variables. +* Use Generic APIKey for OAuth2 group lookup. +* Fix bug in which resizing any cell in a dashboard causes a Gauge cell to resize. +* Don't sort Single Stat & Gauge thresholds when editing threshold values. +* Maintain y-axis labels in dashboard cells. +* Deprecate `--new-sources` in CLI. + +## v1.4.1.5 [2018-04-12] + +### Bug Fixes + +* Require that emails on GitHub & Generic OAuth2 principals be verified & primary if those fields are provided. + +## v1.4.1.3 [2018-02-14] + +### Bug fixes + +* Allow self-signed certificates for InfluxDB Enterprise meta nodes. + +## v1.4.1.2 [2018-02-13] + +### Bug fixes + +* Respect `basepath` when fetching server API routes. +* Set default `tempVar` `:interval`: with Data Explorer CSV download call. +* Display series with value of `0` in a cell legend. + +## v1.4.1.1 [2018-02-12] + +### Features + +- Allow multiple event handlers per rule. +- Add "Send Test Alert" button to test Kapacitor alert configurations. +- Link to Kapacitor config panel from Alert Rule builder. +- Add auto-refresh widget to Hosts List page. +- Upgrade to Go 1.9.4 and Node 6.12.3. +- Allow users to delete themselves. +- Add All Users page, visible only to SuperAdmins. +- Introduce `chronoctl` binary for user CRUD operations. +- Introduce mappings to allow control over new user organization assignments. + +### UI improvements + +- Clarify terminology regarding InfluxDB and Kapacitor connections. +- Separate saving TICKscript from exiting editor page. +- Enable Save (`⌘ + Enter`) and Cancel (`Escape`) hotkeys in Cell Editor Overlay. +- Enable customization of Single Stat "Base Color". + +### Bug fixes + +- Fix TICKscript Sensu alerts when no GROUP BY tags selected. +- Display 200 most-recent TICKscript log messages; prevent overlapping. +- Add `TO` to kapacitor SMTP config; improve config update error messages. +- Remove CLI options from `sysvinit` service file. +- Remove CLI options from `systemd` service file. +- Fix disappearance of text in Single Stat graphs during editing. +- Redirect to Alerts page after saving Alert Rule. + +## v1.4.0.3 [2018-4-12] + +### Bug Fixes + +* Require that emails on GitHub & Generic OAuth2 principals be verified & primary if those fields are provided. + +## v1.4.0.1 [2018-1-9] + +### Features + +* Add separate CLI flag for canned sources, Kapacitors, dashboards, and organizations. +* Add Telegraf interval configuration. + +### Bug fixes + +- Allow insecure (self-signed) certificates for Kapacitor and InfluxDB. +- Fix positioning of custom time indicator. + +## v1.4.0.0 [2017-12-22] + +### Features + +* Add support for multiple organizations, multiple users with role-based access control, and private instances. +* Add Kapacitor logs to the TICKscript editor +* Add time shift feature to DataExplorer and Dashboards +* Add auto group by time to Data Explorer +* Support authentication for Enterprise Meta Nodes +* Add Boolean thresholds for kapacitor threshold alerts +* Update kapacitor alerts to cast to float before sending to influx +* Allow override of generic oauth2 keys for email + +### UI improvements + +* Introduce customizable Gauge visualization type for dashboard cells +* Improve performance of Hosts, Alert History, and TICKscript logging pages when there are many items to display +* Add filtering by name to Dashboard index page +* Improve performance of hoverline rendering + +### Bug fixes + +* Fix `.jsdep` step fails when LDFLAGS is exported +* Fix logscale producing console errors when only one point in graph +* Fix 'Cannot connect to source' false error flag on Dashboard page +* Add fractions of seconds to time field in csv export +* Fix Chronograf requiring Telegraf's CPU and system plugins to ensure that all Apps appear on the HOST LIST page. +* Fix template variables in dashboard query building. +* Fix several kapacitor alert creation panics. +* Add shadow-utils to RPM release packages +* Source extra command line options from defaults file +* After CREATE/DELETE queries, refresh list of databases in Data Explorer +* Visualize CREATE/DELETE queries with Table view in Data Explorer +* Include tag values alongside measurement name in Data Explorer result tabs +* Redesign cell display options panel +* Fix queries that include regex, numbers and wildcard +* Fix apps on hosts page from parsing tags with null values +* Fix updated Dashboard names not updating dashboard list +* Fix create dashboard button +* Fix default y-axis labels not displaying properly +* Gracefully scale Template Variables Manager overlay on smaller displays +* Fix Influx Enterprise users from deletion in race condition +* Fix oauth2 logout link not having basepath +* Fix supplying a role link to sources that do not have a metaURL +* Fix hoverline intermittently not rendering +* Update MySQL pre-canned dashboard to have query derivative correctly + +## v1.3.10.0 [2017-10-24] + +### Bug fixes + +* Improve the copy in the retention policy edit page. +* Fix `Could not connect to source` bug on source creation with unsafe-ssl. +* Fix when exporting `SHOW DATABASES` CSV has bad data. +* Fix not-equal-to highlighting in Kapacitor Rule Builder. +* Fix undescriptive error messages for database and retention policy creation. +* Fix drag and drop cancel button when writing data in the data explorer. +* Fix persistence of "SELECT AS" statements in queries. + +### Features + +* Every dashboard can now have its own time range. +* Add CSV download option in dashboard cells. +* Implicitly prepend source URLs with `http://` +* Add support for graph zooming and point display on the millisecond-level. +* Add manual refresh button for Dashboard, Data Explorer, and Host Pages. + +### UI improvements + +* Increase size of Cell Editor query tabs to reveal more of their query strings. +* Improve appearance of Admin Page tabs on smaller screens. +* Add cancel button to TICKscript editor. +* Redesign dashboard naming & renaming interaction. +* Redesign dashboard switching dropdown. + +## v1.3.9.0 [2017-10-06] + +### Bug fixes + +* Fix Data Explorer disappearing query templates in dropdown. +* Fix missing alert for duplicate db name. +* Chronograf shows real status for Windows hosts when metrics are saved in non-default db. +* Fix false error warning for duplicate Kapacitor name +* Fix unresponsive display options and query builder in dashboards. + +### Features + +* Add fill options to Data Explorer and dashboard queries. +* Support editing kapacitor TICKScript. +* Introduce the TICKscript editor UI. +* Add CSV download button to the Data Explorer. +* Add Data Explorer InfluxQL query and location query synchronization, so queries can be shared using a URL. +* Able to switch InfluxDB sources on a per graph basis. + +### UI improvements + +* Require a second click when deleting a dashboard cell. +* Sort database list in Schema Explorer alphabetically. +* Improve usability of dashboard cell context menus. +* Move dashboard cell renaming UI into Cell Editor Overlay. +* Prevent the legend from overlapping graphs at the bottom of the screen. +* Add a "Plus" icon to every button with an Add or Create action for clarity and consistency. +* Make hovering over series smoother. +* Reduce the number of pixels per cell to one point per 3 pixels. +* Remove tabs from Data Explorer. +* Improve appearance of placeholder text in inputs. +* Add ability to use "Default" values in Source Connection form. +* Display name & port in SourceIndicator tool tip. + +## v1.3.8.3 [2017-09-29] + +### Bug fixes + +* Fix duration for single value and custom time ranges. +* Fix Data Explorer query templates dropdown disappearance. +* Fix no alert for duplicate db name. +* Fix unresponsive display options and query builder in dashboards. + +## v1.3.8.2 [2017-09-22] + +### Bug fixes + +* Fix duration for custom time ranges. + +## v1.3.8.1 [2017-09-08] + +### Bug fixes + +* Fix return code on meta nodes when raft redirects to leader. +* Reduce points per graph to one point per 3 pixels. + +## v1.3.8.0 [2017-09-07] + +### Bug fixes + +* Fix the limit of 100 alert rules on alert rules page. +* Fix graphs when y-values are constant. +* Fix crosshair not being removed when user leaves graph. +* Fix inability to add kapacitor from source page on fresh install. +* Fix DataExplorer crashing if a field property is not present in the queryConfig. +* Fix the max y value of stacked graphs preventing display of the upper bounds of the chart. +* Fix for delayed selection of template variables using URL query params. + +### Features + +* Add prefix, suffix, scale, and other y-axis formatting for cells in dashboards. +* Update the group by time when zooming in graphs. +* Add the ability to link directly to presentation mode in dashboards with the `present` Boolean query parameter in the URL. +* Add the ability to select a template variable via a URL parameter. + +### UI improvements + +* Use line-stacked graph type for memory information. +* Improve cell sizes in Admin Database tables. +* Polish appearance of optional alert parameters in Kapacitor rule builder. +* Add active state for Status page navbar icon. +* Improve UX of navigation to a sub-nav item in the navbar. + + +## v1.3.7.0 [2017-08-23] + +### Bug fixes + + * Chronograf now renders on Internet Explorer (IE) 11. + * Resolve Kapacitor config for PagerDuty via the UI. + * Fix Safari display issues in the Cell Editor display options. + * Fix uptime status on Windows hosts running Telegraf. + * Fix console error for 'placing prop on div'. + * Fix Write Data form upload button and add `onDragExit` handler. + * Fix missing cell type (and consequently single-stat). + * Fix regression and redesign drag & drop interaction. + * Prevent stats in the legend from wrapping line. + * Fix raw query editor in Data Explorer, not using selected time. + +### Features + + * Improve 'new-sources' server flag example by adding 'type' key. + * Add an input and validation to custom time range calendar dropdowns. + * Add support for selecting template variables with URL params. + +### UI improvements + * Show "Add Graph" button on cells with no queries. + +## v1.3.6.1 [2017-08-14] + +**Upgrade Note** This release (1.3.6.1) fixes a possibly data corruption issue with dashboard cells' graph types. If you upgraded to 1.3.6.0 and visited any dashboard, once you have then upgraded to this release (1.3.6.1) you will need to manually reset the graph type for every cell via the cell's caret --> Edit --> Display Options. If you upgraded directly to 1.3.6.1, you should not experience this issue. + +### Bug fixes + + * Fix inaccessible scroll bar in Data Explorer table. + * Fix non-persistence of dashboard graph types. + +### Features + + * Add y-axis controls to the API for layouts. + +### UI improvements + + * Increase screen real estate of Query Maker in the Cell Editor Overlay. + +## v1.3.6.0 [2017-08-08] + +### Bug fixes + + * Fix domain not updating in visualizations when changing time range manually. + * Prevent console error spam from Dygraph's synchronize method when a dashboard has only one graph. + * Guarantee UUID for each Alert Table key to prevent dropping items when keys overlap. + +### Features + + * Add a few time range shortcuts to the custom time range menu. + * Add ability to edit a dashboard graph's y-axis bounds. + * Add ability to edit a dashboard graph's y-axis label. + +### UI improvements + * Add spinner in write data modal to indicate data is being written. + * Fix bar graphs overlapping. + * Assign a series consistent coloring when it appears in multiple cells. + * Increase size of line protocol manual entry in Data Explorer's Write Data overlay. + * Improve error message when request for Status Page News Feed fails. + * Provide affirmative UI choice for 'auto' in DisplayOptions with new toggle-based component. + +## v1.3.5.0 [2017-07-27] + +### Bug fixes + + * Fix z-index issue in dashboard cell context menu. + * Clarify BoltPath server flag help text by making example the default path. + * Fix cell name cancel not reverting to original name. + * Fix typo that may have affected PagerDuty node creation in Kapacitor. + * Prevent 'auto' GROUP BY as option in Kapacitor rule builder when applying a function to a field. + * Prevent clipped buttons in Rule Builder, Data Explorer, and Configuration pages. + * Fix JWT for the write path. + * Disentangle client Kapacitor rule creation from Data Explorer query creation. + +### Features + + * View server generated TICKscripts. + * Add the ability to select Custom Time Ranges in the Hostpages, Data Explorer, and Dashboards. + * Clarify BoltPath server flag help text by making example the default path + * Add shared secret JWT authorization to InfluxDB. + * Add Pushover alert support. + * Restore all supported Kapacitor services when creating rules, and add most optional message parameters. + +### UI improvements + + * Polish alerts table in status page to wrap text less. + * Specify that version is for Chronograf on Configuration page. + * Move custom time range indicator on cells into corner when in presentation mode. + * Highlight legend "Snip" toggle when active. + +## v1.3.4.0 [2017-07-10] + +### Bug fixes + * Disallow writing to \_internal in the Data Explorer. + * Add more than one color to Line+Stat graphs. + * Fix updating Retention Policies in single-node InfluxDB instances. + * Lock the width of Template Variable dropdown menus to the size of their longest option. + +### Features + + * Add Auth0 as a supported OAuth2 provider. + * Add ability to add custom links to User menu via server CLI or ENV vars. + * Allow users to configure custom links on startup that will appear under the User menu in the sidebar. + * Add support for Auth0 organizations. + * Allow users to configure InfluxDB and Kapacitor sources on startup. + +### UI improvements + + * Redesign Alerts History table on Status Page to have sticky headers. + * Refresh Template Variable values on Dashboard page load. + * Display current version of Chronograf at the bottom of Configuration page. + * Redesign Dashboards table and sort them alphabetically. + * Bring design of navigation sidebar in line with Branding Documentation. + +## v1.3.3.0 [2017-06-19] + +### Bug fixes + + * Prevent legend from flowing over window bottom bound + * Prevent Kapacitor configurations from having the same name + * Limit Kapacitor configuration names to 33 characters to fix display bug + +### Features + + * Synchronize vertical crosshair at same time across all graphs in a dashboard + * Add automatic `GROUP BY (time)` functionality to dashboards + * Add a Status Page with Recent Alerts bar graph, Recent Alerts table, News Feed, and Getting Started widgets + +### UI improvements + + * When dashboard time range is changed, reset graphs that are zoomed in + * [Bar graph](/chronograf/v1.9/guides/visualization-types/#bar-graph) option added to dashboard + * Redesign source management table to be more intuitive + * Redesign [Line + Single Stat](/chronograf/v1.9/guides/visualization-types/#line-graph-single-stat) cells to appear more like a sparkline, and improve legibility + + +## v1.3.2.0 [2017-06-05] + +### Bug fixes + + * Update the query config's field ordering to always match the input query + * Allow users to add functions to existing Kapacitor rules + * Fix logout menu item regression + * Fix InfluxQL parsing with multiple tag values for a tag key + * Fix load localStorage and warning UX on fresh Chronograf install + * Show submenus when the alert notification is present + +### Features + + * Add UI to the Data Explorer for [writing data to InfluxDB](/chronograf/v1.9/guides/querying-data/) + +### UI improvements + + * Make the enter and escape keys perform as expected when renaming dashboards + * Improve copy on the Kapacitor configuration page + * Reset graph zoom when the user selects a new time range + * Upgrade to new version of Influx Theme, and remove excess stylesheets + * Replace the user icon with a solid style + * Disable query save in cell editor mode if the query does not have a database, measurement, and field + * Improve UX of applying functions to fields in the query builder + +## v1.3.1.0 [2017-05-22] + +### Release notes + +In versions 1.3.1+, installing a new version of Chronograf automatically clears the localStorage settings. + +### Bug fixes + + * Fix infinite spinner when `/chronograf` is a [basepath](/chronograf/v1.9/administration/config-options/#basepath-p) + * Remove the query templates dropdown from dashboard cell editor mode + * Fix the backwards sort arrows in table column headers + * Make the logout button consistent with design + * Fix the loading spinner on graphs + * Filter out any template variable values that are empty, whitespace, or duplicates + * Allow users to click the add query button after selecting singleStat as the [visualization type](/chronograf/v1.9/guides/visualization-types) + * Add a query for windows uptime - thank you, @brianbaker! + +### Features + + * Add log event handler- thank you, @mpchadwick! + * Update Go (golang) vendoring to dep and committed vendor directory + * Add autocomplete functionality to [template variable](/chronograf/v1.9/guides/dashboard-template-variables) dropdowns + +### UI improvements + + * Refactor scrollbars to support non-webkit browsers + * Increase the query builder's default height in cell editor mode and in the data explorer + * Make the [template variables](/chronograf/v1.9/guides/dashboard-template-variables) manager more space efficient + * Add page spinners to pages that did not have them + * Denote which source is connected in the sources table + * Use milliseconds in the InfluxDB dashboard instead of nanoseconds + * Notify users when local settings are cleared + +## v1.3.0 [2017-05-09] + +### Bug fixes + + * Fix the link to home when using the [`--basepath` option](/chronograf/v1.9/administration/config-options/#basepath-p) + * Remove the notification to login on the login page + * Support queries that perform math on functions + * Prevent the creation of blank template variables + * Ensure thresholds for Kapacitor Rule Alerts appear on page load + * Update the Kapacitor configuration page when the configuration changes + * Fix Authentication when using Chronograf with a set [basepath](/chronograf/v1.9/administration/config-options/#basepath-p) + * Show red indicator on Hosts Page for an offline host + * Support escaping from presentation mode in Safari + * Re-implement level colors on the alerts page + * Fix router bug introduced by upgrading to react-router v3.0 + * Show legend on [Line+Stat](/chronograf/v1.9/guides/visualization-types/#line-graph-single-stat) visualization type + * Prevent queries with `:dashboardTime:` from breaking the query builder + +### Features + + * Add line-protocol proxy for InfluxDB/InfluxDB Enterprise Cluster data sources + * Add `:dashboardTime:` to support cell-specific time ranges on dashboards + * Add support for enabling and disabling [TICKscripts that were created outside Chronograf](/chronograf/v1.9/guides/advanced-kapacitor/#tickscript-management) + * Allow users to delete Kapacitor configurations + +### UI improvements + + * Save user-provided relative time ranges in cells + * Improve how cell legends and options appear on dashboards + * Combine the measurements and tags columns in the Data Explorer and implement a new design for applying functions to fields. + * Normalize the terminology in Chronograf + * Make overlays full-screen + * Change the default global time range to past 1 hour + * Add the Source Indicator icon to the Configuration and Admin pages + + +{{% note %}} +See Chronograf's [CHANGELOG](https://github.com/influxdata/chronograf/blob/master/CHANGELOG.md) on GitHub for information about the 1.2.0-beta releases. +{{% /note %}} diff --git a/content/chronograf/v1.9/administration/_index.md b/content/chronograf/v1.9/administration/_index.md new file mode 100644 index 000000000..d56443aa8 --- /dev/null +++ b/content/chronograf/v1.9/administration/_index.md @@ -0,0 +1,14 @@ +--- +title: Administering Chronograf +description: > + Upgrade and configure Chronograf, plus manage connections, users, security, and organizations. +menu: + chronograf_1_9: + name: Administration + weight: 40 + +--- + +Follow the links below for more information. + +{{< children >}} diff --git a/content/chronograf/v1.9/administration/chrono-on-clusters.md b/content/chronograf/v1.9/administration/chrono-on-clusters.md new file mode 100644 index 000000000..ef82d22d2 --- /dev/null +++ b/content/chronograf/v1.9/administration/chrono-on-clusters.md @@ -0,0 +1,22 @@ +--- +title: Connecting Chronograf to InfluxDB Enterprise clusters +description: Work with InfluxDB Enterprise clusters through the Chronograf UI. +menu: + chronograf_1_9: + name: Connecting Chronograf to InfluxDB Enterprise + weight: 40 + parent: Administration +--- + +The connection details form requires additional information when connecting Chronograf to an [InfluxDB Enterprise cluster](/{{< latest "enterprise_influxdb" >}}/). + +When you enter the InfluxDB HTTP bind address in the `Connection String` input, Chronograf automatically checks if that InfluxDB instance is a data node. +If it is a data node, Chronograf automatically adds the `Meta Service Connection URL` input to the connection details form. +Enter the HTTP bind address of one of your cluster's meta nodes into that input and Chronograf takes care of the rest. + +![Cluster connection details](/img/chronograf/1-6-faq-cluster-connection.png) + +Note that the example above assumes that you do not have authentication enabled. +If you have authentication enabled, the form requires username and password information. + +For details about monitoring InfluxDB Enterprise clusters, see [Monitoring InfluxDB Enterprise clusters](/chronograf/v1.9/guides/monitoring-influxenterprise-clusters). diff --git a/content/chronograf/v1.9/administration/config-options.md b/content/chronograf/v1.9/administration/config-options.md new file mode 100644 index 000000000..9a2a9da1c --- /dev/null +++ b/content/chronograf/v1.9/administration/config-options.md @@ -0,0 +1,682 @@ +--- +title: Chronograf configuration options +description: > + Options available in the Chronograf configuration file and environment variables. +menu: + chronograf_1_9: + name: Configuration options + weight: 30 + parent: Administration +--- + +Chronograf is configured using the configuration file (/etc/default/chronograf) and environment variables. If you do not uncomment a configuration option, the system uses its default setting. The configuration settings in this document are set to their default settings. For more information, see [Configure Chronograf](/chronograf/v1.9/administration/configuration/). + +* [Usage](#usage) +* [Chronograf service options](#chronograf-service-options) + - [InfluxDB connection options](#influxdb-connection-options) + - [Kapacitor connection options](#kapacitor-connection-options) + - [TLS (Transport Layer Security) options](#tls-transport-layer-security-options) + - [etcd options](#etcd-options) + - [Other service options](#other-service-options) +* [Authentication options](#authentication-options) + * [General authentication options](#general-authentication-options) + * [GitHub-specific OAuth 2.0 authentication options](#github-specific-oauth-20-authentication-options) + * [Google-specific OAuth 2.0 authentication options](#google-specific-oauth-20-authentication-options) + * [Auth0-specific OAuth 2.0 authentication options](#auth0-specific-oauth-20-authentication-options) + * [Heroku-specific OAuth 2.0 authentication options](#heroku-specific-oauth-20-authentication-options) + * [Generic OAuth 2.0 authentication options](#generic-oauth-20-authentication-options) + +## Usage + +Start the Chronograf service, and include any options after `chronograf`, where `[OPTIONS]` are options separated by spaces: + +```sh +chronograf [OPTIONS] +``` + +**Linux examples** + +- To start `chronograf` without options: + +```sh +sudo systemctl start chronograf +``` + +- To start `chronograf` and set options for develop mode and to disable reporting: + +```sh +sudo systemctl start chronograf --develop --reporting-disabled +``` + +**MacOS X examples** + +- To start `chronograf` without options: + +```sh +chronograf +``` + +- To start `chronograf` and add shortcut options for develop mode and to disable reporting: + +```sh +chronograf -d -r +``` + +{{% note %}} +***Note:*** Command line options take precedence over corresponding environment variables. +{{% /note %}} + +## Chronograf service options + +#### `--host=` + +The IP that the `chronograf` service listens on. + +Default value: `0.0.0.0` + +Example: `--host=0.0.0.0` + +Environment variable: `$HOST` + +#### `--port=` + +The port that the `chronograf` service listens on for insecure connections. + +Default: `8888` + +Environment variable: `$PORT` + +#### `--bolt-path=` | `-b` + +The file path to the BoltDB file. + +Default value: `./chronograf-v1.db` + +Environment variable: `$BOLT_PATH` + +#### `--canned-path=` | `-c` + +The path to the directory of [canned dashboards](/chronograf/v1.9/guides/using-precreated-dashboards) files. Canned dashboards (also known as pre-created dashboards or application layouts) cannot be edited. They're delivered with Chronograf and available depending on which Telegraf input plugins you have enabled. + +Default value: `/usr/share/chronograf/canned` + +Environment variable: `$CANNED_PATH` + +#### `--resources-path=` + +Path to directory of sources (.src files), Kapacitor connections (.kap files), organizations (.org files), and dashboards (.dashboard files). + +{{% note %}} +**Note:** If you have a dashboard with the `.json` extension, rename it with the `.dashboard` extension in this directory to ensure the dashboard is loaded. +{{% /note %}} + +Default value: `/usr/share/chronograf/resources` + +Environment variable: `$RESOURCES_PATH` + +#### `--basepath=` | `-p` + +The URL path prefix under which all `chronograf` routes will be mounted. + +Environment variable: `$BASE_PATH` + +#### `--status-feed-url=` + +URL of JSON feed to display as a news feed on the client Status page. + +Default value: `https://www.influxdata.com/feed/json` + +Environment variable: `$STATUS_FEED_URL` + +#### `--version` | `-v` + +Displays the version of the Chronograf service. + +Example: +```sh +$ chronograf -v +2018/01/03 14:11:19 Chronograf 1.4.0.0-rc1-26-gb74ae387 (git: b74ae387) +``` + +## InfluxDB connection options + +{{% note %}} +InfluxDB connection details specified via command line when starting Chronograf do not persist when Chronograf is shut down. +To persist connection details, [include them in a `.src` file](/chronograf/v1.9/administration/creating-connections/#manage-influxdb-connections-using-src-files) located in your [`--resources-path`](#resources-path). + +**Only InfluxDB 1.x connections are configurable in a `.src` file.** +Configure InfluxDB 2.x and Cloud connections with CLI flags or in the +[Chronograf UI](/chronograf/v1.9/administration/creating-connections/#manage-influxdb-connections-using-the-chronograf-ui). +{{% /note %}} + +### `--influxdb-url` + +The location of your InfluxDB instance, including the protocol, IP address, and port. + +Example: `--influxdb-url http://localhost:8086` + +Environment variable: `$INFLUXDB_URL` + +### `--influxdb-username` + +The [username] for your InfluxDB instance. + +Environment variable: `$INFLUXDB_USERNAME` + +### `--influxdb-password` + +The [password] for your InfluxDB instance. + +Environment variable: `$INFLUXDB_PASSWORD` + +### `--influxdb-org` + +InfluxDB 2.x or InfluxDB Cloud organization name. + +Environment variable: `$INFLUXDB_ORG` + +### `--influxdb-token` + +InfluxDB 2.x or InfluxDB Cloud [authentication token](/influxdb/cloud/security/tokens/). + +Environment variable: `$INFLUXDB_TOKEN` + +## Kapacitor connection options + +{{% note %}} +Kapacitor connection details specified via command line when starting Chronograf do not persist when Chronograf is shut down. +To persist connection details, [include them in a `.kap` file](/chronograf/v1.9/administration/creating-connections/#manage-kapacitor-connections-using-kap-files) located in your [`--resources-path`](#resources-path). +{{% /note %}} + +### `--kapacitor-url=` + +The location of your Kapacitor instance, including `http://`, IP address, and port. + +Example: `--kapacitor-url=http://0.0.0.0:9092`. + +Environment variable: `$KAPACITOR_URL` + +### `--kapacitor-username=` + +The username for your Kapacitor instance. + +Environment variable: `$KAPACITOR_USERNAME` + +### `--kapacitor-password=` + +The password for your Kapacitor instance. + +Environment variable: `$KAPACITOR_PASSWORD` + +### TLS (Transport Layer Security) options + +See [Configuring TLS (Transport Layer Security) and HTTPS](/chronograf/v1.9/administration/managing-security/#configure-tls-transport-layer-security-and-https) for more information. + +#### `--cert=` + +The file path to PEM-encoded public key certificate. + +Environment variable: `$TLS_CERTIFICATE` + +#### `--key=` + +The file path to private key associated with given certificate. + +Environment variable: `$TLS_PRIVATE_KEY` + +### etcd options + +#### `--etcd-endpoints=` | `-e` + +List of etcd endpoints. + +##### CLI example + +```sh +## Single parameter +--etcd-endpoints=localhost:2379 + +## Mutiple parameters +--etcd-endpoints=localhost:2379 \ +--etcd-endpoints=192.168.1.61:2379 \ +--etcd-endpoints=192.192.168.1.100:2379 +``` + +Environment variable: `$ETCD_ENDPOINTS` + +##### Environment variable example + +```sh +## Single parameter +ETCD_ENDPOINTS=localhost:2379 + +## Mutiple parameters +ETCD_ENDPOINTS=localhost:2379,192.168.1.61:2379,192.192.168.1.100:2379 +``` + +#### `--etcd-username=` + +Username to log into etcd. + +Environment variable: `$ETCD_USERNAME` + +#### `--etcd-password=` + +Password to log into etcd. + +Environment variable: `$ETCD_PASSWORD` + +#### `--etcd-dial-timeout=` + +Total time to wait before timing out while connecting to etcd endpoints. +0 means no timeout. +The default is 1s. + +Environment variable: `$ETCD_DIAL_TIMEOUT` + +#### `--etcd-request-timeout=` + +Total time to wait before timing out an etcd view or update request. +0 means no timeout. +The default is 1s. + +Environment variable: `$ETCD_REQUEST_TIMEOUT` + +#### `--etcd-cert=` + +Path to etcd PEM-encoded TLS public key certificate. + +Environment variable: `$ETCD_CERTIFICATE` + +#### `--etcd-key=` + +Path to private key associated with specified etcd certificate. + +Environment variable: `$ETCD_PRIVATE_KEY` + +#### `--etcd-root-ca` + +Path to root CA certificate for TLS verification. + +Environment variable: `$ETCD_ROOT_CA` + +### Other service options + +#### `--custom-auto-refresh` + +Add custom auto-refresh intervals to the list of available auto-refresh intervals in Chronograf dashboards. +Provide a semi-colon-delimited list of key-value pairs where the key is the interval +name that appears in the auto-refresh dropdown menu and the value is the auto-refresh interval in milliseconds. + +Example: `--custom-auto-refresh "500ms=500;1s=1000"` + +Environment variable: `$CUSTOM_AUTO_REFRESH` + +#### `--custom-link :` + +Custom link added to Chronograf User menu options. Useful for providing links to internal company resources for your Chronograf users. Can be used when any OAuth 2.0 authentication is enabled. To add another custom link, repeat the custom link option. + +Example: `--custom-link InfluxData:http://www.influxdata.com/` + +#### `--develop` | `-d` + +Run the `chronograf` service in developer mode. + +#### `--help` | `-h` + +Displays the command line help for `chronograf`. + +#### `--host-page-disabled` | `-H` + +Disables rendering and serving of the Hosts List page (/sources/$sourceId/hosts). + +Environment variable: `$HOST_PAGE_DISABLED` + +#### `--log-level=` | `-l` + +Set the logging level. + +Valid values: `debug` | `info` | `error` + +Default value: `info` + +Example: `--log-level=debug` + +Environment variable: `$LOG_LEVEL` + +#### `--reporting-disabled` | `-r` + +Disables reporting of usage statistics. +Usage statistics reported once every 24 hours include: `OS`, `arch`, `version`, `cluster_id`, and `uptime`. + +Environment variable: `$REPORTING_DISABLED` + +## Authentication options + +### General authentication options + +#### `--auth-duration=` + +The total duration (in hours) of cookie life for authentication. + +Default value: `720h` + +Authentication expires on browser close when `--auth-duration=0`. + +Environment variable: `$AUTH_DURATION` + +#### `--inactivity-duration=` + +The duration that a token is valid without any new activity. + +Default value: `5m` + +Environment variable: `$INACTIVITY_DURATION` + +#### `--public-url=` + +The public URL required to access Chronograf using a web browser. For example, if you access Chronograf using the default URL, the public URL value would be `http://localhost:8888`. +Required for Google OAuth 2.0 authentication. Used for Auth0 and some generic OAuth 2.0 authentication providers. + +Environment variable: `$PUBLIC_URL` + +#### `--token-secret=` | `-t` + +The secret for signing tokens. + +Environment variable: `$TOKEN_SECRET` + +### GitHub-specific OAuth 2.0 authentication options + +See [Configuring GitHub authentication](/chronograf/v1.9/administration/managing-security/#configure-github-authentication) for more information. + +#### `--github-url` + +{{< req "Required if using Github Enterprise" >}} +GitHub base URL. Default is `https://github.com`. + +Environment variable: `$GH_URL` + +#### `--github-client-id` | `-i` + +The GitHub client ID value for OAuth 2.0 support. + +Environment variable: `$GH_CLIENT_ID` + +#### `--github-client-secret` | `-s` + +The GitHub Client Secret value for OAuth 2.0 support. + +Environment variable: `$GH_CLIENT_SECRET` + +#### `--github-organization` | `-o` + +[Optional] Specify a GitHub organization membership required for a user. + +##### CLI example + +```sh +## Single parameter +--github-organization=org1 + +## Mutiple parameters +--github-organization=org1 \ +--github-organization=org2 \ +--github-organization=org3 +``` + +Environment variable: `$GH_ORGS` + +##### Environment variable example + +```sh +## Single parameter +GH_ORGS=org1 + +## Mutiple parameters +GH_ORGS=org1,org2,org3 +``` + +### Google-specific OAuth 2.0 authentication options + +See [Configuring Google authentication](/chronograf/v1.9/administration/managing-security/#configure-google-authentication) for more information. + +#### `--google-client-id=` + +The Google Client ID value required for OAuth 2.0 support. + +Environment variable: `$GOOGLE_CLIENT_ID` + +#### `--google-client-secret=` + +The Google Client Secret value required for OAuth 2.0 support. + +Environment variable: `$GOOGLE_CLIENT_SECRET` + +#### `--google-domains=` + +[Optional] Restricts authorization to users from specified Google email domains. + +##### CLI example + +```sh +## Single parameter +--google-domains=delorean.com + +## Mutiple parameters +--google-domains=delorean.com \ +--google-domains=savetheclocktower.com +``` + +Environment variable: `$GOOGLE_DOMAINS` + +##### Environment variable example + +```sh +## Single parameter +GOOGLE_DOMAINS=delorean.com + +## Mutiple parameters +GOOGLE_DOMAINS=delorean.com,savetheclocktower.com +``` + +### Auth0-specific OAuth 2.0 authentication options + +See [Configuring Auth0 authentication](/chronograf/v1.9/administration/managing-security/#configure-auth0-authentication) for more information. + +#### `--auth0-domain=` + +The subdomain of your Auth0 client; available on the configuration page for your Auth0 client. + +Example: https://myauth0client.auth0.com + +Environment variable: `$AUTH0_DOMAIN` + +#### `--auth0-client-id=` + +The Auth0 Client ID value required for OAuth 2.0 support. + +Environment variable: `$AUTH0_CLIENT_ID` + +#### `--auth0-client-secret=` + +The Auth0 Client Secret value required for OAuth 2.0 support. + +Environment variable: `$AUTH0_CLIENT_SECRET` + +#### `--auth0-organizations=` + +[Optional] The Auth0 organization membership required to access Chronograf. +Organizations are set using an "organization" key in the user's `app_metadata`. +Lists are comma-separated and are only available when using environment variables. + +##### CLI example + +```sh +## Single parameter +--auth0-organizations=org1 + +## Mutiple parameters +--auth0-organizations=org1 \ +--auth0-organizations=org2 \ +--auth0-organizations=org3 +``` + +Environment variable: `$AUTH0_ORGS` + +##### Environment variable example + +```sh +## Single parameter +AUTH0_ORGS=org1 + +## Mutiple parameters +AUTH0_ORGS=org1,org2,org3 +``` + +### Heroku-specific OAuth 2.0 authentication options + +See [Configuring Heroku authentication](/chronograf/v1.9/administration/managing-security/#configure-heroku-authentication) for more information. + +### `--heroku-client-id=` +The Heroku Client ID for OAuth 2.0 support. + +**Environment variable:** `$HEROKU_CLIENT_ID` + +### `--heroku-secret=` +The Heroku Secret for OAuth 2.0 support. + +**Environment variable:** `$HEROKU_SECRET` + +### `--heroku-organization=` +The Heroku organization memberships required for access to Chronograf. + +##### CLI example + +```sh +## Single parameter +--heroku-organization=org1 + +## Mutiple parameters +--heroku-organization=org1 \ +--heroku-organization=org2 \ +--heroku-organization=org3 +``` + +**Environment variable:** `$HEROKU_ORGS` + +##### Environment variable example + +```sh +## Single parameter +HEROKU_ORGS=org1 + +## Mutiple parameters +HEROKU_ORGS=org1,org2,org3 +``` + +### Generic OAuth 2.0 authentication options + +See [Configure OAuth 2.0](/chronograf/v1.9/administration/managing-security/#configure-oauth-2-0) for more information. + +#### `--generic-name=` + +The generic OAuth 2.0 name presented on the login page. + +Environment variable: `$GENERIC_NAME` + +#### `--generic-client-id=` + +The generic OAuth 2.0 Client ID value. +Can be used for a custom OAuth 2.0 service. + +Environment variable: `$GENERIC_CLIENT_ID` + +#### `--generic-client-secret=` + +The generic OAuth 2.0 Client Secret value. + +Environment variable: `$GENERIC_CLIENT_SECRET` + +#### `--generic-scopes=` + +The scopes requested by provider of web client. + +Default value: `user:email` + +##### CLI example + +```sh +## Single parameter +--generic-scopes=api + +## Mutiple parameters +--generic-scopes=api \ +--generic-scopes=openid \ +--generic-scopes=read_user +``` + +Environment variable: `$GENERIC_SCOPES` + +##### Environment variable example + +```sh +## Single parameter +GENERIC_SCOPES=api + +## Mutiple parameters +GENERIC_SCOPES=api,openid,read_user +``` + +#### `--generic-domains=` + +The email domain required for user email addresses. + +Example: `--generic-domains=example.com` + +##### CLI example + +```sh +## Single parameter +--generic-domains=delorean.com + +## Mutiple parameters +--generic-domains=delorean.com \ +--generic-domains=savetheclocktower.com +``` + +Environment variable: `$GENERIC_DOMAINS` + +##### Environment variable example + +```sh +## Single parameter +GENERIC_DOMAINS=delorean.com + +## Mutiple parameters +GENERIC_DOMAINS=delorean.com,savetheclocktower.com +``` + +#### `--generic-auth-url` + +The authorization endpoint URL for the OAuth 2.0 provider. + +Environment variable: `$GENERIC_AUTH_URL` + +#### `--generic-token-url` + +The token endpoint URL for the OAuth 2.0 provider. + +Environment variable: `$GENERIC_TOKEN_URL` + +#### `--generic-api-url` + +The URL that returns OpenID UserInfo-compatible information. + +Environment variable: `$GENERIC_API_URL` + +#### `--oauth-no-pkce` + +Disable OAuth PKCE (Proof Key for Code Exchange). + +Environment variable: `$OAUTH_NO_PKCE` diff --git a/content/chronograf/v1.9/administration/configuration.md b/content/chronograf/v1.9/administration/configuration.md new file mode 100644 index 000000000..8c5e85ab6 --- /dev/null +++ b/content/chronograf/v1.9/administration/configuration.md @@ -0,0 +1,70 @@ +--- +title: Configure Chronograf +description: > + Configure Chronograf, including security, multiple users, and multiple organizations. +menu: + chronograf_1_9: + name: Configure + weight: 20 + parent: Administration +--- + +Configure Chronograf by passing command line options when starting the Chronograf service. Or set custom default configuration options in the filesystem so they don’t have to be passed in when starting Chronograf. + +- [Start the Chronograf service](#start-the-chronograf-service) +- [Set custom default Chronograf configuration options](#set-custom-default-chronograf-configuration-options) +- [Set up security, organizations, and users](#set-up-security-organizations-and-users) + +## Start the Chronograf service + +Use one of the following commands to start Chronograf: + +- **If you installed Chronograf using an official Debian or RPM package and are running a distro with `systemd`. For example, Ubuntu 15 or later.** + + ```sh + systemctl start chronograf + ``` + +- **If you installed Chronograf using an official Debian or RPM package:** + + ```sh + service chronograf start + ``` + +- **If you built Chronograf from source:** + + ```bash + $GOPATH/bin/chronograf + ``` + +## Set custom default Chronograf configuration options + +Custom default Chronograf configuration settings can be defined in `/etc/default/chronograf`. +This file consists of key-value pairs. See keys (environment variables) for [Chronograf configuration options](/chronograf/v1.9/administration/config-options), and set values for the keys you want to configure. + +```conf +HOST=0.0.0.0 +PORT=8888 +TLS_CERTIFICATE=/path/to/cert.pem +TOKEN_SECRET=MySup3rS3cretT0k3n +LOG_LEVEL=info +``` + +{{% note %}} +**Note:** `/etc/default/chronograf` is only created when installing the `.deb or `.rpm` package. +{{% /note %}} + +## Set up security, organizations, and users + +To set up security for Chronograf, configure: + +* [OAuth 2.0 authentication](/chronograf/v1.9/administration/managing-security/#configure-oauth-2-0) +* [TLS (Transport Layer Security) for HTTPS](/chronograf/v1.9/administration/managing-security/#configure-tls-transport-layer-security-and-https) + +After you configure OAuth 2.0 authentication, you can set up multiple organizations, roles, and users. For details, check out the following topics: + +* [Managing organizations](/chronograf/v1.9/administration/managing-organizations/) +* [Managing Chronograf users](/chronograf/v1.9/administration/managing-chronograf-users/) + + + diff --git a/content/chronograf/v1.9/administration/create-high-availability.md b/content/chronograf/v1.9/administration/create-high-availability.md new file mode 100644 index 000000000..0d110b839 --- /dev/null +++ b/content/chronograf/v1.9/administration/create-high-availability.md @@ -0,0 +1,72 @@ +--- +title: Create a Chronograf HA configuration +description: Create a Chronograf high-availability (HA) cluster using etcd. +menu: + chronograf_1_9: + weight: 10 + parent: Administration +--- + +To create a Chronograf high-availability (HA) configuration using an etcd cluster as a shared data store, do the following: + +1. [Install and start etcd](#install-and-start-etcd) +2. Set up a load balancer for Chronograf +3. [Start Chronograf](#start-chronograf) + +Have an existing Chronograf configuration store that you want to use with a Chronograf HA configuration? Learn how to [migrate your Chrongraf configuration](/chronograf/v1.9/administration/migrate-to-high-availability/) to a shared data store. + +## Architecture + +{{< svg "/static/img/chronograf/1-8-ha-architecture.svg" >}} + +## Install and start etcd + +1. Download the latest etcd release [from GitHub](https://github.com/etcd-io/etcd/releases/). + (For detailed installation instructions specific to your operating system, see [Install and deploy etcd](http://play.etcd.io/install).) +2. Extract the `etcd` binary and place it in your system PATH. +3. Start etcd. + + +## Start Chronograf + +Run the following command to start Chronograf using `etcd` as the storage layer. The syntax depends on whether you're using command line flags or the `ETCD_ENDPOINTS` environment variable. + +##### Define etcd endpoints with command line flags +```sh +# Syntax +chronograf --etcd-endpoints= +# Examples + +# Add a single etcd endpoint when starting Chronograf + +chronograf --etcd-endpoints=localhost:2379 + +# Add multiple etcd endpoints when starting Chronograf +chronograf \ + --etcd-endpoints=localhost:2379 \ + --etcd-endpoints=192.168.1.61:2379 \ + --etcd-endpoints=192.192.168.1.100:2379 +``` + +##### Define etcd endpoints with the ETCD_ENDPOINTS environment variable +```sh + +# Provide etcd endpoints in a comma-separated list +export ETCD_ENDPOINTS=localhost:2379,192.168.1.61:2379,192.192.168.1.100:2379 + +# Start Chronograf +chronograf +``` + +##### Define etcd endpoints with TLS enabled +Use the `--etcd-cert` flag to specify the path to the etcd PEM-encoded public +certificate file and the `--etcd-key` flag to specify the path to the private key +associated with the etcd certificate. + +```sh +chronograf --etcd-endpoints=localhost:2379 \ + --etcd-cert=path/to/etcd-certificate.pem \ + --etcd-key=path/to/etcd-private-key.key +``` + +For more information, see [Chronograf etcd configuration options](/chronograf/v1.9/administration/config-options#etcd-options). diff --git a/content/chronograf/v1.9/administration/creating-connections.md b/content/chronograf/v1.9/administration/creating-connections.md new file mode 100644 index 000000000..ee63d8405 --- /dev/null +++ b/content/chronograf/v1.9/administration/creating-connections.md @@ -0,0 +1,289 @@ +--- +title: Create InfluxDB and Kapacitor connections +description: Create and manage InfluxDB and Kapacitor connections in the UI. +menu: + chronograf_1_9: + name: Create InfluxDB and Kapacitor connections + weight: 50 + parent: Administration +related: + - /influxdb/v2.0/tools/chronograf/ +--- + +Connections to InfluxDB and Kapacitor can be configured through the Chronograf user interface (UI) or with JSON configuration files: + +- [Manage InfluxDB connections using the Chronograf UI](#manage-influxdb-connections-using-the-chronograf-ui) +- [Manage InfluxDB connections using .src files](#manage-influxdb-connections-using-src-files) +- [Manage Kapacitor connections using the Chronograf UI](#manage-kapacitor-connections-using-the-chronograf-ui) +- [Manage Kapacitor connections using .kap files](#manage-kapacitor-connections-using-kap-files) + +{{% note %}} +**Note:** Connection details are stored in Chronograf’s internal database `chronograf-v1.db`. +You may administer the internal database when [restoring a Chronograf database](/chronograf/v1.9/administration/restoring-chronograf-db/) +or when [migrating a Chronograf configuration from BoltDB to etcd](/chronograf/v1.9/administration/migrate-to-high-availability/). +{{% /note %}} + +## Manage InfluxDB connections using the Chronograf UI + +To create an InfluxDB connection in the Chronograf UI: + +1. Open Chronograf and click **Configuration** (wrench icon) in the navigation menu. +2. Click **Add Connection**. + + ![Chronograf connections landing page](/img/chronograf/1-6-connection-landing-page.png) + +3. Provide the necessary connection credentials. + + {{< tabs-wrapper >}} + {{% tabs %}} +[InfluxDB 1.x](#) +[InfluxDB Cloud or OSS 2.x ](#) + {{% /tabs %}} + {{% tab-content %}} + + +- **Connection URL**: hostname or IP address and port of the InfluxDB 1.x instance +- **Connection Name**: Unique name for this connection. +- **Username**: InfluxDB 1.x username + _(Required only if [authorization is enabled](/{{< latest "influxdb" "v1" >}}/administration/authentication_and_authorization/) in InfluxDB)_ +- **Password**: InfluxDB password + _(Required only if [authorization is enabled](/{{< latest "influxdb" "v1" >}}/administration/authentication_and_authorization/) in InfluxDB)_ +- **Telegraf Database Name**: the database Chronograf uses to populate parts of the application, including the Host List page (default is `telegraf`) +- **Default Retention Policy**: default [retention policy](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#retention-policy-rp) + (if left blank, defaults to `autogen`) +- **Default connection**: use this connection as the default connection for data exploration, dashboards, and administrative actions + {{% /tab-content %}} + {{% tab-content %}} + + +- **Enable the {{< req "InfluxDB v2 Auth" >}} option** +- **Connection URL**: [InfluxDB Cloud region URL](/influxdb/cloud/reference/regions/) + or [InfluxDB OSS 2.x URL](/influxdb/v2.0/reference/urls/) + + ``` + http://localhost:8086 + ``` + +- **Connection Name**: Unique name for this connection. +- **Organiziation**: InfluxDB [organization](/influxdb/v2.0/organizations/) +- **Token**: InfluxDB [authentication token](/influxdb/v2.0/security/tokens/) +- **Telegraf Database Name:** InfluxDB [bucket](/influxdb/v2.0/organizations/buckets/) + Chronograf uses to populate parts of the application, including the Host List page (default is `telegraf`) +- **Default Retention Policy:** default [retention policy](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#retention-policy-rp) + _**(leave blank)**_ +- **Default connection**: use this connection as the default connection for data exploration and dashboards + +{{% note %}} +For more information about connecting Chronograf to an InfluxDB Cloud or OSS 2.x instance, see: + +- [Use Chronograf with InfluxDB Cloud](/influxdb/cloud/tools/chronograf/) +- [Use Chronograf with InfluxDB OSS 2.x](/{{< latest "influxdb" "v2" >}}/tools/chronograf/) +{{% /note %}} + {{% /tab-content %}} + {{< /tabs-wrapper >}} + +4. Click **Add Connection** + * If the connection is valid, the Dashboards window appears, allowing you to import dashboard templates you can use to display and analyze your data. For details, see [Creating dashboards](/chronograf/v1.9/guides/create-a-dashboard). + * If the connection cannot be created, the following error message appears: + "Unable to create source: Error contacting source." + If this occurs, ensure all connection credentials are correct and that the InfluxDB instance is running and accessible. + +The following dashboards are available: + +- Docker +- Kubernetes Node +- Riak +- Consul +- Kubernetes Overview +- Mesos +- IIS +- RabbitMQ +- System +- VMware vSphere Overview +- Apache +- Elastisearch +- InfluxDB +- Memcached +- NSQ +- PostgreSQL +- Consul Telemetry +- HAProxy +- Kubernetes Pod +- NGINX +- Redis +- VMware vSphere VMs +- VMware vSphere Hosts +- PHPfpm +- Win System +- MySQL +- Ping + +## Manage InfluxDB connections using .src files + +Manually create `.src` files to store InfluxDB connection details. +`.src` files are simple JSON files that contain key-value paired connection details. +The location of `.src` files is defined by the [`--resources-path`](/chronograf/v1.9/administration/config-options/#resources-path) +command line option, which is, by default, the same as the [`--canned-path`](/chronograf/v1.9/administration/config-options/#canned-path-c). +A `.src` file contains the details for a single InfluxDB connection. + +{{% note %}} +**Only InfluxDB 1.x connections are configurable in a `.src` file.** +Configure InfluxDB 2.x and Cloud connections with [CLI flags](/chronograf/v1.9/administration/config-options/#influxdb-connection-options) +or in the [Chronograf UI](#manage-influxdb-connections-using-the-chronograf-ui). +{{% /note %}} + +Create a new file named `example.src` (the filename is arbitrary) and place it at Chronograf's `resource-path`. +All `.src` files should contain the following: + +{{< keep-url >}} +```json +{ + "id": "10000", + "name": "My InfluxDB", + "username": "test", + "password": "test", + "url": "http://localhost:8086", + "type": "influx", + "insecureSkipVerify": false, + "default": true, + "telegraf": "telegraf", + "organization": "example_org" +} +``` + +#### `id` +A unique, stringified non-negative integer. +Using a 4 or 5 digit number is recommended to avoid interfering with existing datasource IDs. + +#### `name` +Any string you want to use as the display name of the source. + +#### `username` +Username used to access the InfluxDB server or cluster. +*Only required if [authorization is enabled](/{{< latest "influxdb" "v1" >}}/administration/authentication_and_authorization/) on the InfluxDB instance to which you're connecting.* + +#### `password` +Password used to access the InfluxDB server or cluster. +*Only required if [authorization is enabled](/{{< latest "influxdb" "v1" >}}/administration/authentication_and_authorization/) on the InfluxDB instance to which you're connecting.* + +#### `url` +URL of the InfluxDB server or cluster. + +#### `type` +Defines the type or distribution of InfluxDB to which you are connecting. +Below are the following options: + +| InfluxDB Distribution | `type` Value | +| --------------------- | ------------ | +| InfluxDB OSS | `influx` | +| InfluxDB Enterprise | `influx-enterprise` | + +#### `insecureSkipVerify` +Skips the SSL certificate verification process. +Set to `true` if you are using a self-signed SSL certificate on your InfluxDB server or cluster. + +#### `default` +Set to `true` if you want the connection to be the default data connection used upon first login. + +#### `telegraf` +The name of the Telegraf database on your InfluxDB server or cluster. + +#### `organization` +The ID of the organization you want the data source to be associated with. + +### Environment variables in .src files +`.src` files support the use of environment variables to populate InfluxDB connection details. +Environment variables can be loaded using the `"{{ .VARIABLE_KEY }}"` syntax: + +```JSON +{ + "id": "10000", + "name": "My InfluxDB", + "username": "{{ .INFLUXDB_USER }}", + "password": "{{ .INFLUXDB_PASS }}", + "url": "{{ .INFLUXDB_URL }}", + "type": "influx", + "insecureSkipVerify": false, + "default": true, + "telegraf": "telegraf", + "organization": "example_org" +} +``` + +## Manage Kapacitor connections using the Chronograf UI + +Kapacitor is the data processing component of the TICK stack. +To use Kapacitor in Chronograf, create Kapacitor connections and configure alert endpoints. +To create a Kapacitor connection using the Chronograf UI: + +1. Open Chronograf and click **Configuration** (wrench icon) in the navigation menu. +2. Next to an existing [InfluxDB connection](#manage-influxdb-connections-using-the-chronograf-ui), click **Add Kapacitor Connection** if there are no existing Kapacitor connections or select **Add Kapacitor Connection** in the **Kapacitor Connection** dropdown list. + ![Add a new Kapacitor connection in Chronograf](/img/chronograf/1-6-connection-kapacitor.png) + +3. In the **Connection Details** section, enter values for the following fields: + + + + * **Kapacitor URL**: Enter the hostname or IP address of the Kapacitor instance and the port. The field is prefilled with `http://localhost:9092`. + * **Name**: Enter the name for this connection. + * **Username**: Enter the username that will be shared for this connection. + *Only required if [authorization is enabled](/{{< latest "kapacitor" >}}/administration/security/#kapacitor-authentication-and-authorization) on the Kapacitor instance or cluster to which you're connecting.* + * **Password**: Enter the password. + *Only required if [authorization is enabled](/{{< latest "kapacitor" >}}/administration/security/#kapacitor-authentication-and-authorization) on the Kapacitor instance or cluster to which you're connecting.* + +4. Click **Continue**. If the connection is valid, the message "Kapacitor Created! Configuring endpoints is optional." appears. To configure alert endpoints, see [Configuring alert endpoints](/chronograf/v1.9/guides/configuring-alert-endpoints/). + +## Manage Kapacitor connections using .kap files + +Manually create `.kap` files to store Kapacitor connection details. +`.kap` files are simple JSON files that contain key-value paired connection details. +The location of `.kap` files is defined by the `--resources-path` command line option, which is, by default, the same as the [`--canned-path`](/chronograf/v1.9/administration/config-options/#canned-path-c). +A `.kap` files contains the details for a single InfluxDB connection. + +Create a new file named `example.kap` (the filename is arbitrary) and place it at Chronograf's `resource-path`. +All `.kap` files should contain the following: + +```json +{ + "id": "10000", + "srcID": "10000", + "name": "My Kapacitor", + "url": "http://localhost:9092", + "active": true, + "organization": "example_org" +} +``` + +#### `id` +A unique, stringified non-negative integer. +Using a 4 or 5 digit number is recommended to avoid interfering with existing datasource IDs. + +#### `srcID` +The unique, stringified non-negative integer `id` of the InfluxDB server or cluster with which the Kapacitor service is associated. + +#### `name` +Any string you want to use as the display name of the Kapacitor connection. + +#### `url` +URL of the Kapacitor server. + +#### `active` +If `true`, specifies that this is the Kapacitor connection that should be used when displaying Kapacitor-related information in Chronograf. + +#### `organization` +The ID of the organization you want the Kapacitor connection to be associated with. + +### Environment variables in .kap files +`.kap` files support the use of environment variables to populate Kapacitor connection details. +Environment variables can be loaded using the `"{{ .VARIABLE_KEY }}"` syntax: + +```JSON +{ + "id": "10000", + "srcID": "10000", + "name": "My Kapacitor", + "url": "{{ .KAPACITOR_URL }}", + "active": true, + "organization": "example_org" +} +``` diff --git a/content/chronograf/v1.9/administration/import-export-dashboards.md b/content/chronograf/v1.9/administration/import-export-dashboards.md new file mode 100644 index 000000000..662f9a2c0 --- /dev/null +++ b/content/chronograf/v1.9/administration/import-export-dashboards.md @@ -0,0 +1,60 @@ +--- +title: Import and export Chronograf dashboards +description: Share dashboard JSON files between Chronograf instances, or add dashboards as resources to include in a deployment. +menu: + chronograf_1_9: + weight: 120 + parent: Administration +--- + +Chronograf makes it easy to recreate robust dashboards without having to manually configure them from the ground up. Import and export dashboards between instances, or add dashboards as resources to include in a deployment. + +- [Export a dashboard](#export-a-dashboard) +- [Load a dashboard as a resource](#load-a-dashboard-as-a-resource) +- [Import a dashboard](#import-a-dashboard) +- [Required user roles](#required-user-roles) + +## Required user roles + +All users can export a dashboard. To import a dashboard, a user must have an Admin or Editor role. + +| Task vs Role | Admin | Editor | Viewer | +|------------------|:-----:|:------:|:------:| +| Export Dashboard | ✅ | ✅ | ✅ | +| Import Dashboard | ✅ | ✅ | ❌ | + +## Export a dashboard + +1. On the Dashboards page, hover over the dashboard you want to export, and then click the **Export** + button on the right. + + Exporting a Chronograf dashboard + + This downloads a JSON file containing dashboard information including template variables, cells and cell information such as the query, cell-sizing, color scheme, visualization type, etc. + + > No time series data is exported with a dashboard. + > Exports include only dashboard-related information as mentioned above. + +## Load a dashboard as a resource + +Automatically load the dashboard as a resource (useful for adding a dashboard to a deployment). + +1. Rename the dashboard `.json` extension to `.dashboard`. +2. Use the [`resources-path` configuration option](/chronograf/v1.9/administration/config-options/#--resources-path) to save the dashboard in the `/resources` directory (by default, `/usr/share/chronograf/resources`). + +## Import a dashboard + +1. On your Dashboards page, click the **Import Dashboard** button. +2. Either drag and drop or select the JSON export file to import. +3. Click the **Upload Dashboard** button. + +The newly imported dashboard is included in your list of dashboards. + +![Importing a Chronograf dashboard](/img/chronograf/1-6-dashboard-import.gif) + +### Reconcile unmatched sources + +If the data sources defined in the imported dashboard file do not match any of your local sources, +reconcile each of the unmatched sources during the import process, and then click **Done**. + +![Reconcile unmatched sources](/img/chronograf/1-6-dashboard-import-reconcile.png) diff --git a/content/chronograf/v1.9/administration/managing-chronograf-users.md b/content/chronograf/v1.9/administration/managing-chronograf-users.md new file mode 100644 index 000000000..c1f005595 --- /dev/null +++ b/content/chronograf/v1.9/administration/managing-chronograf-users.md @@ -0,0 +1,252 @@ +--- +title: Manage Chronograf users +description: > + Manage users and roles, including SuperAdmin permission and organization-bound users. +menu: + chronograf_1_9: + name: Manage Chronograf users + weight: 90 + parent: Administration +--- + +**On this page** + +* [Manage Chronograf users and roles](#manage-chronograf-users-and-roles) +* [Organization-bound users](#organization-bound-users) + * [InfluxDB and Kapacitor users within Chronograf](#influxdb-and-kapacitor-users-within-chronograf) + * [Chronograf-owned resources](#chronograf-owned-resources) + * [Chronograf-accessed resources](#chronograf-accessed-resources) + * [Members](#members-role-member) + * [Viewers](#viewers-role-viewer) + * [Editors](#editors-role-editor) + * [Admins](#admins-role-admin) +* [Cross-organization SuperAdmin permission](#cross-organization-superadmin-permission) + * [All New Users are SuperAdmins configuration option](#all-new-users-are-superadmins-configuration-option) +* [Create users](#create-users) +* [Update users](#update-users) +* [Remove users](#remove-users) +* [Navigate organizations](#navigate-organizations) + * [Log in and log out](#log-in-and-log-out) + * [Switch the current organization](#switch-the-current-organization) + * [Purgatory](#purgatory) + +## Manage Chronograf users and roles + +{{% note %}} +**Note:** Support for organizations and user roles is available in Chronograf 1.4 or later. +First, OAuth 2.0 authentication must be configured (if it is, you'll see the +Chronograf Admin tab on the Admin menu). +For more information, see [Managing security](/chronograf/v1.9/administration/managing-security/). +{{% /note %}} + +Chronograf includes four organization-bound user roles and one cross-organization SuperAdmin permission. In an organization, admins (with the `admin` role) or users with SuperAdmin permission can create, update, and assign roles to a user or remove a role assignment. + +### Organization-bound users + +Chronograf users are assigned one of the following four organization-bound user roles, listed here in order of increasing capabilities: + +- [`member`](#members-role-member) +- [`viewer`](#viewers-role-viewer) +- [`editor`](#editors-role-editor) +- [`admin`](#admins-role-admin) + +Each of these four roles, described in detail below, have different capabilities for the following Chronograf-owned or Chronograf-accessed resources. + +#### InfluxDB and Kapacitor users within Chronograf + +Chronograf uses InfluxDB and Kapacitor connections to manage user access control to InfluxDB and Kapacitor resources within Chronograf. The permissions of the InfluxDB and Kapacitor user specified within such a connection determine the capabilities for any Chronograf user with access (i.e., viewers, editors, and administrators) to that connection. Administrators include either an admin (`admin` role) or a user of any role with SuperAdmin permission. + +{{% note %}} +**Note:** Chronograf users are entirely separate from InfluxDB and Kapacitor users. +The Chronograf user and authentication system applies to the Chronograf user interface. +InfluxDB and Kapacitor users and their permissions are managed separately. +[Chronograf connections](/chronograf/v1.9/administration/creating-connections/) +determine which InfluxDB or Kapacitor users to use when when connecting to each service. +{{% /note %}} + +#### Chronograf-owned resources + +Chronograf-owned resources include internal resources that are under the full control of Chronograf, including: + +- Kapacitor connections +- InfluxDB connections +- Dashboards +- Canned layouts +- Chronograf organizations +- Chronograf users +- Chronograf Status Page content for News Feeds and Getting Started + +#### Chronograf-accessed resources + +Chronograf-accessed resources include external resources that can be accessed using Chronograf, but are under limited control by Chronograf. Chronograf users with the roles of `viewer`, `editor`, and `admin`, or users with SuperAdmin permission, have equal access to these resources: + +- InfluxDB databases, users, queries, and time series data (if using InfluxDB Enterprise, InfluxDB roles can be accessed too) +- Kapacitor alerts and alert rules (called tasks in Kapacitor) + +#### Members (role:`member`) + +Members are Chronograf users who have been added to organizations but do not have any functional capabilities. Members cannot access any resources within an organization and thus effectively cannot use Chronograf. Instead, a member can only access Purgatory, where the user can [switch into organizations](#navigate-organizations) based on assigned roles. + +By default, new organizations have a default role of `member`. If the Default organization is Public, then anyone who can authenticate, would become a member, but not be able to use Chronograf until an administrator assigns a different role. + +#### Viewers (role:`viewer`) + +Viewers are Chronograf users with effectively read-only capabilities for Chronograf-owned resources within their current organization: + +- View canned dashboards +- View canned layouts +- View InfluxDB connections +- Switch current InfluxDB connection to other available connections +- Access InfluxDB resources through the current connection +- View the name of the current Kapacitor connection associated with each InfluxDB connection +- Access Kapacitor resources through the current connection +- [Switch into organizations](#navigate-organizations) where the user has a role + +For Chronograf-accessed resources, viewers can: + +- InfluxDB + - Read and write time series data + - Create, view, edit, and delete databases and retention policies + - Create, view, edit, and delete InfluxDB users + - View and kill queries + - _InfluxDB Enterprise_: Create, view, edit, and delete InfluxDB Enterprise roles +- Kapacitor + - View alerts + - Create, edit, and delete alert rules + +#### Editors (role:`editor`) + +Editors are Chronograf users with limited capabilities for Chronograf-owned resources within their current organization: + +- Create, view, edit, and delete dashboards +- View canned layouts +- Create, view, edit, and delete InfluxDB connections +- Switch current InfluxDB connection to other available connections +- Access InfluxDB resources through the current connection +- Create, view, edit, and delete Kapacitor connections associated with InfluxDB connections +- Switch current Kapacitor connection to other available connections +- Access Kapacitor resources through the current connection +- [Switch into organizations](#navigate-organizations) where the user has a role + +For Chronograf-accessed resources, editors can: + +- InfluxDB + - Read and write time series data + - Create, view, edit, and delete databases and retention policies + - Create, view, edit, and delete InfluxDB users + - View and kill queries + - _InfluxDB Enterprise_: Create, view, edit, and delete InfluxDB Enterprise roles +- Kapacitor + - View alerts + - Create, edit, and delete alert rules + +#### Admins (role:`admin`) + +Admins are Chronograf users with all capabilities for the following Chronograf-owned resources within their current organization: + +- Create, view, update, and remove Chronograf users +- Create, view, edit, and delete dashboards +- View canned layouts +- Create, view, edit, and delete InfluxDB connections +- Switch current InfluxDB connection to other available connections +- Access InfluxDB resources through the current connection +- Create, view, edit, and delete Kapacitor connections associated with InfluxDB connections +- Switch current Kapacitor connection to other available connections +- Access Kapacitor resources through the current connection +- [Switch into organizations](#navigate-organizations) where the user has a role + +For Chronograf-accessed resources, admins can: + +- InfluxDB + - Read and write time series data + - Create, view, edit, and delete databases and retention policies + - Create, view, edit, and delete InfluxDB users + - View and kill queries + - _InfluxDB Enterprise_: Create, view, edit, and delete InfluxDB Enterprise roles +- Kapacitor + - View alerts + - Create, edit, and delete alert rules + +### Cross-organization SuperAdmin permission + +SuperAdmin permission is a Chronograf permission that allows any user, regardless of role, to perform all administrator functions both within organizations, as well as across organizations. A user with SuperAdmin permission has _unlimited_ capabilities, including for the following Chronograf-owned resources: + +* Create, view, update, and remove organizations +* Create, view, update, and remove users within an organization +* Grant or revoke the SuperAdmin permission of another user +* [Switch into any organization](#navigate-organizations) +* Toggle the Public setting of the Default organization +* Toggle the global config setting for [All new users are SuperAdmin](#all-new-users-are-superadmins-configuration-option) + +Important SuperAdmin behaviors: + +* SuperAdmin permission grants any user (whether `member`, `viewer`, `editor`, or `admin`) the full capabilities of admins and the SuperAdmin capabilities listed above. +* When a Chronograf user with SuperAdmin permission creates a new organization or switches into an organization where that user has no role, that SuperAdmin user is automatically assigned the `admin` role by default. +* SuperAdmin users cannot revoke their own SuperAdmin permission. +* SuperAdmin users are the only ones who can change the SuperAdmin permission of other Chronograf users. Regular admins who do not have SuperAdmin permission can perform normal operations on SuperAdmin users (create that user within their organization, change roles, and remove them), but they will not see that these users have SuperAdmin permission, nor will any of their actions affect the SuperAdmin permission of these users. +* If a user has their SuperAdmin permission revoked, that user will retain their assigned roles within their organizations. + +#### All New Users are SuperAdmins configuration option + +By default, the **Config** setting for "**All new users are SuperAdmins"** is **On**. Any user with SuperAdmin permission can toggle this under the **Admin > Chronograf > Organizations** tab. If this setting is **On**, any new user (who is created or who authenticates) will_ automatically have SuperAdmin permisison. If this setting is **Off**, any new user (who is created or who authenticates) will _not_ have SuperAdmin permisison unless they are explicitly granted it later by another user with SuperAdmin permission. + +### Create users + +Role required: `admin` + +**To create a user:** + +1. Open Chronograf in your web browser and select **Admin (crown icon) > Chronograf**. +2. Click the **Users** tab and then click **Create User**. +3. Add the following user information: + * **Username**: Enter the username as provided by the OAuth provider. + * **Role**: Select the Chronograf role. + * **Provider**: Enter the OAuth 2.0 provider to be used for authentication. Valid values are: `github`, `google`, `auth0`, `heroku`, or other names defined in the [`GENERIC_NAME` environment variable](/chronograf/v1.9/administration/config-options#generic-name). + * **Scheme**: Displays `oauth2`, which is the only supported authentication scheme in this release. +4. Click **Save** to finish creating the user. + +### Update users + +Role required: `admin` + +Only a user's role can be updated. A user's username, provider, and scheme cannot be updated. (Effectively, to "update" a user's username, provider, or scheme, the user must be removed and added again with the desired values.) + +**To change a user's role:** + +1. Open Chronograf in your web browser and select **Admin (crown icon) > Chronograf**. +2. Click the **Users** tab to display the list of users within the current organization. +3. Select a new role for the user. The update is automatically persisted. + +### Remove users + +Role required: `admin` + +**To remove a user:** + +1. Open Chronograf in your web browser and select **Admin (crown icon) > Chronograf**. +2. Click the **Users** tab to display the list of users. +3. Hover your cursor over the user you want to remove and then click **Remove** and **Confirm**. + +### Navigate organizations + +Chronograf is always used in the context of an organization. When a user logs in to Chronograf, that user will access only the resources owned by their current organization. The only exception to this is that users with SuperAdmin permission will also be able to [manage organizations](/chronograf/v1.9/administration/managing-organizations/) in the Chronograf Admin page. + +#### Log in and log out + +A user can log in from the Chronograf homepage using any configured OAuth 2.0 provider. + +A user can log out by hovering over the **User (person icon)** in the left navigation bar and clicking **Log out**. + +#### Switch the current organization + +A user's current organization and role is highlighted in the **Switch Organizations** list, which can be found by hovering over the **User (person icon)** in the left navigation bar. + +When a user has a role in more than one organization, that user can switch into any other organization where they have a role by selecting the desired organization in the **Switch Organizations** list. + +#### Purgatory + +If at any time, a user is a `member` within their current organization and does not have SuperAdmin permission, that user will be redirected to a page called Purgatory. There, the user will see their current organization and role, as well as a message to contact an administrator for access. + +On the same page, that user will see a list of all of their organizations and roles. The user can switch into any listed organization where their role is `viewer`, `editor`, or `admin` by clicking **Log in** next to the desired organization. + +**Note** In the rare case that a user is granted SuperAdmin permission while in Purgatory, they will be able to switch into any listed organization, as expected. diff --git a/content/chronograf/v1.9/administration/managing-influxdb-users.md b/content/chronograf/v1.9/administration/managing-influxdb-users.md new file mode 100644 index 000000000..83d6dd51e --- /dev/null +++ b/content/chronograf/v1.9/administration/managing-influxdb-users.md @@ -0,0 +1,327 @@ +--- +title: Manage InfluxDB users in Chronograf +description: > + Enable authentication and manage InfluxDB OSS and InfluxDB Enterprise users in Chronograf. +aliases: + - /chronograf/v1.9/administration/user-management/ +menu: + chronograf_1_9: + name: Manage InfluxDB users + weight: 60 + parent: Administration +--- + +The **Chronograf Admin** provides InfluxDB user management for InfluxDB OSS and InfluxDB Enterprise users. + +{{% note %}} +***Note:*** For details on Chronograf user authentication and management, see [Managing security](/chronograf/v1.9/administration/managing-security/). +{{% /note %}} + +{{% note %}} +#### Disabled administrative features +If connected to **InfluxDB OSS v2.x** or **InfluxDB Cloud**, all InfluxDB administrative +features are disabled in Chronograf. Use the InfluxDB OSS v2.x or InfluxDB Cloud user +interfaces, CLIs, or APIs to complete administrative tasks. +{{% /note %}} + +**On this page:** + +* [Enable authentication](#enable-authentication) +* [InfluxDB OSS user management](#influxdb-oss-user-management) +* [InfluxDB Enterprise user management](#influxdb-enterprise-user-management) + +## Enable authentication + +Follow the steps below to enable authentication. +The steps are the same for InfluxDB OSS instances and InfluxDB Enterprise clusters. + +{{% note %}} +_**InfluxDB Enterprise clusters:**_ +Repeat the first three steps for each data node in a cluster. +{{% /note %}} + +### Step 1: Enable authentication. + +Enable authentication in the InfluxDB configuration file. +For most Linux installations, the configuration file is located in `/etc/influxdb/influxdb.conf`. + +In the `[http]` section of the InfluxDB configuration file (`influxdb.conf`), uncomment the `auth-enabled` option and set it to `true`, as shown here: + +``` +[http] + # Determines whether HTTP endpoint is enabled. + # enabled = true + + # The bind address used by the HTTP service. + # bind-address = ":8086" + + # Determines whether HTTP authentication is enabled. + auth-enabled = true # +``` + +### Step 2: Restart the InfluxDB service. + +Restart the InfluxDB service for your configuration changes to take effect: + +``` +~# sudo systemctl restart influxdb +``` + +### Step 3: Create an admin user. + +Because authentication is enabled, you need to create an [admin user](/{{< latest "influxdb" "v1" >}}/administration/authentication_and_authorization/#user-types-and-privileges) before you can do anything else in the database. +Run the `curl` command below to create an admin user, replacing: + +* `localhost` with the IP or hostname of your InfluxDB OSS instance or one of your InfluxDB Enterprise data nodes +* `chronothan` with your own username +* `supersecret` with your own password (note that the password requires single quotes) + +{{< keep-url >}} +``` +~# curl -XPOST "http://localhost:8086/query" --data-urlencode "q=CREATE USER chronothan WITH PASSWORD 'supersecret' WITH ALL PRIVILEGES" +``` + +A successful `CREATE USER` query returns a blank result: + +``` +{"results":[{"statement_id":0}]} <--- Success! +``` + +### Step 4: Edit the InfluxDB source in Chronograf. + +If you've already [connected your database to Chronograf](/chronograf/v1.9/introduction/installation/#connect-chronograf-to-your-influxdb-instance-or-influxdb-enterprise-cluster), update the connection configuration in Chronograf with your new username and password. +Edit existing InfluxDB database sources by navigating to the Chronograf configuration page and clicking on the name of the source. + +## InfluxDB OSS User Management + +On the **Chronograf Admin** page: + +* View, create, and delete admin and non-admin users +* Change user passwords +* Assign admin and remove admin permissions to or from a user + +![InfluxDB OSS user management](/img/chronograf/1-6-admin-usermanagement-oss.png) + +InfluxDB users are either admin users or non-admin users. +See InfluxDB's [authentication and authorization](/{{< latest "influxdb" "v1" >}}/administration/authentication_and_authorization/#user-types-and-privileges) documentation for more information about those user types. + +{{% note %}} +Chronograf currently does not support assigning InfluxDB database `READ`or `WRITE` access to non-admin users. + +As a workaround, grant `READ`, `WRITE`, or `ALL` (`READ` and `WRITE`) permissions to non-admin users with the following curl commands, replacing anything inside `< >` with your own values: + +#### Grant `READ` permission: +```sh +curl --request POST "http://:8086/query?u=&p=" \ + --data-urlencode "q=GRANT READ ON TO " +``` + +#### Grant `WRITE` permission: +```sh +curl --request POST "http://:8086/query?u=&p=" \ + --data-urlencode "q=GRANT WRITE ON TO " +``` + +#### Grant `ALL` permission: +```sh +curl --request POST "http://:8086/query?u=&p=" \ + --data-urlencode "q=GRANT ALL ON TO " +``` + +In all cases, a successful `GRANT` query returns a blank result: + +```sh +{"results":[{"statement_id":0}]} # <--- Success! +``` + +Remove `READ`, `WRITE`, or `ALL` permissions from non-admin users by replacing `GRANT` with `REVOKE` in the curl commands above. +{{% /note %}} + +## InfluxDB Enterprise user management + +On the `Admin` page: + +* View, create, and delete users +* Change user passwords +* Assign and remove permissions to or from a user +* Create, edit, and delete roles +* Assign and remove roles to or from a user + +![InfluxDB Enterprise user management](/img/chronograf/1-6-admin-usermanagement-cluster.png) + +### User types + +Admin users have the following permissions by default: + +* [CreateDatabase](#createdatabase) +* [CreateUserAndRole](#createuserandrole) +* [DropData](#dropdata) +* [DropDatabase](#dropdatabase) +* [ManageContinuousQuery](#managecontinuousquery) +* [ManageQuery](#managequery) +* [ManageShard](#manageshard) +* [ManageSubscription](#managesubscription) +* [Monitor](#monitor) +* [ReadData](#readdata) +* [WriteData](#writedata) + +Non-admin users have no permissions by default. +Assign permissions and roles to both admin and non-admin users. + +### Permissions + +#### AddRemoveNode +Permission to add or remove nodes from a cluster. + +**Relevant `influxd-ctl` arguments**: +[`add-data`](/{{< latest "enterprise_influxdb" >}}/administration/cluster-commands/#add-data), +[`add-meta`](/{{< latest "enterprise_influxdb" >}}/administration/cluster-commands/#add-meta), +[`join`](/{{< latest "enterprise_influxdb" >}}/administration/cluster-commands/#join), +[`remove-data`](/{{< latest "enterprise_influxdb" >}}/administration/cluster-commands/#remove-data), +[`remove-meta`](/{{< latest "enterprise_influxdb" >}}/administration/cluster-commands/#remove-meta), and +[`leave`](/{{< latest "enterprise_influxdb" >}}/administration/cluster-commands/#leave) + +**Pages in Chronograf that require this permission**: NA + +#### CopyShard +Permission to copy shards. + +**Relevant `influxd-ctl` arguments**: +[`copy-shard`](/{{< latest "enterprise_influxdb" >}}/administration/cluster-commands/#copy-shard) + +**Pages in Chronograf that require this permission**: NA + +#### CreateDatabase +Permission to create databases, create [retention policies](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#retention-policy-rp), alter retention policies, and view retention policies. + +**Relevant InfluxQL queries**: +[`CREATE DATABASE`](/{{< latest "influxdb" "v1" >}}/query_language/manage-database/#create-database), +[`CREATE RETENTION POLICY`](/{{< latest "influxdb" "v1" >}}/query_language/manage-database/#create-retention-policies-with-create-retention-policy), +[`ALTER RETENTION POLICY`](/{{< latest "influxdb" "v1" >}}/query_language/manage-database/#modify-retention-policies-with-alter-retention-policy), and +[`SHOW RETENTION POLICIES`](/{{< latest "influxdb" "v1" >}}/query_language/explore-schema/#show-retention-policies) + +**Pages in Chronograf that require this permission**: Dashboards, Data Explorer, and Databases on the Admin page + +#### CreateUserAndRole +Permission to manage users and roles; create users, drop users, grant admin status to users, grant permissions to users, revoke admin status from users, revoke permissions from users, change user passwords, view user permissions, and view users and their admin status. + +**Relevant InfluxQL queries**: +[`CREATE USER`](/{{< latest "influxdb" "v1" >}}/administration/authentication_and_authorization/#user-management-commands), +[`DROP USER`](/{{< latest "influxdb" "v1" >}}/administration/authentication_and_authorization/#general-admin-and-non-admin-user-management), +[`GRANT ALL PRIVILEGES`](/{{< latest "influxdb" "v1" >}}/administration/authentication_and_authorization/#user-management-commands), +[`GRANT [READ,WRITE,ALL]`](/{{< latest "influxdb" "v1" >}}/administration/authentication_and_authorization/#non-admin-user-management), +[`REVOKE ALL PRIVILEGES`](/{{< latest "influxdb" "v1" >}}/administration/authentication_and_authorization/#user-management-commands), +[`REVOKE [READ,WRITE,ALL]`](/{{< latest "influxdb" "v1" >}}/administration/authentication_and_authorization/#non-admin-user-management), +[`SET PASSWORD`](/{{< latest "influxdb" "v1" >}}/administration/authentication_and_authorization/#general-admin-and-non-admin-user-management), +[`SHOW GRANTS`](/{{< latest "influxdb" "v1" >}}/administration/authentication_and_authorization/#non-admin-user-management), and +[`SHOW USERS`](/{{< latest "influxdb" "v1" >}}/administration/authentication_and_authorization/#user-management-commands) + +**Pages in Chronograf that require this permission**: Data Explorer, Dashboards, Users and Roles on the Admin page + +#### DropData +Permission to drop data, in particular [series](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#series) and [measurements](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#measurement). + +**Relevant InfluxQL queries**: +[`DROP SERIES`](/{{< latest "influxdb" "v1" >}}/query_language/manage-database/#drop-series-from-the-index-with-drop-series), +[`DELETE`](/{{< latest "influxdb" "v1" >}}/query_language/manage-database/#delete-series-with-delete), and +[`DROP MEASUREMENT`](/{{< latest "influxdb" "v1" >}}/query_language/manage-database/#delete-measurements-with-drop-measurement) + +**Pages in Chronograf that require this permission**: NA + +#### DropDatabase +Permission to drop databases and retention policies. + +**Relevant InfluxQL queries**: +[`DROP DATABASE`](/{{< latest "influxdb" "v1" >}}/query_language/manage-database/#delete-a-database-with-drop-database) and +[`DROP RETENTION POLICY`](/{{< latest "influxdb" "v1" >}}/query_language/manage-database/#delete-retention-policies-with-drop-retention-policy) + +**Pages in Chronograf that require this permission**: Data Explorer, Dashboards, Databases on the Admin page + +#### KapacitorAPI +Permission to access the API for InfluxKapacitor Enterprise. +This does not include configuration-related API calls. + +**Pages in Chronograf that require this permission**: NA + +#### KapacitorConfigAPI +Permission to access the configuration-related API calls for InfluxKapacitor Enterprise. + +**Pages in Chronograf that require this permission**: NA + +#### ManageContinuousQuery +Permission to create, drop, and view [continuous queries](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#continuous-query-cq). + +**Relevant InfluxQL queries**: +[`CreateContinuousQueryStatement`](/{{< latest "influxdb" "v1" >}}/query_language/continuous_queries/), +[`DropContinuousQueryStatement`](/{{< latest "influxdb" "v1" >}}/query_language/continuous_queries/#deleting-continuous-queries), and +[`ShowContinuousQueriesStatement`](/{{< latest "influxdb" "v1" >}}/query_language/continuous_queries/#listing-continuous-queries) + +**Pages in Chronograf that require this permission**: Data Explorer, Dashboards + +#### ManageQuery +Permission to view and kill queries. + +**Relevant InfluxQL queries**: +[`SHOW QUERIES`](/{{< latest "influxdb" "v1" >}}/troubleshooting/query_management/#list-currently-running-queries-with-show-queries) and +[`KILL QUERY`](/{{< latest "influxdb" "v1" >}}/troubleshooting/query_management/#stop-currently-running-queries-with-kill-query) + +**Pages in Chronograf that require this permission**: Queries on the Admin page + +#### ManageShard +Permission to copy, delete, and view [shards](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#shard). + +**Relevant InfluxQL queries**: +[`DropShardStatement`](/{{< latest "influxdb" "v1" >}}/query_language/manage-database/#delete-a-shard-with-drop-shard), +[`ShowShardGroupsStatement`](/{{< latest "influxdb" "v1" >}}/query_language/spec/#show-shard-groups), and +[`ShowShardsStatement`](/{{< latest "influxdb" "v1" >}}/query_language/spec/#show-shards) + +**Pages in Chronograf that require this permission**: NA + +#### ManageSubscription +Permission to create, drop, and view [subscriptions](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#subscription). + +**Relevant InfluxQL queries**: +[`CREATE SUBSCRIPTION`](/{{< latest "influxdb" "v1" >}}/query_language/spec/#create-subscription), +[`DROP SUBSCRIPTION`](/{{< latest "influxdb" "v1" >}}/query_language/spec/#drop-subscription), and +[`SHOW SUBSCRIPTIONS`](/{{< latest "influxdb" "v1" >}}/query_language/spec/#show-subscriptions) + +**Pages in Chronograf that require this permission**: Alerting + +#### Monitor +Permission to view cluster statistics and diagnostics. + +**Relevant InfluxQL queries**: +[`SHOW DIAGNOSTICS`](/{{< latest "influxdb" "v1" >}}/administration/server_monitoring/#show-diagnostics) and +[`SHOW STATS`](/{{< latest "influxdb" "v1" >}}/administration/server_monitoring/#show-stats) + +**Pages in Chronograf that require this permission**: Data Explorer, Dashboards + +#### ReadData +Permission to read data. + +**Relevant InfluxQL queries**: +[`SHOW FIELD KEYS`](/{{< latest "influxdb" "v1" >}}/query_language/explore-schema/#show-field-keys), +[`SHOW MEASUREMENTS`](/{{< latest "influxdb" "v1" >}}/query_language/explore-schema/#show-measurements), +[`SHOW SERIES`](/{{< latest "influxdb" "v1" >}}/query_language/explore-schema/#show-series), +[`SHOW TAG KEYS`](/{{< latest "influxdb" "v1" >}}/query_language/explore-schema/#show-tag-keys), +[`SHOW TAG VALUES`](/{{< latest "influxdb" "v1" >}}/query_language/explore-schema/#show-tag-values), and +[`SHOW RETENTION POLICIES`](/{{< latest "influxdb" "v1" >}}/query_language/explore-schema/#show-retention-policies) + +**Pages in Chronograf that require this permission**: Admin, Alerting, Dashboards, Data Explorer, Host List + +#### WriteData +Permission to write data. + +**Relevant InfluxQL queries**: NA + +**Pages in Chronograf that require this permission**: NA + +### Roles +Roles are groups of permissions. +Assign roles to one or more users. + +For example, the image below contains three roles: `CREATOR`, `DESTROYER`, and `POWERLESS`. +`CREATOR` includes two permissions (`CreateDatbase` and `CreateUserAndRole`) and is assigned to one user (`chrononut`). +`DESTROYER` also includes two permissions (`DropDatabase` and `DropData`) and is assigned to two users (`chrononut` and `chronelda`). + +![InfluxDB OSS user management](/img/chronograf/1-6-admin-usermanagement-roles.png) diff --git a/content/chronograf/v1.9/administration/managing-organizations.md b/content/chronograf/v1.9/administration/managing-organizations.md new file mode 100644 index 000000000..1115fae78 --- /dev/null +++ b/content/chronograf/v1.9/administration/managing-organizations.md @@ -0,0 +1,120 @@ +--- +title: Manage Chronograf organizations +description: Create, configure, map, and remove organizations in Chronograf. +menu: + chronograf_1_9: + name: Manage Chronograf organizations + weight: 80 + parent: Administration +--- + +**On this page:** + +* [About Chronograf organizations](#about-chronograf-organizations) +* [Use the default organization](#use-the-default-organization) +* [Create organizations](#create-organizations) +* [Configure organizations](#configure-organizations) +* [Map organizations](#map-organizations) +* [Remove organizations](#remove-organizations) + + +## About Chronograf organizations + +{{% note %}} +**Note:** Support for organizations and user roles is available in Chronograf 1.4 or later. +First, OAuth 2.0 authentication must be configured (if it is, you'll see the Chronograf Admin tab on the Admin menu). +For more information, see [managing security](/chronograf/v1.9/administration/managing-security/). +{{% /note %}} + +For information about the new user roles and SuperAdmin permission, see [Managing Chronograf users](/chronograf/v1.9/administration/managing-chronograf-users/). + +A Chronograf organization is a collection of Chronograf users who share common Chronograf-owned resources, including dashboards, InfluxDB connections, and Kapacitor connections. Organizations can be used to represent companies, functional units, projects, or teams. Chronograf users can be members of multiple organizations. + +{{% note %}} +**Note:** Only users with SuperAdmin permission can manage organizations. Admins, editors, viewers, and members cannot manage organizations unless they have SuperAdmin permission. +{{% /note %}} + +## Use the default organization + +{{% note %}} +**Note:** The default organization can be used to support Chronograf as configured in versions earlier than 1.4. +Upon upgrading, any Chronograf resources that existed prior to 1.4 automatically become owned by the Default organization. +{{% /note %}} + +Upon installation, the default organization is ready for use and allows Chronograf to be used as-is. + +## Create organizations + +Your company, organizational units, teams, and projects may require the creation of additional organizations, beyond the Default organization. Additional organizations can be created as described below. + +**To create an organization:** + +**Required permission:** SuperAdmin + +1) In the Chronograf navigation bar, click **Admin** (crown icon) > **Chronograf** to open the **Chronograf Admin** page. +2) In the **All Orgs** tab, click **Create Organization**. +3) Under **Name**, click on **"Untitled Organization"** and enter the new organization name. +4) Under **Default Role**, select the default role for new users within that organization. Valid options include `member` (default), `viewer`, `editor`, and `admin`. +5) Click **Save**. + +## Configure organizations + +**Required permission:** SuperAdmin + +You can configure existing and new organizations in the **Organizations** tab of the **Chronograf Admin** page as follows: + +* **Name**: The name of the organization. Click on the organization name to change it. + + > ***Note:*** You can change the Default organization's name, but that organization will always be the default organization. + +* **Public**: [Default organization only] Indicates whether a user can authenticate without being explicitly added to the organization. When **Public** is toggled to **Off**, new users cannot authenticate into your Chronograf instance unless they have been explicitly added to the organization by an administrator. + + > ***Note:*** All organizations other than the Default organization require users to be explicitly added by an administrator. + +* **Default Role**: The role granted to new users by default when added to an organization. Valid options are `member` (default), `viewer`, `editor`, and `admin`. + +See the following pages for more information about managing Chronograf users and security: + +* [Manage Chronograf users](/chronograf/v1.9/administration/managing-chronograf-users/) +* [Manage security](/chronograf/v1.9/administration/managing-security/) + +## Map organizations + +**To create an organization mapping:** + +**Required permission:** SuperAdmin + +1) In the Chronograf navigation bar, select **Admin** (crown icon) > **Chronograf** to open the **Chronograf Admin** page. +2) Click the **Org Mappings** tab to view a list of organization mappings. +3) To add an organization mapping, click the **Create Mapping** button. A new row is added to the listing. +4) In the new row, enter the following: + +- **Scheme**, select `oauth2`. +- **Provider**: Enter the provider. Valid values include `Google` and `GitHub`. +- **Provider Org**: [Optional] Enter the email domain(s) you want to accept. +- **Organization**: Select the organization that can use this authentication provider. + +**To remove an organization mapping:** + +**Required permission:** SuperAdmin + +1) In the Chronograf navigation bar, select **Admin** (crown icon) > **Chronograf** to open the **Chronograf Admin** page. +2) Click the **Org Mappings** tab to view a list of organization mappings. +3) To remove an organization mapping, click the **Delete** button at the end of the mapping row you want to remove, and then confirm the action. + +## Remove organizations + +When an organization is removed: + +* Users within that organization are removed from that organization and will be logged out of the application. +* All users with roles in that organization are updated to no longer have a role in that organization +* All resources owned by that organization are deleted. + + +**To remove an organization:** + +**Required permission:** SuperAdmin + +1) In the navigation bar of the Chronograf application, select **Admin** (crown icon) > **Chronograf** to open the **Chronograf Admin** page. +2) Click the **All Orgs** tab to view a list of organizations. +3) To the right of the the organization that you want to remove, click the **Remove** button (trashcan icon) and then confirm by clicking the **Save** button. diff --git a/content/chronograf/v1.9/administration/managing-security.md b/content/chronograf/v1.9/administration/managing-security.md new file mode 100644 index 000000000..0658b0a2e --- /dev/null +++ b/content/chronograf/v1.9/administration/managing-security.md @@ -0,0 +1,599 @@ +--- +title: Manage Chronograf security +description: Manage Chronograf security with OAuth 2.0 providers. +aliases: /chronograf/v1.9/administration/security-best-practices/ +menu: + chronograf_1_9: + name: Manage Chronograf security + weight: 70 + parent: Administration +--- + +To enhance security, configure Chronograf to authenticate and authorize with [OAuth 2.0](https://oauth.net/) and use TLS/HTTPS. +(Basic authentication with username and password is also available.) + +* [Configure Chronograf to authenticate with OAuth 2.0](#configure-chronograf-to-authenticate-with-oauth-20) + 1. [Generate a Token Secret](#generate-a-token-secret) + 2. [Set configurations for your OAuth provider](#set-configurations-for-your-oauth-provider) + 3. [Configure authentication duration](#configure-authentication-duration) +* [Configure Chronograf to authenticate with a username and password](#configure-chronograf-to-authenticate-with-a-username-and-password) +* [Configure TLS (Transport Layer Security) and HTTPS](#configure-tls-transport-layer-security-and-https) + +## Configure Chronograf to authenticate with OAuth 2.0 + +{{% note %}} +After configuring OAuth 2.0, the Chronograf Admin tab becomes visible. +You can then set up [multiple organizations](/chronograf/v1.9/administration/managing-organizations/) +and [users](/chronograf/v1.9/administration/managing-influxdb-users/). +{{% /note %}} + +Configure Chronograf to use an OAuth 2.0 provider and JWT (JSON Web Token) to authenticate users and enable role-based access controls. + +(For more details on OAuth and JWT, see [RFC 6749](https://tools.ietf.org/html/rfc6749) and [RFC 7519](https://tools.ietf.org/html/rfc7519).) + +{{% note %}} +#### OAuth PKCE +OAuth configurations in **Chronograf 1.9+** use [OAuth PKCE](https://oauth.net/2/pkce/) to +mitigate the threat of having the authorization code intercepted during the OAuth token exchange. +OAuth integrations that do no currently support PKCE are not affected. + +**To disable OAuth PKCE** and revert to the previous token exchange, use the +[`--oauth-no-pkce` Chronograf configuration option](/chronograf/v1.9/administration/config-options/#--oauth-no-pkce) +or set the `OAUTH_NO_PCKE` environment variable to `true`. +{{% /note %}} + +### Generate a Token Secret + +To configure any of the supported OAuth 2.0 providers to work with Chronograf, +you must configure the `TOKEN_SECRET` environment variable (or command line option). +Chronograf will use this secret to generate the JWT Signature for all access tokens. + +1. Generate a high-entropy pseudo-random string. + + For example, to do this with OpenSSL, run this command: + ```sh + openssl rand -base64 256 | tr -d '\n' + ``` + +2. Set the environment variable: + + ``` + TOKEN_SECRET= + ``` + +{{% note %}} +***InfluxDB Enterprise clusters:*** If you are running multiple Chronograf servers in a high availability configuration, +set the `TOKEN_SECRET` environment variable on each server to ensure that users can stay logged in. +{{% /note %}} + +### JWKS Signature Verification (optional) + +If the OAuth provider implements OpenID Connect with RS256 signatures, you need to enable this feature with the `USE_ID_TOKEN` variable +and provide a JSON Web Key Set (JWKS) document (holding the certificate chain) to validate the RSA signatures against. +This certificate chain is regularly rolled over (when the certificates expire), so it is fetched from the `JWKS_URL` on demand. + +**Example:** + +```sh +export USE_ID_TOKEN=true +export JWKS_URL=https://example.com/adfs/discovery/keys +``` + +### Set configurations for your OAuth provider + +To enable OAuth 2.0 authorization and authentication in Chronograf, +you must set configuration options that are specific for the OAuth 2.0 authentication provider you want to use. + +Configuration steps for the following supported authentication providers are provided in these sections below: + +* [GitHub](#configure-github-authentication) +* [Google](#configure-google-authentication) +* [Auth0](#configure-auth0-authentication) +* [Heroku](#configure-heroku-authentication) +* [Okta](#configure-okta-authentication) +* [Gitlab](#configure-gitlab-authentication) +* [Azure Active Directory](#configure-azure-active-directory-authentication) +* [Bitbucket](#configure-bitbucket-authentication) +* [Configure Chronograf to use any OAuth 2.0 provider](#configure-chronograf-to-use-any-oauth-20-provider) + +{{% note %}} +If you haven't already, you must first [generate a token secret](#generate-a-token-secret) before proceeding. +{{% /note %}} + +--- + +#### Configure GitHub authentication + +1. Follow the steps to [Register a new OAuth application](https://github.com/settings/applications/new) + on GitHub to obtain your Client ID and Client Secret. + On the GitHub application registration page, enter the following values: + - **Homepage URL**: the full Chronograf server name and port. + For example, to run the application locally with default settings, set the this URL to `http://localhost:8888`. + - **Authorization callback URL**: the **Homepage URL** plus the callback URL path `/oauth/github/callback` + (for example, `http://localhost:8888/oauth/github/callback`). + +2. Set the Chronograf environment variables with the credentials provided by GitHub: + + ```sh + export GH_CLIENT_ID= + export GH_CLIENT_SECRET= + + # If using Github Enterprise + export GH_URL=https://github.custom-domain.com + ``` + +3. If you haven't already, set the Chronograf environment with your token secret: + + ```sh + export TOKEN_SECRET=Super5uperUdn3verGu355! + ``` + +Alternatively, set environment variables using the equivalent command line options: + +- [`--github-url`](/chronograf/v1.9/administration/config-options/#--github-url) +- [`--github-client-id`](/chronograf/v1.9/administration/config-options/#--github-client-id-i) +- [`--github-client-secret`](/chronograf/v1.9/administration/config-options/#--github-client-secret-s) +- [`--token_secret=`](/chronograf/v1.9/administration/config-options/#--token-secret-t) + +For details on the command line options and environment variables, see [GitHub OAuth 2.0 authentication options](/chronograf/v1.9/administration/config-options#github-specific-oauth-20-authentication-options). + +##### GitHub organizations (optional) + +To require GitHub organization membership for authenticating users, set the `GH_ORGS` environment variable with the name of your organization. + +```sh +export GH_ORGS=biffs-gang +``` + +If the user is not a member of the specified GitHub organization, then the user will not be granted access. +To support multiple organizations, use a comma-delimited list. + +```sh +export GH_ORGS=hill-valley-preservation-sociey,the-pinheads +``` + +{{% note %}} +When logging in for the first time, make sure to grant access to the organization you configured. +The OAuth application can only see membership in organizations it has been granted access to. +{{% /note %}} + +##### Example GitHub OAuth configuration + +```bash +# Github Enterprise base URL +export GH_URL=https://github.mydomain.com + +# GitHub Client ID +export GH_CLIENT_ID=b339dd4fddd95abec9aa + +# GitHub Client Secret +export GH_CLIENT_SECRET=260041897d3252c146ece6b46ba39bc1e54416dc + +# Secret used to generate JWT tokens +export TOKEN_SECRET=Super5uperUdn3verGu355! + +# Restrict to specific GitHub organizations +export GH_ORGS=biffs-gang +``` + +#### Configure Google authentication + +1. Follow the steps in [Obtain OAuth 2.0 credentials](https://developers.google.com/identity/protocols/OpenIDConnect#getcredentials) + to obtain the required Google OAuth 2.0 credentials, including a Google Client ID and Client Secret, by +2. Verify that Chronograf is publicly accessible using a fully-qualified domain name so that Google can properly redirect users back to the application. +3. Set the Chronograf environment variables for the Google OAuth 2.0 credentials and **Public URL** used to access Chronograf: + + ```sh + export GOOGLE_CLIENT_ID=812760930421-kj6rnscmlbv49pmkgr1jq5autblc49kr.apps.googleusercontent.com + export GOOGLE_CLIENT_SECRET=wwo0m29iLirM6LzHJWE84GRD + export PUBLIC_URL=http://localhost:8888 + ``` + +4. If you haven't already, set the Chronograf environment with your token secret: + + ```sh + export TOKEN_SECRET=Super5uperUdn3verGu355! + ``` + +Alternatively, the environment variables discussed above can be set using their corresponding command line options: + +* [`--google-client-id=`](/chronograf/v1.9/administration/config-options/#google-client-id) +* [`--google-client-secret=`](/chronograf/v1.9/administration/config-options/#google-client-secret) +* [`--public-url=`](/chronograf/v1.9/administration/config-options/#public-url) +* [`--token_secret=`](/chronograf/v1.9/administration/config-options/#token-secret-t) + +For details on Chronograf command line options and environment variables, see [Google OAuth 2.0 authentication options](/chronograf/v1.9/administration/config-options#google-specific-oauth-20-authentication-options). + +##### Optional Google domains + +Configure Google authentication to restrict access to Chronograf to specific domains. +Set the `GOOGLE_DOMAINS` environment variable or the [`--google-domains`](/chronograf/v1.9/administration/config-options/#google-domains) command line option. +Separate multiple domains using commas. +For example, to permit access only from `biffspleasurepalace.com` and `savetheclocktower.com`, set the environment variable as follows: + +```sh +export GOOGLE_DOMAINS=biffspleasurepalace.com,savetheclocktower.com +``` + +#### Configure Auth0 authentication + +See [OAuth 2.0](https://auth0.com/docs/protocols/oauth2) for details about the Auth0 implementation. + +1. Set up your Auth0 account to obtain the necessary credentials. + 1. From the Auth0 user dashboard, click **Create Application**. + 2. Choose **Regular Web Applications** as the type of application and click **Create**. + 3. In the **Settings** tab, set **Token Endpoint Authentication** to **None**. + 4. Set **Allowed Callback URLs** to `https://www.example.com/oauth/auth0/callback` (substituting `example.com` with the [`PUBLIC_URL`](/chronograf/v1.9/administration/config-options/#general-authentication-options) of your Chronograf instance) + 5. Set **Allowed Logout URLs** to `https://www.example.com` (substituting `example.com` with the [`PUBLIC_URL`](/chronograf/v1.9/administration/config-options/#general-authentication-options) of your Chronograf instance) + + +2. Set the Chronograf environment variables based on your Auth0 client credentials: + + * `AUTH0_DOMAIN` (Auth0 domain) + * `AUTH0_CLIENT_ID` (Auth0 Client ID) + * `AUTH0_CLIENT_SECRET` (Auth0 client Secret) + * `PUBLIC_URL` (Public URL, used in callback URL and logout URL above) + +3. If you haven't already, set the Chronograf environment with your token secret: + + ```sh + export TOKEN_SECRET=Super5uperUdn3verGu355! + ``` + +Alternatively, the environment variables discussed above can be set using their corresponding command line options: + +* [`--auth0-domain`](/chronograf/v1.9/administration/config-options/#auth0-specific-oauth-20-authentication-options) +* [`--auth0-client-id`](/chronograf/v1.9/administration/config-options/#auth0-specific-oauth-20-authentication-options) +* [`--auth0-client-secret`](/chronograf/v1.9/administration/config-options/#auth0-specific-oauth-20-authentication-options) +* [`--public-url`](/chronograf/v1.9/administration/config-options/#general-authentication-options) + +##### Auth0 organizations (optional) + +Auth0 can be customized to the operator's requirements, so it has no official concept of an "organization." +Organizations are supported in Chronograf using a lightweight `app_metadata` key that can be inserted into Auth0 user profiles automatically or manually. + +To assign a user to an organization, add an `organization` key to the user `app_metadata` field with the value corresponding to the user's organization. +For example, you can assign the user Marty McFly to the "time-travelers" organization by setting `app_metadata` to `{"organization": "time-travelers"}`. +This can be done either manually by an operator or automatically through the use of an [Auth0 Rule](https://auth0.com/docs/rules) or a [pre-user registration Auth0 Hook](https://auth0.com/docs/hooks/concepts/pre-user-registration-extensibility-point). + +Next, you will need to set the Chronograf [`AUTH0_ORGS`](/chronograf/v1.9/administration/config-options/#auth0-organizations) environment variable to a comma-separated list of the allowed organizations. +For example, if you have one group of users with an `organization` key set to `biffs-gang` and another group with an `organization` key set to `time-travelers`, you can permit access to both with this environment variable: `AUTH0_ORGS=biffs-gang,time-travelers`. + +An `--auth0-organizations` command line option is also available, but it is limited to a single organization and does not accept a comma-separated list like its environment variable equivalent. + +#### Configure Heroku authentication + +1. Obtain a client ID and application secret for Heroku by following the guide posted [here](https://devcenter.heroku.com/articles/oauth#register-client). +2. Set the Chronograf environment variables based on your Heroku client credentials: + + ```sh + export HEROKU_CLIENT_ID= + export HEROKU_SECRET= + ``` + +3. If you haven't already, set the Chronograf environment with your token secret: + + ```sh + export TOKEN_SECRET=Super5uperUdn3verGu355! + ``` + +##### Heroku organizations (optional) + +To restrict access to members of specific Heroku organizations, +use the `HEROKU_ORGS` environment variable (or associated command line option). +Multiple values must be comma-separated. + +For example, to permit access from the `hill-valley-preservation-society` organization and `the-pinheads` organization, +use the following environment variable: + +```sh +export HEROKU_ORGS=hill-valley-preservation-sociey,the-pinheads +``` + +#### Configure Okta authentication + +1. Create an Okta web application by following the steps in the Okta documentation: [Implement the Authorization Code Flow](https://developer.okta.com/docs/guides/implement-auth-code/overview/). + 1. In the **General Settings** section, find the **Allowed grant types** listing and select + only the **Client acting on behalf of a user:** **Authorization Code** option. + 2. In the **LOGIN** section, set the **Login redirect URIs* and **Initiate login URI** to `http://localhost:8888/oauth/okta/callback` (the default callback URL for Chronograf). + +2. Set the following Chronograf environment variables: + + ```bash + GENERIC_NAME=okta + + # The client ID is provided in the "Client Credentials" section of the Okta dashboard. + GENERIC_CLIENT_ID= + + # The client secret is in the "Client Credentials" section of the Okta dashboard. + GENERIC_CLIENT_SECRET= + + GENERIC_AUTH_URL=https://dev-553212.oktapreview.com/oauth2/default/v1/authorize + GENERIC_TOKEN_URL=https://dev-553212.oktapreview.com/oauth2/default/v1/token + GENERIC_API_URL=https://dev-553212.oktapreview.com/oauth2/default/v1/userinfo + PUBLIC_URL=http://localhost:8888 + TOKEN_SECRET=secretsecretsecret + GENERIC_SCOPES=openid,profile,email + ``` + +3. If you haven't already, set the Chronograf environment with your token secret: + + ```sh + export TOKEN_SECRET=Super5uperUdn3verGu355! + ``` + +#### Configure GitLab authentication + +1. In your GitLab profile, [create a new OAuth2 authentication service](https://docs.gitlab.com/ee/integration/oauth_provider.html#adding-an-application-through-the-profile). + 1. Provide a name for your application, then enter your publicly accessible Chronograf URL with the `/oauth/gitlab/callback` path as your GitLab **callback URL**. + (For example, `http://:8888/oauth/gitlab/callback`.) + 2. Click **Submit** to save the service details. + 3. Make sure your application has **openid** and **read_user** scopes. + +2. Copy the provided **Application Id** and **Secret** and set the following environment variables: + + > In the examples below, note the use of `gitlab-server-example.com` and `chronograf-server-example.com` in urls. + > These should be replaced by the actual URLs used to access each service. + + ```bash + GENERIC_NAME="gitlab" + GENERIC_CLIENT_ID= + GENERIC_CLIENT_SECRET= + GENERIC_AUTH_URL="https://gitlab.com/oauth/authorize" + GENERIC_TOKEN_URL="https://gitlab.com/oauth/token" + TOKEN_SECRET= + GENERIC_SCOPES="api,openid,read_user" + PUBLIC_URL="http://:8888" + GENERIC_API_URL="https://gitlab.com/api/v3/user" + ``` + + The equivalent command line options are: + + ```bash + --generic-name=gitlab + --generic-client-id= + --generic-client-secret= + --generic-auth-url=https://gitlab.com/oauth/authorize + --generic-token-url=https://gitlab.com/oauth/token + --token-secret= + --generic-scopes=openid,read_user + --generic-api-url=https://gitlab.com/api/v3/user + --public-url=http://:8888/ + ``` + +#### Configure Azure Active Directory authentication + +1. [Create an Azure Active Directory application](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#create-an-azure-active-directory-application). + Note the following information: ``, ``, and ``. + You'll need these to define your Chronograf environment. + +2. Be sure to register a reply URL in your Azure application settings. + This should match the calling URL from Chronograf. + Otherwise, you will get an error stating no reply address is registered for the application. + For example, if Chronograf is configured with a `GENERIC_NAME` value of AzureAD, the reply URL would be `http://localhost:8888/oauth/AzureAD/callback`. + +3. After completing the application provisioning within Azure AD, you can now complete the configuration with Chronograf. + Using the metadata from your Azure AD instance, proceed to export the following environment variables: + + Set the following environment variables in `/etc/default/chronograf`: + + ``` + GENERIC_TOKEN_URL=https://login.microsoftonline.com/<>/oauth2/token + TENANT=<> + GENERIC_NAME=AzureAD + GENERIC_API_KEY=userPrincipalName + GENERIC_SCOPES=openid + GENERIC_CLIENT_ID=<> + GENERIC_AUTH_URL=https://login.microsoftonline.com/<>/oauth2/authorize?resource=https://graph.windows.net + GENERIC_CLIENT_SECRET=<> + TOKEN_SECRET=secret + GENERIC_API_URL=https://graph.windows.net/<>/me?api-version=1.6 + PUBLIC_URL=http://localhost:8888 + ``` + + Note: If you’ve configured TLS/SSL, modify the `PUBLIC_URL` to ensure you're using HTTPS. + +#### Configure Bitbucket authentication + +1. Complete the instructions to [Use OAuth on Bitbucket Cloud](https://support.atlassian.com/bitbucket-cloud/docs/use-oauth-on-bitbucket-cloud/), and include the following information: + + - **Callback URL**: + - **Permissions**: Account read, email + +2. Run the following command to set Chronograf environment variables for Bitbucket in `/etc/default/chronograf`: + + ```sh + export TOKEN_SECRET=... + export GENERIC_CLIENT_ID=... + export GENERIC_CLIENT_SECRET=... + export GENERIC_AUTH_URL=https://bitbucket.org/site/oauth2/authorize + export GENERIC_TOKEN_URL=https://bitbucket.org/site/oauth2/access_token + export GENERIC_API_URL=https://api.bitbucket.org/2.0/user + export GENERIC_SCOPES=account + export PUBLIC_URL=http://localhost:8888 + export GENERIC_NAME=bitbucket + ``` + +#### Configure Chronograf to use any OAuth 2.0 provider + +Chronograf can be configured to work with any OAuth 2.0 provider, including those defined above, by using the generic configuration options below. +Additionally, the generic provider implements OpenID Connect (OIDC) as implemented by Active Directory Federation Services (AD FS). + +When using the generic configuration, some or all of the following environment variables (or corresponding command line options) are required (depending on your OAuth 2.0 provider): + +* `GENERIC_CLIENT_ID`: Application client [identifier](https://tools.ietf.org/html/rfc6749#section-2.2) issued by the provider +* `GENERIC_CLIENT_SECRET`: Application client [secret](https://tools.ietf.org/html/rfc6749#section-2.3.1) issued by the provider +* `GENERIC_AUTH_URL`: Provider's authorization [endpoint](https://tools.ietf.org/html/rfc6749#section-3.1) URL +* `GENERIC_TOKEN_URL`: Provider's token [endpoint](https://tools.ietf.org/html/rfc6749#section-3.2) URL used by the Chronograf client to obtain an access token +* `USE_ID_TOKEN`: Enable OpenID [id_token](https://openid.net/specs/openid-connect-core-1_0.html#rfc.section.3.1.3.3) processing +* `JWKS_URL`: Provider's JWKS [endpoint](https://tools.ietf.org/html/rfc7517#section-4.7) used by the client to validate RSA signatures +* `GENERIC_API_URL`: Provider's [OpenID UserInfo endpoint](https://connect2id.com/products/server/docs/api/userinfo) URL used by Chronograf to request user data +* `GENERIC_API_KEY`: JSON lookup key for [OpenID UserInfo](https://connect2id.com/products/server/docs/api/userinfo) (known to be required for Microsoft Azure, with the value `userPrincipalName`) +* `GENERIC_SCOPES`: [Scopes](https://tools.ietf.org/html/rfc6749#section-3.3) of user data required for your instance of Chronograf, such as user email and OAuth provider organization + - Multiple values must be space-delimited, e.g. `user:email read:org` + - These may vary by OAuth 2.0 provider + - Default value: `user:email` +* `PUBLIC_URL`: Full public URL used to access Chronograf from a web browser, i.e. where Chronograf is hosted + - Used by Chronograf, for example, to construct the callback URL +* `TOKEN_SECRET`: Used to validate OAuth [state](https://tools.ietf.org/html/rfc6749#section-4.1.1) response. (see above) + +##### Optional environment variables + +The following environment variables (and corresponding command line options) are also available for optional use: + +* `GENERIC_DOMAINS`: Email domain where email address must include. +* `GENERIC_NAME`: Value used in the callback URL in conjunction with `PUBLIC_URL`, e.g. `/oauth//callback` + - This value is also used in the text for the Chronograf Login button + - Default value is `generic` + - So, for example, if `PUBLIC_URL` is `https://localhost:8888` and `GENERIC_NAME` is its default value, then the callback URL would be `https://localhost:8888/oauth/generic/callback`, and the Chronograf Login button would read `Log in with Generic` + - While using Chronograf, this value should be supplied in the `Provider` field when adding a user or creating an organization mapping. + +##### Example: OIDC with AD FS + +See [Enabling OpenID Connect with AD FS 2016](https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/development/enabling-openid-connect-with-ad-fs) for a walk through of the server configuration. + +Exports for Chronograf (e.g. in `/etc/default/chronograf`): + +```sh +PUBLIC_URL="https://example.com:8888" +GENERIC_CLIENT_ID="chronograf" +GENERIC_CLIENT_SECRET="KW-TkvH7vzYeJMAKj-3T1PdHx5bxrZnoNck2KlX8" +GENERIC_AUTH_URL="https://example.com/adfs/oauth2/authorize" +GENERIC_TOKEN_URL="https://example.com/adfs/oauth2/token" +GENERIC_SCOPES="openid" +GENERIC_API_KEY="upn" +USE_ID_TOKEN="true" +JWKS_URL="https://example.com/adfs/discovery/keys" +TOKEN_SECRET="ZNh2N9toMwUVQxTVEe2ZnnMtgkh3xqKZ" +``` + +{{% note %}} +Do not use special characters for the `GENERIC_CLIENT_ID` as AD FS may split strings at the special character, resulting in an identifier mismatch. +{{% /note %}} + +{{% note %}} +#### Troubleshoot OAuth errors + +##### ERRO[0053] +A **ERRO[0053]** error indicates that a primary email is not found for the specified user. +A user must have a primary email. + +``` +ERRO[0053] Unable to get OAuth Group malformed email address, expected "..." to contain @ symbol +``` +{{% /note %}} + +### Configure authentication duration + +By default, user authentication remains valid for 30 days using a cookie stored in the web browser. +To configure a different authorization duration, set a duration using the `AUTH_DURATION` environment variable. + +**Example:** + +To set the authentication duration to 1 hour, use the following shell command: +```sh +export AUTH_DURATION=1h +``` + +The duration uses the Go (golang) [time duration format](https://golang.org/pkg/time/#ParseDuration), so the largest time unit is `h` (hours). +So to change it to 45 days, use: + +```sh +export AUTH_DURATION=1080h +``` + +To require re-authentication every time the browser is closed, set `AUTH_DURATION` to `0`. +This makes the cookie transient (aka "in-memory"). + +## Configure Chronograf to authenticate with a username and password + +Chronograf can be configured to authenticate users by username and password ("basic authentication"). +Turn on basic authentication access to restrict HTTP requests to Chronograf to selected users. + +{{% warn %}} +[OAuth 2.0](#configure-chronograf-to-authenticate-with-oauth-20) is the preferred method for authentication. +Only use basic authentication in cases where an OAuth 2.0 integration is not possible. +{{% /warn %}} + +When using basic authentication, *all users have SuperAdmin status*; Chronograf authorization rules are not enforced. +For more information, see [Cross-organization SuperAdmin status](/chronograf/v1.9/administration/managing-chronograf-users/#cross-organization-superadmin-status). + +To enable basic authentication, run chronograf with the `--htpasswd` flag or use the `HTPASSWD` environment variable. + +```sh +chronograf --htpasswd +``` + +The `.htpasswd` file contains users and their passwords, and should be created with a password file utility tool such as `apache2-utils`. +For more information about how to restrict access with basic authentication, see NGINX documentation on [Restricting Access with HTTP Basic Authentication](https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-http-basic-authentication/). + +## Configure TLS (Transport Layer Security) and HTTPS + +The TLS (Transport Layer Security) cryptographic protocol is supported in Chronograf to provides server authentication, data confidentiality, and data integrity. +Using TLS secures traffic between a server and web browser and enables the use of HTTPS. + +InfluxData recommends using HTTPS to communicate securely with Chronograf applications. +If you are not using a TLS termination proxy, you can run your Chronograf server with TLS connections. + +Chronograf includes command line and environment variable options for configuring TLS (Transport Layer Security) certificates and key files. +Use of the TLS cryptographic protocol provides server authentication, data confidentiality, and data integrity. +When configured, users can use HTTPS to securely communicate with your Chronograf applications. + +{{% note %}} +Using HTTPS helps guard against nefarious agents sniffing the JWT and using it to spoof a valid user against the Chronograf server. +{{% /note %}} + +### Configuring TLS for Chronograf + +Chronograf server has command line and environment variable options to specify the certificate and key files. +The server reads and parses a public/private key pair from these files. +The files must contain PEM-encoded data. + +All Chronograf command line options have corresponding environment variables. + +**To configure Chronograf to support TLS:** + +1. Specify the certificate file using the `TLS_CERTIFICATE` environment variable (or the `--cert` CLI option). +2. Specify the key file using the `TLS_PRIVATE_KEY` environment variable (or `--key` CLI option). + +{{% note %}} +If both the TLS certificate and key are in the same file, specify them using the `TLS_CERTIFICATE` environment variable (or the `--cert` CLI option). +{{% /note %}} + +#### Example with CLI options +```sh +chronograf --cert=my.crt --key=my.key +``` + +#### Example with environment variables +```sh +TLS_CERTIFICATE=my.crt TLS_PRIVATE_KEY=my.key chronograf +``` + +#### Docker example with environment variables +```sh +docker run -v /host/path/to/certs:/certs -e TLS_CERTIFICATE=/certs/my.crt -e TLS_PRIVATE_KEY=/certs/my.key quay.io/influxdb/chronograf:latest +``` + +### Testing with self-signed certificates +In a production environment you should not use self-signed certificates, but for testing it is fast to create your own certificates. + +To create a certificate and key in one file with OpenSSL: + +```sh +openssl req -x509 -newkey rsa:4096 -sha256 -nodes -keyout testing.pem -out testing.pem -subj "/CN=localhost" -days 365 +``` + +Next, set the environment variable `TLS_CERTIFICATE`: +```sh +export TLS_CERTIFICATE=$PWD/testing.pem +``` + +Run Chronograf: + +```sh +./chronograf +INFO[0000] Serving chronograf at https://[::]:8888 component=server +``` + +In the first log message you should see `https` rather than `http`. diff --git a/content/chronograf/v1.9/administration/migrate-to-high-availability.md b/content/chronograf/v1.9/administration/migrate-to-high-availability.md new file mode 100644 index 000000000..dba2843b9 --- /dev/null +++ b/content/chronograf/v1.9/administration/migrate-to-high-availability.md @@ -0,0 +1,58 @@ +--- +title: Migrate to a Chronograf HA configuration +description: > + Migrate a Chronograf single instance configuration using BoltDB to a Chronograf high-availability (HA) cluster configuration using etcd. +menu: + chronograf_1_9: + weight: 10 + parent: Administration +--- + +Use [`chronoctl`](/chronograf/v1.9/tools/chronoctl/) to migrate your Chronograf configuration store from BoltDB to a shared `etcd` data store used for Chronograf high-availability (HA) clusters. + +{{% note %}} +#### Update resource IDs +Migrating Chronograf to a shared data source creates new source IDs for each resource. +Update external links to Chronograf dashboards to reflect new source IDs. +{{% /note %}} + +1. Stop the Chronograf server by killing the `chronograf` process. +2. To prevent data loss, we **strongly recommend** that you back up your Chronograf data store before migrating to a Chronograf cluster. +3. [Install and start etcd](/chronograf/v1.9/administration/create-high-availability/#install-and-start-etcd). +4. Run the following command, specifying the local BoltDB file and the `etcd` endpoint beginning with `etcd://`. + (We recommend adding the prefix `bolt://` to an absolute path. + Do not use the prefix to specify a relative path to the BoltDB file.) + + ```sh + chronoctl migrate \ + --from bolt:///path/to/chronograf-v1.db \ + --to etcd://localhost:2379 + ``` + + ##### Provide etcd authentication credentials + If authentication is enabled on `etcd`, use the standard URI basic + authentication format to define a username and password. For example: + + ```sh + etcd://username:password@localhost:2379 + ``` + + ##### Provide etcd TLS credentials + If TLS is enabled on `etcd`, provide your TLS certificate credentials using + the following query parameters in your etcd URL: + + - **cert**: Path to client certificate file or PEM file + - **key**: Path to client key file + - **ca**: Path to trusted CA certificates + + ```sh + etcd://127.0.0.1:2379?cert=/tmp/client.crt&key=/tst/client.key&ca=/tst/ca.crt + ``` + +5. Update links to Chronograf (for example, from external sources) to reflect your new URLs: + - **from BoltDB:** + http://localhost:8888/sources/1/status + - **to etcd:** + http://localhost:8888/sources/373921399246786560/status +6. Set up a load balancer for Chronograf. +7. [Start Chronograf](/chronograf/v1.9/administration/create-high-availability/#start-chronograf). diff --git a/content/chronograf/v1.9/administration/prebuilt-dashboards.md b/content/chronograf/v1.9/administration/prebuilt-dashboards.md new file mode 100644 index 000000000..2d7ed3000 --- /dev/null +++ b/content/chronograf/v1.9/administration/prebuilt-dashboards.md @@ -0,0 +1,486 @@ +--- +title: Prebuilt dashboards in Chronograf +description: Import prebuilt dashboards into Chronograf based on Telegraf plugins. +menu: + chronograf_1_9: + name: Prebuilt dashboards in Chronograf + weight: 50 + parent: Administration +--- +Chronograf lets you import a variety of prebuilt dashboards that visualize metrics collect by specific [Telegraf input plugins](/{{< latest "telegraf" >}}/plugins). The following Telegraf-related dashboards templates are available. + +For details on how to import dashboards while adding a connection in Chronograf, see [Creating connections](/chronograf/v1.9/administration/creating-connections/#manage-influxdb-connections-using-the-chronograf-ui). + +## Docker + +The Docker dashboard displays the following information: + +- nCPU +- Total Memory +- Containers +- System Memory Usage +- System Load +- Disk I/O +- Filesystem Usage +- Block I/O per Container +- CPU Usage per Container +- Memory Usage % per Container +- Memory Usage per Container +- Net I/O per Container + +### Plugins + +- [`docker` plugin](/{{< latest "telegraf" >}}/plugins/#docker) +- [`disk` plugin](/{{< latest "telegraf" >}}/plugins/#disk) +- [`mem` plugin](/{{< latest "telegraf" >}}/plugins/#mem) +- [`diskio` plugin](/{{< latest "telegraf" >}}/plugins/#diskio) +- [`system` plugin](/{{< latest "telegraf" >}}/plugins/#system) +- [`cpu` plugin](/{{< latest "telegraf" >}}/plugins/#cpu) + +## Kubernetes Node +The Kubernetes Node dashboard displays the following information: + +- Total Nodes +- Total Pod Count +- Total Containers +- K8s - Node Millicores +- K8s - Node Memory Bytes +- K8s - Pod Millicores +- K8s - Pod Memory Bytes +- K8s - Pod TX Bytes/Second +- K8s - Pod RX Bytes/Second +- K8s - Kubelet Millicores +- K8s - Kubelet Memory Bytes + +### Plugins +- [kubernetes](/{{< latest "telegraf" >}}/plugins/#kubernetes) + +## Kubernetes Overview +The Kubernetes Node dashboard displays the following information: + +- Total Nodes +- Total Pod Count +- Total Containers +- K8s - Node Millicores +- K8s - Node Memory Bytes +- K8s - Pod Millicores +- K8s - Pod Memory Bytes +- K8s - Pod TX Bytes/Second +- K8s - Pod RX Bytes/Second +- K8s - Kubelet Millicores +- K8s - Kubelet Memory Bytes + +### Plugins + +- [kubernetes](/{{< latest "telegraf" >}}/plugins/#kubernetes) + +## Kubernetes Pod +The Kubernetes Pod dashboard displays the following information: + +- Total Nodes +- Total Pod Count +- Total Containers +- K8s - Pod Millicores +- K8s - Pod Memory Bytes +- K8s - Pod Millicores +- K8s - Pod Memory Bytes +- K8s - Pod TX Bytes/Second + +### Plugins +- [kubernetes](/{{< latest "telegraf" >}}/plugins/#kubernetes) + +## Riak +The Riak dashboard displays the following information: + +- Riak - Total Memory Bytes +- Riak - Object Byte Size +- Riak - Number of Siblings/Minute +- Riak - Latency (ms) +- Riak - Reads and Writes/Minute +- Riak - Active Connections +- Riak - Read Repairs/Minute + +### Plugins +- [`riak` plugin](/{{< latest "telegraf" >}}/plugins/#riak) + +## Consul +The Consul dashboard displays the following information: + +- Consul - Number of Critical Health Checks +- Consul - Number of Warning Health Checks + +### Plugins +- [`consul` plugin](/{{< latest "telegraf" >}}/plugins/#consul) + +## Consul Telemetry +The Consul Telemetry dashboard displays the following information: + +- Consul Agent - Number of Go Routines +- Consul Agent - Runtime Alloc Bytes +- Consul Agent - Heap Objects +- Consul - Number of Agents +- Consul - Leadership Election +- Consul - HTTP Request Time (ms) +- Consul - Leadership Change +- Consul - Number of Serf Events + +### Plugins +[`consul` plugin](/{{< latest "telegraf" >}}/plugins/#consul) + + +## Mesos +The Mesos dashboard displays the following information: + +- Mesos Active Slaves +- Mesos Tasks Active +- Mesos Tasks +- Mesos Outstanding Offers +- Mesos Available/Used CPUs +- Mesos Available/Used Memory +- Mesos Master Uptime + +### Plugins +- [`mesos` plugin](/{{< latest "telegraf" >}}/plugins/#mesos) + +## RabbitMQ +The RabbitMQ dashboard displays the following information: + +- RabbitMQ - Overview +- RabbitMQ - Published/Delivered per Second +- RabbitMQ - Acked/Unacked per Second + +### Plugins + +- [`rabbitmq` plugin](/{{< latest "telegraf" >}}/plugins/#rabbitmq) + +## System + +The System dashboard displays the following information: + +- System Uptime +- CPUs +- RAM +- Memory Used % +- Load +- I/O +- Network +- Processes +- Swap + + +### Plugins + +- [`system` plugin](/{{< latest "telegraf" >}}/plugins/#system) +- [`mem` plugin](/{{< latest "telegraf" >}}/plugins/#mem) +- [`cpu` plugin](/{{< latest "telegraf" >}}/plugins/#cpu) +- [`disk` plugin](/{{< latest "telegraf" >}}/plugins/#disk) +- [`diskio` plugin](/{{< latest "telegraf" >}}/plugins/#diskio) +- [`net` plugin](/{{< latest "telegraf" >}}/plugins/#net) +- [`processes` plugin](/{{< latest "telegraf" >}}/plugins/#processes) +- [`swap` plugin](/{{< latest "telegraf" >}}/plugins/#swap) + + + +## VMware vSphere Overview +The VMware vSphere Overview dashboard gives an overview of your VMware vSphere Clusters and uses metrics from the `vsphere_cluster_*` and `vsphere_vm_*` set of measurements. It displays the following information: + +- Cluster Status + - Uptime for :clustername: + - CPU Usage for :clustername: + - RAM Usage for :clustername: + - Datastores - Usage Capacity + - Network Usage for :clustername: + - Disk Throughput for :clustername: +- VM Status + - VM CPU Usage MHz for :clustername: + - VM Mem Usage for :clustername: + - VM Network Usage for :clustername: + - VM CPU % Ready for :clustername: + +### Plugins +- [`vsphere` plugin](/{{< latest "telegraf" >}}/plugins/#vmware-vsphere) + +## Apache +The Apache dashboard displays the following information: + +- System Uptime +- CPUs +- RAM +- Memory Used % +- Load +- I/O +- Network +- Workers +- Scoreboard +- Apache Uptime +- CPU Load +- Requests per Sec +- Throughput +- Response Codes +- Apache Log + +### Plugins + +- [`apache` plugin](/{{< latest "telegraf" >}}/plugins/#apache) +- [`system` plugin](/{{< latest "telegraf" >}}/plugins/#system) +- [`mem` plugin](/{{< latest "telegraf" >}}/plugins/#mem) +- [`diskio` plugin](/{{< latest "telegraf" >}}/plugins/#diskio) +- [`net` plugin](/{{< latest "telegraf" >}}/plugins/#net) +- [`logparser` plugin](/{{< latest "telegraf" >}}/plugins/#logparser) + +## ElasticSearch +The ElasticSearch dashboard displays the following information: + +- ElasticSearch - Query Throughput +- ElasticSearch - Open Connections +- ElasticSearch - Query Latency +- ElasticSearch - Fetch Latency +- ElasticSearch - Suggest Latency +- ElasticSearch - Scroll Latency +- ElasticSearch - Indexing Latency +- ElasticSearch - JVM GC Collection Counts +- ElasticSearch - JVM GC Latency +- ElasticSearch - JVM Heap Usage + +### Plugins +- [`elasticsearch` plugin](/{{< latest "telegraf" >}}/plugins/#elasticsearch) + + +## InfluxDB +The InfluxDB dashboard displays the following information: + +- System Uptime +- System Load +- Network +- Memory Usage +- CPU Utilization % +- Filesystems Usage +- # Measurements +- nCPU +- # Series +- # Measurements per DB +- # Series per DB +- InfluxDB Memory Heap +- InfluxDB Active Requests +- InfluxDB - HTTP Requests/Min +- InfluxDB GC Activity +- InfluxDB - Written Points/Min +- InfluxDB - Query Executor Duration +- InfluxDB - Write Errors +- InfluxDB - Client Errors +- # CQ/Minute + +### Plugins + +- [`influxdb` plugin](/{{< latest "telegraf" >}}/plugins/#influxdb) +- [`cpu` plugin](/{{< latest "telegraf" >}}/plugins/#cpu) +- [`system` plugin](/{{< latest "telegraf" >}}/plugins/#system) +- [`mem` plugin](/{{< latest "telegraf" >}}/plugins/#mem) +- [`diskio` plugin](/{{< latest "telegraf" >}}/plugins/#diskio) +- [`net` plugin](/{{< latest "telegraf" >}}/plugins/#net) + + + +## Memcached +The Memcached dashboard displays the following information: + +- Memcached - Current Connections +- Memcached - Get Hits/Second +- Memcached - Get Misses/Second +- Memcached - Delete Hits/Second +- Memcached - Delete Misses/Second +- Memcached - Incr Hits/Second +- Memcached - Incr Misses/Second +- Memcached - Current Items +- Memcached - Total Items +- Memcached - Bytes Stored +- Memcached - Bytes Read/Sec +- Memcached - Bytes Written/Sec +- Memcached - Evictions/10 Seconds + +### Plugins +- [`memcached` plugin](/{{< latest "telegraf" >}}/plugins/#memcached) + + +## NSQ +The NSQ dashboard displays the following information: + +- NSQ - Channel Client Count +- NSQ - Channel Messages Count +- NSQ - Topic Count +- NSQ - Server Count +- NSQ - Topic Messages +- NSQ - Topic Messages on Disk +- NSQ - Topic Ingress +- NSQ - Topic Egress + +### Plugins +- [`nsq` plugin](/{{< latest "telegraf" >}}/plugins/#nsq) + +## PostgreSQL +The PostgreSQL dashboard displays the following information: + +- System Uptime +- nCPU +- System Load +- Total Memory +- Memory Usage +- Filesystems Usage +- CPU Usage +- System Load +- I/O +- Network +- Processes +- Swap +- PostgreSQL rows out/sec +- PostgreSQL rows in/sec +- PostgreSQL - Buffers +- PostgreSQL commit/rollback per sec +- Postgres deadlocks/conflicts + +### Plugins + +- [`postgresql` plugin](/{{< latest "telegraf" >}}/plugins/#postgresql) +- [`system` plugin](/{{< latest "telegraf" >}}/plugins/#system) +- [`mem` plugin](/{{< latest "telegraf" >}}/plugins/#mem) +- [`cpu` plugin](/{{< latest "telegraf" >}}/plugins/#cpu) +- [`diskio` plugin](/{{< latest "telegraf" >}}/plugins/#diskio) + + +## HAProxy +The HAProxy dashboard displays the following information: + +- HAProxy - Number of Servers +- HAProxy - Sum HTTP 2xx +- HAProxy - Sum HTTP 4xx +- HAProxy - Sum HTTP 5xx +- HAProxy - Frontend HTTP Requests/Second +- HAProxy - Frontend Sessions/Second +- HAProxy - Frontend Session Usage % +- HAProxy - Frontend Security Denials/Second +- HAProxy - Frontend Request Errors/Second +- HAProxy - Frontend Bytes/Second +- HAProxy - Backend Average Response Time +- HAProxy - Backend Connection Errors/Second +- HAProxy - Backend Queued Requests/Second +- HAProxy - Backend Average Requests Queue Time (ms) +- HAProxy - Backend Error Responses/Second + +### Plugins +- [`haproxy` plugin](/{{< latest "telegraf" >}}/plugins/#haproxy) + + +## NGINX +The NGINX dashboard displays the following information: + +- NGINX - Client Connection +- NGINX - Client Errors +- NGINX - Client Requests +- NGINX - Active Client State + +### Plugins +- [`nginx` plugin](/{{< latest "telegraf" >}}/plugins/#nginx) + +## Redis +The Redis dashboard displays the following information: + +- Redis - Connected Clients +- Redis - Blocked Clients +- Redis - CPU +- Redis - Memory + +### Plugins +- [`redis` plugin](/{{< latest "telegraf" >}}/plugins/#redis) + + +## VMware vSphere VMs +The VMWare vSphere VMs dashboard gives an overview of your VMware vSphere virtual machines and includes metrics from the `vsphere_vm_*` set of measurements. It displays the following information: + +- Uptime for :vmname: +- CPU Usage for :vmname: +- RAM Usage for :vmname: +- CPU Usage Average for :vmname: +- RAM Usage Average for :vmname: +- CPU Ready Average % for :vmname: +- Network Usage for:vmname: +- Total Disk Latency for :vmname: + +### Plugins +- [`vsphere` plugin](/{{< latest "telegraf" >}}/plugins/#vsphere) + +## VMware vSphere Hosts + +The VMWare vSphere Hosts dashboard displays the following information: + +- Uptime for :esxhostname: +- CPU Usage for :esxhostname: +- RAM Usage for :esxhostname: +- CPU Usage Average for :esxhostname: +- RAM Usage Average for :esxhostname: +- CPU Ready Average % for :esxhostname: +- Network Usage for :esxhostname: +- Total Disk Latency for :esxhostname: + +### Plugins +- [`vsphere` plugin](/{{< latest "telegraf" >}}/plugins/#vsphere) + +## PHPfpm +The PHPfpm dashboard displays the following information: + +- PHPfpm - Accepted Connections +- PHPfpm - Processes +- PHPfpm - Slow Requests +- PHPfpm - Max Children Reached + +### Plugins +- [`phpfpm` plugin](/{{< latest "telegraf" >}}/plugins/#nginx) + +## Win System +The Win System dashboard displays the following information: + +- System - CPU Usage +- System - Available Bytes +- System - TX Bytes/Second +- System - RX Bytes/Second +- System - Load + +### Plugins +- [`win_services` plugin](/{{< latest "telegraf" >}}/plugins/#windows-services) + + +## MySQL +The MySQL dashboard displays the following information: + +- System Uptime +- nCPU +- MySQL uptime +- Total Memory +- System Load +- Memory Usage +- InnoDB Buffer Pool Size +- InnoDB Buffer Usage +- Max Connections +- Open Connections +- I/O +- Network +- MySQL Connections/User +- MySQL Received Bytes/Sec +- MySQL Sent Bytes/Sec +- MySQL Connections +- MySQL Queries/Sec +- MySQL Slow Queries +- InnoDB Data + +### Plugins +- [`mySQL` plugin](/{{< latest "telegraf" >}}/plugins/#mysql) +- [`system` plugin](/{{< latest "telegraf" >}}/plugins/#system) +- [`mem` plugin](/{{< latest "telegraf" >}}/plugins/#mem) + +## Ping +The Ping dashboard displays the following information: + +- Ping - Packet Loss Percent +- Ping - Response Times (ms) + +### Plugins +- [`ping` plugin](/{{< latest "telegraf" >}}/plugins/#ping) diff --git a/content/chronograf/v1.9/administration/restoring-chronograf-db.md b/content/chronograf/v1.9/administration/restoring-chronograf-db.md new file mode 100644 index 000000000..2e79ee94c --- /dev/null +++ b/content/chronograf/v1.9/administration/restoring-chronograf-db.md @@ -0,0 +1,82 @@ +--- +title: Restore a Chronograf database +description: > + If you're rolling back to a previous version of Chronograf, restore your internal database. +menu: + chronograf_1_9: + weight: 110 + parent: Administration +--- + +Chronograf uses [Bolt](https://github.com/boltdb/bolt) to store Chronograf-specific key-value data. +Generally speaking, you should never have to manually administer your internal Chronograf database. +However, rolling back to a previous version of Chronograf does require restoring +the data and data-structure specific to that version. + +Chronograf's internal database, `chronograf-v1.db`, is stored at your specified +[`--bolt-path`](/chronograf/v1.9/administration/config-options/#bolt-path-b) which, +by default, is the current working directory where the `chronograf` binary is executed. +In the upgrade process, an unmodified backup of your Chronograf data is stored inside the +`backup` directory before any necessary migrations are run. +This is done as a convenience in case issues arise with the data migrations +or the upgrade process in general. + +The `backup` directory contains a copy of your previous `chronograf-v1.db` file. +Each backup file is appended with the corresponding Chronograf version. +For example, if you moved from Chronograf 1.4.4.2 to {{< latest-patch >}}, there will be a +file called `backup/chronograf-v1.db.1.4.4.2`. + +_**Chronograf backup directory structure**_ +{{% filesystem-diagram %}} +- chronograf-working-dir/ + - chronograf-v1.db + - backup/ + - chronograf-v1.db.1.4.4.0 + - chronograf-v1.db.1.4.4.1 + - chronograf-v1.db.1.4.4.2 + - ... +{{% /filesystem-diagram %}} + +## Roll back to a previous version +If there is an issue during the upgrade process or you simply want/need to roll +back to an earlier version of Chronograf, you must restore the data file +associated with that specific version, then downgrade and restart Chronograf. + +The process is as follows: + +### 1. Locate your desired backup file +Inside your `backup` directory, locate the database file with a the appended Chronograf +version that corresponds to the version to which you are rolling back. +For example, if rolling back to 1.4.4.2, find `backup/chronograf-v1.db.1.4.4.2`. + +### 2. Stop your Chronograf server +Stop the Chronograf server by killing the `chronograf` process. + +### 3. Replace your current database with the backup +Remove the current database file and replace it with the desired backup file: + +```bash +# Remove the current database +rm chronograf-v1.db + +# Replace it with the desired backup file +cp backup/chronograf-v1.db.1.4.4.2 chronograf-v1.db +``` + +### 4. Install the desired Chronograf version +Install the desired Chronograf version. +Chronograf releases can be viewed and downloaded either from the +[InfluxData downloads](https://portal.influxdata.com/downloads) +page or from the [Chronograf releases](https://github.com/influxdata/chronograf/releases) +page on Github. + +### 5. Start the Chronograf server +Restart the Chronograf server. +Chronograf will use the `chronograf-v1.db` in the current working directory. + +## Rerun update migrations +This process can also be used to rerun Chronograf update migrations. +Go through steps 1-5, but on [step 3](#3-replace-your-current-database-with-the-backup) +select the backup you want to use as a base for the migrations. +When Chronograf starts again, it will automatically run the data migrations +required for the installed version. diff --git a/content/chronograf/v1.9/administration/upgrading.md b/content/chronograf/v1.9/administration/upgrading.md new file mode 100644 index 000000000..ebe73fcba --- /dev/null +++ b/content/chronograf/v1.9/administration/upgrading.md @@ -0,0 +1,19 @@ +--- +title: Upgrade Chronograf +description: Upgrade to the latest version of Chronograf. +menu: + chronograf_1_9: + name: Upgrade + weight: 10 + parent: Administration +--- + +If you're upgrading from Chronograf 1.3.x, first install 1.7.x, and then install {{< latest-patch >}}. + +If you're upgrading from Chronograf 1.4 or later, [download and install](https://portal.influxdata.com/downloads) the most recent version of Chronograf, and then restart Chronograf. + +{{% note %}} +Installing a new version of Chronograf automatically clears the localStorage settings. +{{% /note %}} + +After upgrading, see [Getting Started](/chronograf/v1.9/introduction/getting-started/) to get up and running. diff --git a/content/chronograf/v1.9/guides/_index.md b/content/chronograf/v1.9/guides/_index.md new file mode 100644 index 000000000..187068dda --- /dev/null +++ b/content/chronograf/v1.9/guides/_index.md @@ -0,0 +1,12 @@ +--- +title: Guides for Chronograf +description: Step-by-step instructions for using Chronograf's features. +menu: + chronograf_1_9: + name: Guides + weight: 30 +--- + +Follow the links below to explore Chronograf's features. + +{{< children >}} diff --git a/content/chronograf/v1.9/guides/advanced-kapacitor.md b/content/chronograf/v1.9/guides/advanced-kapacitor.md new file mode 100644 index 000000000..0617ac08b --- /dev/null +++ b/content/chronograf/v1.9/guides/advanced-kapacitor.md @@ -0,0 +1,77 @@ +--- +title: Advanced Kapacitor usage +description: > + Use Kapacitor with Chronograf to manage alert history, TICKscripts, and Flux tasks. +menu: + chronograf_1_9: + weight: 100 + parent: Guides +--- + +Chronograf provides a user interface for [Kapacitor](/{{< latest "kapacitor" >}}/), +InfluxData's processing framework for creating alerts, running ETL jobs, and detecting anomalies in your data. +Learn how Kapacitor interacts with Chronograf. + +- [Manage Kapacitor alerts](#manage-kapacitor-alerts) +- [Manage Kapacitor tasks](#manage-kapacitor-tasks) + +## Manage Kapacitor alerts + +Chronograf provides information about Kapacitor alerts on the Alert History page. +Chronograf writes Kapacitor alert data to InfluxDB as time series data. +It stores the data in the `alerts` [measurement](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#measurement) +in the `chronograf` database. +By default, this data is subject to an infinite [retention policy](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#retention-policy-rp) (RP). +If you expect to have a large number of alerts or do not want to store your alert +history forever, consider shortening the [duration](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#duration) +of the default retention policy. + +### Modify the retention policy of the chronograf database + +Use the Chronograf **Admin page** to modify the retention policy in the `chronograf` database. +In the Databases tab: + +1. Click **{{< icon "crown" >}} InfluxDB Admin** in the left navigation bar. +2. Hover over the retention policy list of the `chronograf` database and click **Edit** + next to the retention policy to update. +3. Update the **Duration** of the retention policy. + The minimum supported duration is one hour (`1h`) and the maximum is infinite (`INF` or `∞`). + _See [supported duration units](/{{< latest "influxdb" "v1" >}}/query_language/spec/#duration-units)._ +4. Click **Save**. + +If you set the retention policy's duration to one hour (`1h`), InfluxDB +automatically deletes any alerts that occurred before the past hour. + +## Manage Kapacitor tasks + +- [Manage Kapacitor TICKscripts](#manage-kapacitor-tickscripts) +- [Manage Kapacitor Flux tasks](#manage-kapacitor-flux-tasks) + +### Manage Kapacitor TICKscripts + +Chronograf lets you manage Kapacitor TICKscript tasks created in Kapacitor or in +Chronograf when [creating a Chronograf alert rule](/chronograf/v1.9/guides/create-alert-rules/). + +To manage Kapacitor TICKscript tasks in Chronograf, click +**{{< icon "alert">}} Alerts** in the left navigation bar. +On this page, you can: + +- View Kapacitor TICKscript tasks. +- View TICKscript task activity. +- Create new TICKscript tasks. +- Update TICKscript tasks. +- Enable and disable TICKscript tasks. +- Delete TICKscript tasks. + +### Manage Kapacitor Flux tasks +**Kapacitor 1.6+** supports Flux tasks. +Chronograf lets you view and manage Kapacitor Flux tasks. + +To manage Kapacitor Flux tasks in Chronograf, click +**{{< icon "alert">}} Alerts** in the left navigation bar. +On this page, you can: + +- View Kapacitor Flux tasks. +- View Kapacitor Flux task activity. +- Enable and disable Kapacitor Flux tasks. +- Delete Kapacitor Flux tasks. diff --git a/content/chronograf/v1.9/guides/analyzing-logs.md b/content/chronograf/v1.9/guides/analyzing-logs.md new file mode 100644 index 000000000..8af2e24fb --- /dev/null +++ b/content/chronograf/v1.9/guides/analyzing-logs.md @@ -0,0 +1,109 @@ +--- +title: Analyze logs with Chronograf +description: Analyze log information using Chronograf. +menu: + chronograf_1_9: + weight: 120 + parent: Guides +--- + +Chronograf gives you the ability to view, search, filter, visualize, and analyze log information from a variety of sources. +This helps to recognize and diagnose patterns, then quickly dive into logged events that lead up to events. + +- [Set up logging](#set-up-logging) +- [View logs in Chronograf](#view-logs-in-chronograf) +- [Configure the log viewer](#configure-the-log-viewer) +- [Show or hide the log status histogram](#show-or-hide-the-log-status-histogram) +- [Logs in dashboards](#logs-in-dashboards) + +## Set up logging +Logs data is a first class citizen in InfluxDB and is populated using available log-related [Telegraf input plugins](/{{< latest "telegraf" >}}/plugins/#input-plugins): + +- [Docker Log](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/docker_log/README.md) +- [Graylog](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/graylog/README.md) +- [Logparser](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/logparser/README.md) +- [Logstash](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/logstash/README.md) +- [Syslog](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/syslog/README.md) +- [Tail](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/tail/README.md) + +## View logs in Chronograf +Chronograf has a dedicated log viewer accessed by clicking the **Log Viewer** button in the left navigation. + +{{< img-hd src="/img/chronograf/1-6-logs-nav-log-viewer.png" alt="Log viewer in the left nav" />}} + +The log viewer provides a detailed histogram showing the time-based distribution of log entries color-coded by log severity. +It also includes a live stream of logs that can be searched, filtered, and paused to analyze specific time ranges. +Logs are pulled from the `syslog` measurement. +_Other log inputs and alternate log measurement options will be available in future updates._ + +{{< img-hd src="/img/chronograf/1-7-log-viewer-overview.png" alt="Chronograf log viewer" />}} + +### Search and filter logs +Search for logs using keywords or regular expressions. +They can also be filtered by clicking values in the log table such as `severity` or `facility`. +Any tag values included with the log entry can be used as a filter. + +You can also use search operators to filter your results. For example, if you want to find results with a severity of critical that don't mention RSS, you can enter: `severity == crit` and `-RSS`. + +![Searching and filtering logs](/img/chronograf/1-7-log-viewer-search-filter.gif) + +{{% note %}} +**Note:** The log search field is case-sensitive. +{{% /note %}} + +To remove filters, click the `×` next to the tag key by which you no longer want to filter. + +### Select specific times +In the log viewer, you can select time ranges from which to view logs. +By default, logs are streamed and displayed relative to "now," but it is possible to view logs from a past window of time. +timeframe selection allows you to go to to a specific event and see logs for a time window both preceding and following that event. The default window is one minute, meaning the graph shows logs from thirty seconds before and the target time. Click the dropdown menu change the window. + +![Selecting time ranges](/img/chronograf/1-7-log-viewer-specific-time.gif) + +## Configure the log viewer +The log viewer can be customized to fit your specific needs. +Open the log viewer configuration options by clicking the gear button in the top right corner of the log viewer. Once done, click **Save** to apply the changes. + +{{< img-hd src="/img/chronograf/1-6-logs-log-viewer-config-options.png" alt="Log viewer configuration options" />}} + +### Severity colors +Every log severity is assigned a color which is used in the display of log entries. +To customize colors, select a color from the available color dropdown. + +### Table columns +Columns in the log viewer are auto-populated with all fields and tags associated with your log data. +Each column can be reordered, renamed, and hidden or shown. + +### Severity format +"Severity Format" specifies how the severity of log entries is displayed in your log table. +Below are the options and how they appear in the log table: + +| Severity Format | Display | +| --------------- |:------- | +| Dot | Log serverity format 'Dot' | +| Dot + Text | Log serverity format 'Dot + Text' | +| Text | Log serverity format 'Text' | + +### Truncate or wrap log messages +By default, text in Log Viewer columns is truncated if it exceeds the column width. You can choose to wrap the text instead to display the full content of each cell. + +Select the **Truncate** or **Wrap** option to determine how text appears when it exceeds the width of the cell. + +To copy the complete, un-truncated log message, select the message cell and click **Copy**. + +## Show or hide the log status histogram +The Chronograf Log Viewer displays a histogram of log status. + +**To hide the log status histogram**, click the **{{< icon "hide" >}} icon** in +the top right corner of the histogram. + +**To show the log status histogram**, click the **{{< icon "bar-chart" >}} icon** +in the top right corner of the log output. + +## Logs in dashboards +An incredibly powerful way to analyze log data is by creating dashboards that include log data. +This is possible by using the [Table visualization type](/chronograf/v1.9/guides/visualization-types/#table) to display log data in your dashboard. + +![Correlating logs with other metrics](/img/chronograf/1-7-log-viewer-dashboard.gif) + +This type of visualization allows you to quickly identify anomalies in other metrics and see logs associated with those anomalies. diff --git a/content/chronograf/v1.9/guides/annotations.md b/content/chronograf/v1.9/guides/annotations.md new file mode 100644 index 000000000..f217c90dc --- /dev/null +++ b/content/chronograf/v1.9/guides/annotations.md @@ -0,0 +1,43 @@ +--- +title: Use annotations in Chronograf views +description: > + Add contextual information to Chronograf dashboards with annotations. +menu: + chronograf_1_9: + name: Use annotations + weight: 50 + parent: Guides +--- + +## Use annotations in the Chronograf interface + +Annotations in Chronograf are notes of explanation or comments added to graph views by editors or administrators. Annotations can provide Chronograf users with useful contextual information about single points in time or time intervals. Users can use annotations to correlate the effects of important events, such as system changes or outages across multiple metrics, with Chronograf data. + +When an annotation is added, a solid white line appears on all graph views for that point in time or an interval of time. + +### Annotations example + +The following screenshot of five graph views displays annotations for a single point in time and a time interval. +The text and timestamp for the single point in time can be seem above the annotation line in the graph view on the lower right. +The annotation displays "`Deploy v3.8.1-2`" and the time "`2018/28/02 15:59:30:00`". + +![Annotations on multiple graph views](/img/chronograf/1-6-annotations-example.png) + + +**To add an annotation using the Chronograf user interface:** + +1. Click the **Edit** button ("pencil" icon) on the graph view. +2. Click **Add Annotation** to add an annotation. +3. Move cursor to point of time and click or drag cursor to set an annotation. +4. Click **Edit** again and then click **Edit Annotation**. +5. Click the cursor on the annotation point or interval. The annotation text box appears above the annotation point or interval. +6. Click on `Name Me` in the annotation and type a note or comment. +7. Click **Done Editing**. +8. Your annotation is now available in all graph views. + +{{% note %}} +Annotations are not associated with specific dashboards and appear in all dashboards. +Annotations are managed per InfluxDB data source. +When a dashboard is deleted, annotation persist until the InfluxDB data source +the annotations are associated with is removed. +{{% /note %}} diff --git a/content/chronograf/v1.9/guides/cloning-in-ui.md b/content/chronograf/v1.9/guides/cloning-in-ui.md new file mode 100644 index 000000000..8fb779195 --- /dev/null +++ b/content/chronograf/v1.9/guides/cloning-in-ui.md @@ -0,0 +1,43 @@ +--- +title: Clone dashboards and cells +description: > + Clone a dashboard or a cell and use the copy as a starting point to create new dashboard or cells. +menu: + chronograf_1_9: + weight: 70 + parent: Guides +--- + +This guide explains how to clone, or duplicate, a dashboard or a cell for use as starting points for creating dashboards or cells using the copy as a template. + +## Clone dashboards + +Dashboards in Chronograf can be cloned (or copied) to be used to create a dashboard based on the original. Rather than building a new dashboard from scratch, you can clone a dashboard and make changes to the dashboard copy. + +### To clone a dashboard + +1. On the **Dashboards** page, hover your cursor over the listing of the dashboard that you want to clone and click the **Clone** button that appears. + + ![Click the Clone button](/img/chronograf/1-6-clone-dashboard.png) + + The cloned dashboard opens and displays the name of the original dashboard with `(clone)` after it. + + ![Cloned dashboard](/img/chronograf/1-6-clone-dashboard-clone.png) + + You can now change the dashboard name and customize the dashboard. + +## Clone cells + +Cells in Chronograf dashboards can be cloned, or copied, to quickly create a cell copy that can be edited for another use. + +### To clone a cell + +1. On the dashboard cell that you want to make a copy of, click the **Clone** icon and then confirm by clicking **Clone Cell**. + + ![Click the Clone icon](/img/chronograf/1-6-clone-cell-click-button.png) + +2. The cloned cell appears in the dashboard displaying the nameof the original cell with `(clone)` after it. + + ![Cloned cell](/img/chronograf/1-6-clone-cell-cell-copy.png) + + You can now change the cell name and customize the cell. diff --git a/content/chronograf/v1.9/guides/configuring-alert-endpoints.md b/content/chronograf/v1.9/guides/configuring-alert-endpoints.md new file mode 100644 index 000000000..e12cd0ce2 --- /dev/null +++ b/content/chronograf/v1.9/guides/configuring-alert-endpoints.md @@ -0,0 +1,370 @@ +--- +title: Configure Chronograf alert endpoints +aliases: + - /chronograf/v1.9/guides/configure-kapacitor-event-handlers/ +description: Send alert messages with Chronograf alert endpoints. +menu: + chronograf_1_9: + name: Configure alert endpoints + weight: 70 + parent: Guides +--- + +Chronograf alert endpoints can be configured using the Chronograf user interface to create Kapacitor-based event handlers that send alert messages. +You can use Chronograf to send alert messages to specific URLs as well as to applications. + +This guide offers step-by-step instructions for configuring Chronograf alert endpoints. + +## Kapacitor event handlers supported in Chronograf + +Chronograf integrates with [Kapacitor](/{{< latest "kapacitor" >}}/), InfluxData's data processing platform, to send alert messages to event handlers. +Chronograf supports the following event handlers: + +- [Alerta](#alerta) +- [BigPanda](#bigpanda) +- [Kafka](#kafka) +- [OpsGenie](#opsgenie) +- [OpsGenie2](#opsgenie2) +- [PagerDuty](#pagerduty) +- [PagerDuty2](#pagerduty2) +- [Pushover](#pushover) +- [Sensu](#sensu) +- [ServiceNow](#servicenow) +- [Slack](#slack) +- [SMTP](#smtp) +- [Talk](#talk) +- [Teams](#talk) +- [Telegram](#telegram) +- [VictorOps](#victorops) +- [Zenoss](#zenoss) + +To configure a Kapacitor event handler in Chronograf, [install Kapacitor](/{{< latest "kapacitor" >}}/introduction/installation/) and [connect it to Chronograf](/{{< latest "kapacitor" >}}/working/kapa-and-chrono/#add-a-kapacitor-instance). +The **Configure Kapacitor** page includes the event handler configuration options. + +## Alert endpoint configurations + +Alert endpoint configurations appear on the Chronograf Configure Kapacitor page. +You must have a connected Kapacitor instance to access the configurations. +For more information, see [Kapacitor installation instructions](/{{< latest "kapacitor" >}}/introduction/installation/) and how to [connect a Kapacitor instance](/{{< latest "kapacitor" >}}/working/kapa-and-chrono/#add-a-kapacitor-instance) to Chronograf. + +Note that the configuration options in the **Configure alert endpoints** section are not all-inclusive. +Some event handlers allow users to customize event handler configurations per [alert rule](/chronograf/v1.9/guides/create-a-kapacitor-alert/). +For example, Chronograf's Slack integration allows users to specify a default channel in the **Configure alert endpoints** section and a different channel for individual alert rules. + +### Alerta + +**To configure an Alerta alert endpoint:** + +1. In the **Configure Alert Endpoints** of the **Configure Kapacitor Connection** page, click the **Alerta** tab. +2. Enter the following: + + - **Environment**: Alerta environment. Can be a template and has access to the same data as the AlertNode.Details property. Default is set from the configuration. + - **Origin**: Alerta origin. If empty, uses the origin from the configuration. + - **Token**: Default Alerta authentication token.. + - **Token Prefix**: Default token prefix. If you receive invalid token errors, you may need to change this to “Key”. + - **User**: Alerta user. + - **Configuration Enabled**: Check to enable configuration. + +3. Click **Save Changes** to save the configuration settings. +4. Click **Send Test Alert** to verify the configuration. + +### BigPanda + +**To configure an BigPanda alert endpoint:** + +1. In the **Configure Alert Endpoints** of the **Configure Kapacitor Connection** page, + click the **BigPanda** tab. +2. Enter the following: + + - **URL**: BigPanda [alerts API URL](https://docs.bigpanda.io/reference#alerts-how-it-works). + Default is `https://api.bigpanda.io/data/v2/alerts`. + - **Token**: BigPanda [API Authorization token (API key)](https://docs.bigpanda.io/docs/api-key-management). + - **Application Key**: BigPanda [App Key](https://docs.bigpanda.io/reference#integrating-monitoring-systems). + - **Insecure Skip Verify**: Required if using a self-signed TLS certificate. Select to skip TLS certificate chain and host + verification when connecting over HTTPS. + - **Configuration Enabled**: Select to enable configuration. + +3. Click **Save Changes** to save the configuration settings. +4. Click **Send Test Alert** to verify the configuration. + +### Kafka + +**To configure a Kafka alert endpoint:** + +1. In the **Configure Alert Endpoints** of the **Configure Kapacitor Connection** page, + click the **Kafka** tab. +2. Enter the following: + + - **ID**: Unique identifier for a Kafka cluster. Default is `localhost`. + - **Brokers**: List of Kafka broker addresses, using the `host:port` format. + - **Timeout**: Maximum amount of time to wait before flushing an incomplete batch. Default is `10s`. + - **Batch Size**: Number of messages batched before sending to Kafka. Default is `100`. + - **Batch Timeout**: Timeout period for the batch. Default is `1s`. + - **Use SSL**: Select to enable SSL communication. + - **SSL CA**: Path to the SSL CA (certificate authority) file. + - **SSL Cert**: Path to the SSL host certificate. + - **SSL Key**: Path to the SSL certificate private key file. + - **Insecure Skip Verify**: Required if using a self-signed TLS certificate. Select to skip TLS certificate chain and host + verification when connecting over HTTPS. + - **Configuration Enabled**: Check to enable configuration. + +3. Click **Save Changes** to save the configuration settings. +4. Click **Send Test Alert** to verify the configuration. + +To enable Kafka services using TICKscript, see [Kafka event handler (Kapacitor)](/{{< latest "kapacitor" >}}/event_handlers/kafka/). + +### OpsGenie + +{{% warn %}} +**Note:** Support for OpsGenie Events API 1.0 is deprecated (as [noted by OpGenie](https://docs.opsgenie.com/docs/migration-guide-for-alert-rest-api)). +As of June 30, 2018, the OpsGenine Events API 1.0 is disabled. +Use the [OpsGenie2](#opsgenie2) alert endpoint. +{{% /warn %}} + +### OpsGenie2 + +Send an incident alert to OpsGenie teams and recipients using the Chronograf alert endpoint. + +**To configure a OpsGenie alert endpoint:** + +1. In the **Configure Alert Endpoints** of the **Configure Kapacitor Connection** page, + click the **OpsGenie** tab. +2. Enter the following information: + + - **API Key**: API key (or GenieKey). + To find the API key, sign into your [OpsGenie account](https://app.opsgenie.com/auth/login) + and select the **Settings** menu option in the **Admin** menu. + - **Teams**: List of [OpsGenie teams](https://docs.opsgenie.com/docs/teams) to be alerted. + - **Recipients** List of [OpsGenie team members](https://docs.opsgenie.com/docs/teams#section-team-members)) to receive alerts. + - **Select recovery action**: Actions to take when an alert recovers: + - Add a note to the alert + - Close the alert + - **Configuration Enabled**: Select to enable configuration. + +4. Click **Save Changes** to save the configuration settings. +5. Click **Send Test Alert** to verify the configuration. + +See [Alert API](https://docs.opsgenie.com/docs/alert-api) in the OpsGenie documentation for details on the OpsGenie Alert API + +See [OpsGenie V2 event handler](/{{< latest "kapacitor" >}}/event_handlers/opsgenie/v2/) in the Kapacitor documentation for details about the OpsGenie V2 event handler. + +See the [AlertNode (Kapacitor TICKscript node) - OpsGenie v2](/{{< latest "kapacitor" >}}/nodes/alert_node/#opsgenie-v2) in the Kapacitor documentation for details about enabling OpsGenie services using TICKscripts. + +### PagerDuty + +{{% warn %}} +The original PagerDuty alert endpoint is deprecated. +Use the [PagerDuty2](#pagerduty2) alert endpoint. +{{% /warn %}} + +### PagerDuty2 + +**To configure a PagerDuty alert endpoint:** + +1. In the **Configure Alert Endpoints** of the **Configure Kapacitor Connection** page, + click the **PagerDuty** tab. +2. Enter the following: + + - **Routing Key**: GUID of your PagerDuty Events API V2 integration, listed as "Integration Key" on the Events API V2 integration's detail page. See [Create a new service](https://support.pagerduty.com/docs/services-and-integrations#section-create-a-new-service) in the PagerDuty documentation details on getting an "Integration Key" (`routing_key`). + - **PagerDuty URL**: URL used to POST a JSON body representing the event. This value should not be changed. Valid value is `https://events.pagerduty.com/v2/enqueue`. + - **Configuration Enabled**: Select to enable this configuration. + +3. Click **Save Changes** to save the configuration settings. +4. Click **Send Test Alert** to verify the configuration. + +See the [PagerDuty Events API V2 Overview](https://v2.developer.pagerduty.com/docs/events-api-v2) +for details on the PagerDuty Events API and recognized event types (`trigger`, `acknowledge`, and `resolve`). + +To enable a new "Generic API" service using TICKscript, see [AlertNode (Kapacitor TICKscript node) - PagerDuty v2](/{{< latest "kapacitor" >}}/nodes/alert_node/#pagerduty-v2). + +### Pushover + +**To configure a Pushover alert endpoint:** + +1. In the **Configure Alert Endpoints** of the **Configure Kapacitor Connection** page, + click the **Pushover** tab. +2. Enter the following: + + - **User Key**: Pushover USER_TOKEN. + - **Token**: Pushover API token. + - **Pushover URL**: Pushover API URL. + Default is `https://api.pushover.net/1/messages.json`. + - **Configuration Enabled**: Check to enable configuration. + +3. Click **Save Changes** to save the configuration settings. +4. Click **Send Test Alert** to verify the configuration. + +### Sensu + +**To configure a Sensu alert endpoint:** + +1. In the **Configure Alert Endpoints** of the **Configure Kapacitor Connection** page, + click the **Sensu** tab. +2. Enter the following: + + - **Source**: Event source. Default is `Kapacitor`. + - **Address**: URL of [Sensu HTTP API](https://docs.sensu.io/sensu-go/latest/migrate/#architecture). + - **Configuration Enabled**: Select to enable configuration. + +3. Click **Save Changes** to save the configuration settings. +4. Click **Send Test Alert** to verify the configuration. + +### ServiceNow + +**To configure a ServiceNow alert endpoint:** + +1. In the **Configure Alert Endpoints** of the **Configure Kapacitor Connection** page, + click the **ServiceNow** tab. +2. Enter the following: + + - **URL**: ServiceNow API URL. Default is `https://instance.service-now.com/api/global/em/jsonv2`. + - **Source**: Event source. + - **Username**: ServiceNow username. + - **Password**: ServiceNow password. + - **Configuration Enabled**: Select to enable configuration. + +3. Click **Save Changes** to save the configuration settings. +4. Click **Send Test Alert** to verify the configuration. + +### Slack + +**To configure a Slack alert endpoint:** + +1. In the **Configure Alert Endpoints** of the **Configure Kapacitor Connection** page, + click the **Slack** tab. +2. Enter the following: + + - **Nickname this Configuration**: Unique name for a Slack endpoint if you + have more than one Slack alert endpoint. + - **Slack WebHook URL**: _(Optional)_ Slack webhook URL _(see [Slack webhooks](https://api.slack.com/messaging/webhooks))_ + - **Slack Channel**: _(Optional)_ Slack channel or user to send messages to. + Prefix with `#` to send to a channel. + Prefix with `@` to send directly to a user. + If not specified, Kapacitor sends alert messages to the channel or user + specified in the [alert rule](/chronograf/v1.9/guides/create-a-kapacitor-alert/) + or configured in the **Slack Webhook**. + - **Configuration Enabled**: Check to enable configuration. + +3. Click **Save Changes** to save the configuration settings. +4. Click **Send Test Alert** to verify the configuration. + + +**To add another Slack configuration:** + +1. In the **Configure Alert Endpoints** of the **Configure Kapacitor Connection** page, + click the **Slack** tab. +2. Click **{{< icon "plus" >}} Add Another Config**. +3. Complete steps 2-4 [above](#slack). + +### SMTP + +**To configure a SMTP alert endpoint:** + +1. In the **Configure Alert Endpoints** of the **Configure Kapacitor Connection** page, + click the **SMTP** tab. +2. Enter the following: + + - **SMTP Host**: SMTP host. Default is `localhost`. + - **SMTP Port**: SMTP port. Default is `25`. + - **From Email**: Email address to send messages from. + - **To Email**: Email address to send messages to. + - **User**: SMTP username. + - **Password**: SMTP password. + - **Configuration Enabled**: Select to enable configuration. + +3. Click **Save Changes** to save the configuration settings. +4. Click **Send Test Alert** to verify the configuration. + +### Talk + +**To configure a Talk alert endpoint:** + +1. In the **Configure Alert Endpoints** of the **Configure Kapacitor Connection** page, + click the **Talk** tab. +2. Enter the following: + + - **URL**: Talk API URL. + - **Author Name**: Message author name. + - **Configuration Enabled**: Select to enable configuration. + +3. Click **Save Changes** to save the configuration settings. +4. Click **Send Test Alert** to verify the configuration. + +### Teams + +**To configure a Microsoft Teams alert endpoint:** + +1. In the **Configure Alert Endpoints** of the **Configure Kapacitor Connection** page, + click the **Teams** tab. +2. Enter the following: + + - **Channel URL**: Microsoft Teams channel URL. + - **Configuration Enabled**: Select to enable configuration. + +3. Click **Save Changes** to save the configuration settings. +4. Click **Send Test Alert** to verify the configuration. + +### Telegram + +**To configure a Telegram alert endpoint:** + +1. [Set up a Telegram bot and credentials](/{{< latest "kapacitor" >}}/guides/event-handler-setup/#telegram-setup). +2. In the **Configure Alert Endpoints** of the **Configure Kapacitor Connection** page, + click the **Telegram** tab. +3. Enter the following: + + - **Token**: + - **Chat ID**: + - **Select the alert message format**: Telegram message format + - Markdown _(default)_ + - HTML + - **Disable link previews**: Disable [link previews](https://telegram.org/blog/link-preview) in Telegram messages. + - **Disable notifications**: Disable notifications on iOS devices and sounds on Android devices. + Android users will continue to receive notifications. + - **Configuration Enabled**: Select to enable configuration. + +### VictorOps + +**To configure a VictorOps alert endpoint:** + +1. In the **Configure Alert Endpoints** of the **Configure Kapacitor Connection** page, + click the **VictorOps** tab. +2. Enter the following: + + - **API Key**: VictorOps API key. + - **Routing Key**: VictorOps [routing key](https://help.victorops.com/knowledge-base/routing-keys/). + - **VictorOps URL**: VictorOps alert API URL. + Default is `https://alert.victorops.com/integrations/generic/20131114/alert`. + - **Configuration Enabled**: Select to enable configuration. + +3. Click **Save Changes** to save the configuration settings. +4. Click **Send Test Alert** to verify the configuration. + +### Zenoss + +**To configure a Zenoss alert endpoint:** + +1. In the **Configure Alert Endpoints** of the **Configure Kapacitor Connection** page, + click the **Zenoss** tab. +2. Enter the following: + + - **URL**: Zenoss [router endpoint URL](https://help.zenoss.com/zsd/RM/configuring-resource-manager/enabling-access-to-browser-interfaces/creating-and-changing-public-endpoints). + Default is `https://tenant.zenoss.io:8080/zport/dmd/evconsole_router`. + - **Username**: Zenoss username. Leave blank for no authentication. + - **Password**: Zenoss password. Leave blank for no authentication. + - **Action (Router Name)**: Zenoss [router name](https://help.zenoss.com/dev/collection-zone-and-resource-manager-apis/anatomy-of-an-api-request#AnatomyofanAPIrequest-RouterURL). + Default is `EventsRouter`. + - **Router Method**: [EventsRouter method](https://help.zenoss.com/dev/collection-zone-and-resource-manager-apis/codebase/routers/router-reference/eventsrouter). + Default is `add_event`. + - **Event Type**: Event type. Default is `rpc`. + - **Event TID**: Temporary request transaction ID. Default is `1`. + - **Collector Name**: Zenoss collector name. Default is `Kapacitor`. + - **Kapacitor to Zenoss Severity Mapping**: Map Kapacitor severities to [Zenoss severities](https://help.zenoss.com/docs/using-collection-zones/event-management/event-severity-levels). + - **OK**: Clear _(default)_ + - **Info**: Info _(default)_ + - **Warning**: Warning _(default)_ + - **Critical**: Critical _(default)_ + - **Configuration Enabled**: Select to enable configuration. + +3. Click **Save Changes** to save the configuration settings. +4. Click **Send Test Alert** to verify the configuration. diff --git a/content/chronograf/v1.9/guides/create-a-dashboard.md b/content/chronograf/v1.9/guides/create-a-dashboard.md new file mode 100644 index 000000000..ffe58fa03 --- /dev/null +++ b/content/chronograf/v1.9/guides/create-a-dashboard.md @@ -0,0 +1,164 @@ +--- +title: Create Chronograf dashboards +description: Visualize your data with custom Chronograf dashboards. +menu: + chronograf_1_9: + name: Create dashboards + weight: 30 + parent: Guides +--- + +Chronograf offers a complete dashboard solution for visualizing your data and monitoring your infrastructure: + +- View [pre-created dashboards](/chronograf/v1.9/guides/using-precreated-dashboards) from the Host List page. + Dashboards are available depending on which Telegraf input plugins you have enabled. + These pre-created dashboards cannot be cloned or edited. +- Create custom dashboards from scratch by building queries in the Data Explorer, as described [below](#build-a-dashboard). +- [Export a dashboard](/chronograf/latest/administration/import-export-dashboards/#export-a-dashboard) you create. +- Import a dashboard: + - When you want to [import an exported dashboard](/chronograf/latest/administration/import-export-dashboards/#import-a-dashboard). + - When you want to add or update a connection in Chronograf. See [Dashboard templates](#dashboard-templates) for details. + +By the end of this guide, you'll be aware of the tools available to you for creating dashboards similar to this example: + +![Chronograf dashboard](/img/chronograf/1-6-g-dashboard-possibilities.png) + +## Requirements + +To perform the tasks in this guide, you must have a working Chronograf instance that is connected to an InfluxDB source. +Data is accessed using the Telegraf [system ](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/system) input plugins. +For more information, see [Configuring Chronograf](/chronograf/v1.9/administration/configuration). + +## Build a dashboard + +1. #### Create a new dashboard + Click **Dashboards** in the navigation bar and then click the **{{< icon "plus" >}} Create Dashboard** button. + A new dashboard is created and ready to begin adding cells. + +2. #### Name your dashboard + Click **Name This Dashboard** and type a new name. For example, "ChronoDash". + +3. #### Enter cell editor mode + In the first cell, titled "Untitled Cell", click **{{< icon "plus" >}} Add Data** + to open the cell editor mode. + + {{< img-hd src="/img/chronograf/1-9-dashboard-cell-add-data.png" alt="Add data to a Chronograf cell" />}} + +4. #### Create your query + Click the **Add a Query** button to create an [InfluxQL](/{{< latest "influxdb" "v1" >}}/query_language/) query. + In query editor mode, use the builder to select from your existing data and + allow Chronograf to format the query for you. + Alternatively, manually enter and edit a query. + Chronograf allows you to move seamlessly between using the builder and + manually editing the query; when possible, the interface automatically + populates the builder with the information from your raw query. + + For our example, the query builder is used to generate a query that shows + the average idle CPU usage grouped by host (in this case, there are three hosts). + By default, Chronograf applies the [`MEAN()` function](/{{< latest "influxdb" "v1" >}}/query_language/functions/#mean) + to the data, groups averages into auto-generated time intervals (`:interval:`), + and shows data for the past hour (`:dashboardTime:`). + Those defaults are configurable using the query builder or by manually editing the query. + + In addition, the time range (`:dashboardTime:` and `:upperDashboardTime:`) are + [configurable on the dashboard](#configure-your-dashboard). + + ![Build your query](/img/chronograf/1-6-g-dashboard-builder.png) + +5. #### Choose your visualization type + Chronograf supports many different [visualization types](/chronograf/v1.9/guides/visualization-types/). To choose a visualization type, click **Visualization** and select **Step-Plot Graph**. + + ![Visualization type](/img/chronograf/1-6-g-dashboard-visualization.png) + +6. #### Save your cell + + Click **Save** (the green checkmark icon) to save your cell. + + {{% note %}} +_**Note:**_ If you navigate away from this page without clicking Save, your work will not be saved. + {{% /note %}} + +## Configure your dashboard + +### Customize cells + +- You can change the name of the cell from "Untitled Cell" by returning to the cell editor mode, clicking on the name, and renaming it. Remember to save your changes. +- **Move** your cell around by clicking its top bar and dragging it around the page +- **Resize** your cell by clicking and dragging its bottom right corner + +### Explore cell data + +- **Zoom** in on your cell by clicking and dragging your mouse over the area of interest +- **Pan** over your cell data by pressing the shift key and clicking and dragging your mouse over the graph +- **Reset** your cell by double-clicking your mouse in the cell window + +{{% note %}} +**Note:** These tips only apply to the line, stacked, step-plot, and line+stat +[visualization types](/chronograf/v1.9/guides/visualization-types/). +{{% /note %}} + +### Configure dashboard-wide settings + +- Change the dashboard's *selected time* at the top of the page - the default + time is **Local**, which uses your browser's local time. Select **UTC** to use + Coordinated Universal Time. + + {{% note %}} +**Note:** If your organization spans multiple time zones, we recommend using UTC +(Coordinated Universal Time) to ensure that everyone sees metrics and events for the same time. + {{% /note %}} + +- Change the dashboard's *auto-refresh interval* at the top of the page - the default interval selected is **Every 10 seconds**. + + {{% note %}} +**Note:** A dashboard's refresh rate persists in local storage, so the default +refresh rate is only used when a refresh rate isn't found in local storage. + {{% /note %}} + + {{% note %}} +**To add custom auto-refresh intervals**, use the [`--custom-auto-refresh` configuration +option](/chronograf/v1.9/administration/config-options/#--custom-auto-refresh) +or `$CUSTOM_AUTO_REFRESH` environment variable when starting Chronograf. + {{% /note %}} + +- Modify the dashboard's *time range* at the top of the page - the default range + is **Past 15 minutes**. + +## Dashboard templates + +Select from a variety of dashboard templates to import and customize based on which Telegraf plugins you have enabled, such as the following examples: + +###### Kubernetes dashboard template +{{< img-hd src="/img/chronograf/1-7-protoboard-kubernetes.png" alt="Kubernetes Chronograf dashboard template" />}} + +###### MySQL dashboard template +{{< img-hd src="/img/chronograf/1-7-protoboard-mysql.png" alt="MySQL Chronograf dashboard template" />}} + +###### System metrics dashboard template +{{< img-hd src="/img/chronograf/1-7-protoboard-system.png" alt="System metrics Chronograf dashboard template" />}} + +###### vSphere dashboard template +{{< img-hd src="/img/chronograf/1-7-protoboard-vsphere.png" alt="vSphere Chronograf dashboard template" />}} + +### Import dashboard templates + +1. From the Configuration page, click **Add Connection** or select an existing connection to edit it. +2. In the **InfluxDB Connection** window, enter or verify your connection details and click **Add** or **Update Connection**. +3. In the **Dashboards** window, select from the available dashboard templates to import based on which Telegraf plugins you have enabled. + + {{< img-hd src="/img/chronograf/1-7-protoboard-select.png" alt="Select dashboard template" />}} + +4. Click **Create (x) Dashboards**. +5. Edit, clone, or configure the dashboards as needed. + +## Extra Tips + +### Full screen mode + +View your dashboard in full screen mode by clicking on the full screen icon (**{{< icon "fullscreen" >}}**) in the top right corner of your dashboard. +To exit full screen mode, press the Esc key. + +### Template variables + +Dashboards support template variables. +See the [Dashboard Template Variables](/chronograf/v1.9/guides/dashboard-template-variables/) guide for more information. diff --git a/content/chronograf/v1.9/guides/create-alert-rules.md b/content/chronograf/v1.9/guides/create-alert-rules.md new file mode 100644 index 000000000..3a0e3e629 --- /dev/null +++ b/content/chronograf/v1.9/guides/create-alert-rules.md @@ -0,0 +1,162 @@ +--- +title: Create Chronograf alert rules +description: > + Trigger alerts by building Kapacitor alert rules in the Chronograf user interface (UI). +aliases: + - /chronograf/v1.9/guides/create-a-kapacitor-alert/ +menu: + chronograf_1_9: + name: Create alert rules + weight: 60 + parent: Guides +--- + + +Chronograf provides a user interface for [Kapacitor](/{{< latest "kapacitor" >}}/), InfluxData's processing framework for creating alerts, ETL jobs (running extract, transform, load), and detecting anomalies in your data. +Chronograf alert rules correspond to Kapacitor tasks that trigger alerts whenever certain conditions are met. +Behind the scenes, these tasks are stored as [TICKscripts](/{{< latest "kapacitor" >}}/tick/) that can be edited manually or through Chronograf. +Common alerting use cases that can be managed using Chronograf include: + +* Thresholds with static ceilings, floors, and ranges. +* Relative thresholds based on unit or percentage changes. +* Deadman switches. + +Complex alerts and other tasks can be defined directly in Kapacitor as TICKscripts, but can be viewed and managed within Chronograf. + +This guide walks through creating a Chronograf alert rule that sends an alert message to an existing [Slack](https://slack.com/) channel whenever your idle CPU usage crosses the 80% threshold. + +## Requirements + +[Getting started with Chronograf](/chronograf/v1.9/introduction/getting-started/) offers step-by-step instructions for each of the following requirements: + +* Downloaded and install the entire TICKstack (Telegraf, InfluxDB, Chronograf, and Kapacitor). +* Configure Telegraf to collect data using the InfluxDB [system statistics](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/system) input plugin and write data to your InfluxDB instance. +* [Create a Kapacitor connection in Chronograf](/chronograf/v1.9/introduction/installation/#connect-chronograf-to-kapacitor). +* Slack is available and configured as an event handler in Chronograf. See [Configuring Chronograf alert endpoints](/chronograf/v1.9/guides/configuring-alert-endpoints/) for detailed configuration instructions. + +## Configure Chronograf alert rules + +Navigate to the **Manage Tasks** page under **Alerting** in the left navigation, then click **+ Build Alert Rule** in the top right corner. + +![Navigate to Manage Tasks](/img/chronograf/1-6-alerts-manage-tasks-nav.png) + +The **Manage Tasks** page is used to create and edit your Chronograf alert rules. +The steps below guide you through the process of creating a Chronograf alert rule. + +![Empty Rule Configuration](/img/chronograf/1-6-alerts-rule-builder.png) + +### Step 1: Name the alert rule + +Under **Name this Alert Rule** provide a name for the alert. +For this example, use "Idle CPU Usage" as your alert name. + +### Step 2: Select the alert type + +Choose from three alert types under the **Alert Types** section of the Rule Configuration page: + +_**Threshold**_ +Alert if data crosses a boundary. + +_**Relative**_ +Alert if data changes relative to data in a different time range. + +_**Deadman**_ +Alert if InfluxDB receives no relevant data for a specified time duration. + +For this example, select the **Threshold** alert type. + +### Step 3: Select the time series data + +Choose the time series data you want the Chronograf alert rule to use. +Navigate through databases, measurements, fields, and tags to select the relevant data. + +In this example, select the `telegraf` [database](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#database), the `autogen` [retention policy](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#retention-policy-rp), the `cpu` [measurement](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#measurement), and the `usage_idle` [field](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#field). + +![Select your data](/img/chronograf/1-6-alerts-time-series.png) + +### Step 4: Define the rule condition + +Define the threshold condition. +Condition options are determined by the [alert type](#step-2-select-the-alert-type). +For this example, the alert conditions are if `usage_idle` is less than `80`. + +![Create a condition](/img/chronograf/1-6-alerts-conditions.png) + +The graph shows a preview of the relevant data and the threshold number. +By default, the graph shows data from the past 15 minutes. +Adjusting the graph's time range is helpful when determining a reasonable threshold number based on your data. + +{{% note %}} +We set the threshold number to `80` for demonstration purposes. +Setting the threshold for idle CPU usage to a high number ensures that we'll be able to see the alert in action. +In practice, you'd set the threshold number to better match the patterns in your data and your alerting needs. +{{% /note %}} + +### Step 5: Select and configure the alert handler + +The **Alert Handler** section determines where the system sends the alert (the event handler) +Chronograf supports several event handlers. +Each handler has unique configurable options. + +For this example, choose the **slack** alert handler and enter the desired options. + +![Select the alert handler](/img/chronograf/1-6-alerts-configure-handlers.png) + +{{% note %}} +Multiple alert handlers can be added to send alerts to multiple endpoints. +{{% /note %}} + +### Step 6: Configure the alert message + +The alert message is the text that accompanies an alert. +Alert messages are templates that have access to alert data. +Available data templates appear below the message text field. +As you type your alert message, clicking the data templates will insert them at end of whatever text has been entered. + +In this example, use the alert message, `Your idle CPU usage is {{.Level}} at {{ index .Fields "value" }}.`. + +![Specify event handler and alert message](/img/chronograf/1-6-alerts-message.png) + +*View the Kapacitor documentation for more information about [message template data](/{{< latest "kapacitor" >}}/nodes/alert_node/#message).* + +### Step 7: Save the alert rule + +Click **Save Rule** in the top right corner and navigate to the **Manage Tasks** page to see your rule. +Notice that you can easily enable and disable the rule by toggling the checkbox in the **Enabled** column. + +![See the alert rule](/img/chronograf/1-6-alerts-view-rules.png) + +Next, move on to the section below to experience your alert rule in action. + +## View alerts in practice + +### Step 1: Create some load on your system + +The purpose of this step is to generate enough load on your system to trigger an alert. +More specifically, your idle CPU usage must dip below `80%`. +On the machine that's running Telegraf, enter the following command in the terminal to start some `while` loops: + +``` +while true; do i=0; done +``` + +Let it run for a few seconds or minutes before terminating it. +On most systems, kill the script by using `Ctrl+C`. + +### Step 2: View the alerts + +Go to the Slack channel that you specified in the previous section. +In this example, it's the `#chronocats` channel. + +Assuming the first step was successful, `#ohnos` should reveal at least two alert messages: + +* The first alert message indicates that your idle CPU usage was `CRITICAL`, meaning it dipped below `80%`. +* The second alert message indicates that your idle CPU usage returned to an `OK` level of `80%` or above. + +![See the alerts](/img/chronograf/1-6-alerts-slack-notifications.png) + +You can also see alerts on the **Alert History** page available under **Alerting** in the left navigation. + +![Chronograf alert history](/img/chronograf/1-6-alerts-history.png) + +That's it! You've successfully used Chronograf to configure an alert rule to monitor your idle CPU usage and send notifications to Slack. diff --git a/content/chronograf/v1.9/guides/dashboard-template-variables.md b/content/chronograf/v1.9/guides/dashboard-template-variables.md new file mode 100644 index 000000000..68eb43b91 --- /dev/null +++ b/content/chronograf/v1.9/guides/dashboard-template-variables.md @@ -0,0 +1,608 @@ +--- +title: Use dashboard template variables +description: > + Chronograf dashboard template variables let you update cell queries without editing queries, + making it easy to interact with your dashboard cells and explore your data. +aliases: + - /chronograf/v1.9/introduction/templating/ + - /chronograf/v1.9/templating/ +menu: + chronograf_1_9: + weight: 90 + parent: Guides +--- + +Chronograf dashboard template variables let you update cell queries without editing queries, +making it easy to interact with your dashboard cells and explore your data. + +- [Use template variables](#use-template-variables) +- [Predefined template variables](#predefined-template-variables) +- [Create custom template variables](#create-custom-template-variables) +- [Template variable types](#template-variable-types) +- [Reserved variable names](#reserved-variable-names) +- [Advanced template variable usage](#advanced-template-variable-usage) + +## Use template variables + +When creating Chronograf dashboards, use either [predefined template variables](#predefined-template-variables) +or [custom template variables](#create-custom-template-variables) in your cell queries and titles. +After you set up variables, variables are available to select in your dashboard user interface (UI). + +- [Use template variables in cell queries](#use-template-variables-in-cell-queries) + - [InfluxQL](#influxql) + - [Flux](#flux) +- [Use template variables in cell titles](#use-template-variables-in-cell-titles) + +![Use template variables](/img/chronograf/1-6-template-vars-use.gif) + +### Use template variables in cell queries +Both InfluxQL and Flux support template variables. + +#### InfluxQL +In an InfluxQL query, surround template variables names with colons (`:`) as follows: + +```sql +SELECT :variable_name: FROM "telegraf"."autogen".:measurement: WHERE time < :dashboardTime: +``` + +##### Quoting template variables in InfluxQL + +For **predefined meta queries** such as "Field Keys" and "Tag Values", **do not add quotes** (single or double) to your queries. Chronograf will add quotes as follows: + +```sql +SELECT :variable_name: FROM "telegraf"."autogen".:measurement: WHERE time < :dashboardTime: +``` + +For **custom queries**, **CSV**, or **map queries**, quote the values in the query following standard [InfluxQL](/{{< latest "influxdb" "v1" >}}/query_language/) syntax: + +- For numerical values, **do not quote**. +- For string values, choose to quote the values in the variable definition (or not). See [String examples](#string-examples) below. + +{{% note %}} +**Tips for quoting strings:** +- When using custom meta queries that return strings, typically, you quote the variable values when using them in a dashboard query, given InfluxQL results are returned without quotes. +- If you are using template variable strings in regular expression syntax (when using quotes may cause query syntax errors), the flexibility in query quoting methods is particularly useful. +{{% /note %}} + +##### String examples + +Add single quotes when you define template variables, or in your queries, but not both. + +###### Add single quotes in variable definition + +If you define a custom CSV variable named `host` using single quotes: + +```sh +'host1','host2','host3' +``` + +Do not include quotes in your query: + +```sql +SELECT mean("usage_user") AS "mean_usage_user" FROM "telegraf"."autogen"."cpu" +WHERE "host" = :host: and time > :dashboardTime +``` + +###### Add single quotes in query + +If you define a custom CSV variable named `host` without quotes: + +```sh +host1,host2,host3 +``` + +Add single quotes in your query: + +```sql +SELECT mean("usage_user") AS "mean_usage_user" FROM "telegraf"."autogen"."cpu" +WHERE "host" = ':host:' and time > :dashboardTime +``` + +#### Flux +In Flux, template variables are stored in a `v` record. +Use dot or bracket notation to reference the variable key inside of the `v` record: + +```js +from(bucket: v.bucket) + |> range(start: v.timeRangeStart, stop: v.timeRangeStart) + |> filter(fn: (r) => r._field == v["Field key"]) + |> aggregateWindow(every: v.windowPeriod, fn: v.aggregateFunction) +``` + +### Use template variables in cell titles +To dynamically change the title of a dashboard cell, +use the `:variable-name:` syntax. + +For example, a variable named `field` with a value of `temp` and a variable +named `location` with a value of `San Antonio`, use the following syntax: + +``` +:temp: data for :location: +``` + +Displays as: + +{{< img-hd src= "/img/chronograf/1-9-template-var-title.png" alt="Use template variables in cell titles" />}} + +## Predefined template variables + +Chronograf includes predefined template variables controlled by elements in the Chronograf UI. +Use predefined template variables in your cell queries. + +InfluxQL and Flux include their own sets of predefined template variables: + +{{< tabs-wrapper >}} +{{% tabs %}} +[InfluxQL](#) +[Flux](#) +{{% /tabs %}} +{{% tab-content %}} +- [`:dashboardTime:`](#dashboardtime) +- [`:upperDashboardTime:`](#upperdashboardtime) +- [`:interval:`](#interval) + +### dashboardTime +The `:dashboardTime:` template variable is controlled by the "time" dropdown in your Chronograf dashboard. + +Dashboard time selector + +If using relative times, it represents the time offset specified in the dropdown (-5m, -15m, -30m, etc.) and assumes time is relative to "now". +If using absolute times defined by the date picker, `:dashboardTime:` is populated with lower threshold. + +```sql +SELECT "usage_system" AS "System CPU Usage" +FROM "telegraf".."cpu" +WHERE time > :dashboardTime: +``` + +{{% note %}} +To use the date picker to specify a past time range, construct the query using `:dashboardTime:` +as the start time and [`:upperDashboardTime:`](#upperdashboardtime) as the stop time. +{{% /note %}} + +### upperDashboardTime +The `:upperDashboardTime:` template variable is defined by the upper time limit specified using the date picker. + +Dashboard date picker + +It will inherit `now()` when using relative time frames or the upper time limit when using absolute timeframes. + +```sql +SELECT "usage_system" AS "System CPU Usage" +FROM "telegraf".."cpu" +WHERE time > :dashboardTime: AND time < :upperDashboardTime: +``` + +### interval + +The `:interval:` template variable is defined by the interval dropdown in the Chronograf dashboard. + +Dashboard interval selector + +In cell queries, it should be used in the `GROUP BY time()` clause that accompanies aggregate functions: + +```sql +SELECT mean("usage_system") AS "Average System CPU Usage" +FROM "telegraf".."cpu" +WHERE time > :dashboardtime: +GROUP BY time(:interval:) +``` +{{% /tab-content %}} +{{% tab-content %}} +- [`v.timeRangeStart`](#vtimerangestart) +- [`v.timeRangeStop`](#vtimerangestop) +- [`v.windowPeriod`](#vwindowperiod) + +{{% note %}} +#### Backward compatible Flux template variables +**Chronograf 1.9+** supports the InfluxDB 2.0 variable pattern of storing +[predefined template variables](#predefined-template-variables) and [custom template variables](#create-custom-template-variables) +in a `v` record and using dot or bracket notation to reference variables. +For backward compatibility, Chronograf 1.9+ still supports the following predefined +variables that do not use the `v.` syntax: + +- [`dashboardTime`](/chronograf/v1.8/guides/dashboard-template-variables/?t=Flux#dashboardtime-flux) +- [`upperDashboardTime`](/chronograf/v1.8/guides/dashboard-template-variables/?t=Flux#upperdashboardtime-flux) +- [`autoInterval`](/chronograf/v1.8/guides/dashboard-template-variables/?t=Flux#autointerval) +{{% /note %}} + +### v.timeRangeStart + +The `v.timeRangeStart` template variable is controlled by the "time" dropdown in your Chronograf dashboard. + +Dashboard time selector + +If using relative time, this variable represents the time offset specified in the dropdown (-5m, -15m, -30m, etc.) and assumes time is relative to "now". +If using absolute time defined by the date picker, `v.timeRangeStart` is populated with the start time. + +```js +from(bucket: "telegraf/autogen") + |> range(start: v.timeRangeStart) + |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") +``` + +{{% note %}} +To use the date picker to specify a time range in the past without "now", use +`v.timeRangeStart` as the start time and [`v.timeRangeStop`](#vtimerangestop) +as the stop time. +{{% /note %}} + +### v.timeRangeStop +The `v.timeRangeStop` template variable is defined by the upper time limit specified using the date picker. + +Dashboard date picker + +For relative time frames, this variable inherits `now()`. For absolute time frames, this variable inherits the upper time limit. + +```js +from(bucket: "telegraf/autogen") + |> range(start: v.timeRangeStart, stop: v.timeRangeStop) + |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") +``` + +### v.windowPeriod +The `v.windowPeriod` template variable is controlled by the display width of the +dashboard cell and is calculated by the duration of time that each pixel covers. +Use the `v.windowPeriod` variable to limit downsample data to display a maximum of one point per pixel. + +```js +from(bucket: "telegraf/autogen") + |> range(start: v.timeRangeStart) + |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") + |> aggregateWindow(every: v.windowPeriod, fn: mean) +``` +{{% /tab-content %}} +{{< /tabs-wrapper >}} + +## Create custom template variables + +Chronograf lets you create custom template variables powered by meta queries or CSV uploads that return an array of possible values. + +To create a template variable: + +1. Click on **Template Variables** at the top of your dashboard, then **+ Add Variable**. +2. Select a data source from the **Data Source** dropdown menu. +3. Provide a name for the variable. +4. Select the [variable type](#template-variable-types). + The type defines the method for retrieving the array of possible values. +5. View the list of potential values and select a default. + If using the CSV or Map types, upload or input the CSV with the desired values in the appropriate format then select a default value. +6. Click **Create**. + +Once created, the template variable can be used in any of your cell's queries or titles +and a dropdown for the variable will be included at the top of your dashboard. + +## Template Variable Types +Chronograf supports the following template variable types: + +- [Databases](#databases) +- [Measurements](#measurements) +- [Field Keys](#field-keys) +- [Tag Keys](#tag-keys) +- [Tag Values](#tag-values) +- [CSV](#csv) +- [Map](#map) +- [InfluxQL Meta Query](#influxql-meta-query) +- [Flux Query](#flux-query) +- [Text](#text) + +### Databases +Database template variables allow you to select from multiple target [databases](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#database). + +_**Database meta query**_ +Database template variables use the following meta query to return an array of all databases in your InfluxDB instance. + +```sql +SHOW DATABASES +``` + +_**Example database variable in a cell query**_ +```sql +SELECT "purchases" FROM :databaseVar:."autogen"."customers" +``` + +#### Database variable use cases +Use database template variables when visualizing multiple databases with similar or identical data structures. +Variables let you quickly switch between visualizations for each of your databases. + +### Measurements +Vary the target [measurement](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#measurement). + +_**Measurement meta query**_ +Measurement template variables use the following meta query to return an array of all measurements in a given database. + +```sql +SHOW MEASUREMENTS ON database_name +``` + +_**Example measurement variable in a cell query**_ +```sql +SELECT * FROM "animals"."autogen".:measurementVar: +``` + +#### Measurement variable use cases +Measurement template variables allow you to quickly switch between measurements in a single cell or multiple cells in your dashboard. + + +### Field Keys +Vary the target [field key](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#field-key). + +_**Field key meta query**_ +Field key template variables use the following meta query to return an array of all field keys in a given measurement from a given database. + +```sql +SHOW FIELD KEYS ON database_name FROM measurement_name +``` + +_**Example field key var in a cell query**_ +```sql +SELECT :fieldKeyVar: FROM "animals"."autogen"."customers" +``` + +#### Field key variable use cases +Field key template variables are great if you want to quickly switch between field key visualizations in a given measurement. + + +### Tag Keys +Vary the target [tag key](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#tag-key). + +_**Tag key meta query**_ +Tag key template variables use the following meta query to return an array of all tag keys in a given measurement from a given database. + +```sql +SHOW TAG KEYS ON database_name FROM measurement_name +``` + +_**Example tag key variable in a cell query**_ +```sql +SELECT "purchases" FROM "animals"."autogen"."customers" GROUP BY :tagKeyVar: +``` + +#### Tag key variable use cases +Tag key template variables are great if you want to quickly switch between tag key visualizations in a given measurement. + + +### Tag Values +Vary the target [tag value](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#tag-value). + +_**Tag value meta query**_ +Tag value template variables use the following meta query to return an array of all values associated with a given tag key in a specified measurement and database. + +```sql +SHOW TAG VALUES ON database_name FROM measurement_name WITH KEY tag_key +``` + +_**Example tag value variable in a cell query**_ +```sql +SELECT "purchases" FROM "animals"."autogen"."customers" WHERE "species" = :tagValueVar: +``` + +#### Tag value variable use cases +Tag value template variables are great if you want to quickly switch between tag value visualizations in a given measurement. + + +### CSV +Vary part of a query with a customized list of comma-separated values (CSV). + +_**Example CSVs:**_ +```csv +value1, value2, value3, value4 +``` +```csv +value1 +value2 +value3 +value4 +``` + +{{% note %}} +String field values [require single quotes in InfluxQL](/{{< latest "influxdb" "v1" >}}/troubleshooting/frequently-asked-questions/#when-should-i-single-quote-and-when-should-i-double-quote-in-queries). + +```csv +'string1','string2','string3','string4' +``` +{{% /note %}} + +_**Example CSV variable in a cell query**_ +```sql +SELECT "purchases" FROM "animals"."autogen"."customers" WHERE "petname" = :csvVar: +``` + +#### CSV variable use cases +CSV template variables are great when the array of values necessary for your variable can't be pulled from InfluxDB using a meta query. +They allow you to use custom variable values. + +### Map +Vary part of a query with a customized list of key-value pairs in CSV format. +They key of each key-value pair is used to populate the template variable dropdown in your dashboard. +The value is used when processing cells' queries. + +_**Example CSV:**_ +```csv +key1,value1 +key2,value2 +key3,value3 +key4,value4 +``` + +Map variable dropdown + +{{% note %}} +Wrap string field values in single quotes ([required by InfluxQL](/{{< latest "influxdb" "v1" >}}/troubleshooting/frequently-asked-questions/#when-should-i-single-quote-and-when-should-i-double-quote-in-queries)). +Variable keys do not require quotes. + +```csv +key1,'value1' +key2,'value2' +key3,'value3' +key4,'value4' +``` +{{% /note %}} + +_**Example Map variable in a cell query**_ +```sql +SELECT "purchases" FROM "animals"."autogen"."customers" WHERE "customer" = :mapVar: +``` + +#### Map variable use cases +Map template variables are good when you need to map or alias simple names or keys to longer or more complex values. +For example, you may want to create a `:customer:` variable that populates your cell queries with a long, numeric customer ID (`11394850823894034209`). +With a map variable, you can alias simple names to complex values, so your list of customers would look something like: + +``` +Apple,11394850823894034209 +Amazon,11394850823894034210 +Google,11394850823894034211 +Microsoft,11394850823894034212 +``` + +The customer names would populate your template variable dropdown rather than the customer IDs. + +### InfluxQL Meta Query +Vary part of a query with a customized meta query that pulls a specific array of values from InfluxDB. +InfluxQL meta query variables let you pull a highly customized array of potential +values and offer advanced functionality such as [filtering values based on other template variables](#filter-template-variables-with-other-template-variables). + +Custom meta query + +_**Example custom meta query variable in a cell query**_ +```sql +SELECT "purchases" FROM "animals"."autogen"."customers" WHERE "customer" = :customMetaVar: +``` + +#### InfluxQL meta query variable use cases +Use custom InfluxQL meta query template variables when predefined template variable types aren't able to return the values you want. + +### Flux Query +Flux query template variables let you define variable values using Flux queries. +**Variable values are extracted from the `_value` column returned by your Flux query.** + +#### Flux query variable use cases +Flux query template variables are great when the values necessary for your +variable can't be queried with InfluxQL or if you need the flexibility of Flux +to return your desired list of variable values. + +### Text +Vary a part of a query with a single string of text. +There is only one value per text variable, but this value is easily altered. + +#### Text variable use cases +Text template variables allow you to dynamically alter queries, such as adding or altering `WHERE` clauses, for multiple cells at once. +You could also use a text template variable to alter a regular expression used in multiple queries. +They are great when troubleshooting incidents that affect multiple visualized metrics. + +## Reserved variable names +The following variable names are reserved and cannot be used when creating template variables. +Chronograf accepts [template variables as URL query parameters](#define-template-variables-in-the-url) +as well as many other parameters that control the display of graphs in your dashboard. +These names are either [predefined variables](#predefined-template-variables) or would +conflict with existing URL query parameters. + +- `:database:` +- `:measurement:` +- `:dashboardTime:` +- `:upperDashboardTime:` +- `:interval:` +- `:upper:` +- `:lower:` +- `:zoomedUpper:` +- `:zoomedLower:` +- `:refreshRate:` + +## Advanced template variable usage + +### Filter template variables with other template variables +[Custom InfluxQL meta query template variables](#influxQL-meta-query) let you filter the array of potential variable values using other existing template variables. + +For example, let's say you want to list all the field keys associated with a measurement, but want to be able to change the measurement: + +1. Create a template variable named `:measurementVar:` _(the name "measurement" is [reserved]( #reserved-variable-names))_ that uses the [Measurements](#measurements) variable type to pull in all measurements from the `telegraf` database. + + measurementVar + +2. Create a template variable named `:fieldKey:` that uses the [InfluxQL meta query](#influxql-meta-query) variable type. +The following meta query pulls a list of field keys based on the existing `:measurementVar:` template variable. + + ```sql + SHOW FIELD KEYS ON telegraf FROM :measurementVar: + ``` + + fieldKey + +3. Create a new dashboard cell that uses the `fieldKey` and `measurementVar` template variables in its query. + + {{< code-tabs-wrapper >}} +{{% code-tabs %}} +[InfluxQL](#) +[Flux](#) +{{% /code-tabs %}} +{{% code-tab-content %}} +```sql +SELECT :fieldKey: FROM "telegraf"..:measurementVar: WHERE time > :dashboardTime: +``` +{{% /code-tab-content %}} +{{% code-tab-content %}} +```js +from(bucket: "telegraf/") + |> range(start: v.timeRangeStart) + |> filter(fn: (r) => + r._measurement == v.measurementVar and + r._field == v.fieldKey + ) +``` +{{% /code-tab-content %}} + {{< /code-tabs-wrapper >}} + +The resulting dashboard will work like this: + +![Custom meta query filtering](/img/chronograf/1-6-custom-meta-query-filtering.gif) + +### Define template variables in the URL +Chronograf uses URL query parameters (also known as query string parameters) to set both display options and template variables in the URL. +This makes it easy to share links to dashboards so they load in a specific state with specific template variable values selected. + +URL query parameters are appended to the end of the URL with a question mark (`?`) +indicating the beginning of query parameters. +Chain multiple query parameters together using an ampersand (`&`). + +To declare a template variable or a date range as a URL query parameter, it must follow the following pattern: + +#### Pattern for template variable query parameters +```bash +# Spaces for clarity only +& tempVars %5B variableName %5D = variableValue +``` + +`&` +Indicates the beginning of a new query parameter in a series of multiple query parameters. + +`tempVars` +Informs Chronograf that the query parameter being passed is a template variable. +_**Required for all template variable query parameters.**_ + +`%5B`, `%5D` +URL-encoded `[` and `]` respectively that enclose the template variable name. + +`variableName` +Name of the template variable. + +`variableValue` +Value of the template variable. + +{{% note %}} +When template variables are modified in the dashboard, the corresponding +URL query parameters are automatically updated. +{{% /note %}} + +#### Example template variable query parameter +``` +.../?&tempVars%5BmeasurementVar%5D=cpu +``` + +#### Including multiple template variables in the URL +To chain multiple template variables as URL query parameters, include the full [pattern](#pattern-for-template-variable-query-parameters) for _**each**_ template variable. + +```bash +# Spaces for clarity only +.../? &tempVars%5BmeasurementVar%5D=cpu &tempVars%5BfieldKey%5D=usage_system +``` diff --git a/content/chronograf/v1.9/guides/live_leaderboard.md b/content/chronograf/v1.9/guides/live_leaderboard.md new file mode 100644 index 000000000..cfa7e02b0 --- /dev/null +++ b/content/chronograf/v1.9/guides/live_leaderboard.md @@ -0,0 +1,293 @@ +--- +title: Create a live leaderboard for game scores +description: This example uses Chronograf to build a leaderboard for gamers to be able to see player scores in realtime. +menu: + chronograf_1_9: + name: Live leaderboard of game scores + weight: 20 + parent: Guides +draft: true +--- + +**If you do not have a running Kapacitor instance, check out [Getting started with Kapacitor](/kapacitor/v1.4/introduction/getting-started/) to get Kapacitor up and running on localhost.** + +Today we are game developers. +We host a several game servers, each running an instance of the game code, with about a hundred players per game. + +We need to build a leaderboard so that spectators can see player scores in realtime. +We would also like to have historical data on leaders in order to do postgame +analysis on who was leading for how long, etc. + +We will use Kapacitor stream processing to do the heavy lifting for us. +The game servers can send a [UDP](https://en.wikipedia.org/wiki/User_Datagram_Protocol) packet whenever a player's score changes, +or every 10 seconds if the score hasn't changed. + +### Setup + +{{% note %}} +**Note:** Copies of the code snippets used here can be found in the [scores](https://github.com/influxdata/kapacitor/tree/master/examples/scores) example in Kapacitor project on GitHub. +{{% /note %}} + +First, we need to configure Kapacitor to receive the stream of scores. +In this example, the scores update too frequently to store all of the score data in a InfluxDB database, so the score data will be semt directly to Kapacitor. +Like InfluxDB, you can configure a UDP listener. + +Add the following settings the `[[udp]]` secton in your Kapacitor configuration file (`kapacitor.conf`). + +``` +[[udp]] + enabled = true + bind-address = ":9100" + database = "game" + retention-policy = "autogen" +``` + +Using this configuration, Kapacitor will listen on port `9100` for UDP packets in [Line Protocol](/{{< latest "influxdb" "v1" >}}/write_protocols/line_protocol_tutorial/) format. +Incoming data will be scoped to be in the `game.autogen` database and retention policy. +Restart Kapacitor so that the UDP listener service starts. + +Here is a simple bash script to generate random score data so we can test it without +messing with the real game servers. + +```bash +#!/bin/bash + +# default options: can be overriden with corresponding arguments. +host=${1-localhost} +port=${2-9100} +games=${3-10} +players=${4-100} + +games=$(seq $games) +players=$(seq $players) +# Spam score updates over UDP +while true +do + for game in $games + do + game="g$game" + for player in $players + do + player="p$player" + score=$(($RANDOM % 1000)) + echo "scores,player=$player,game=$game value=$score" > /dev/udp/$host/$port + done + done + sleep 0.1 +done +``` + +Place the above script into a file `scores.sh` and run it: + +```bash +chmod +x ./scores.sh +./scores.sh +``` + +Now we are spamming Kapacitor with our fake score data. +We can just leave that running since Kapacitor will drop +the incoming data until it has a task that wants it. + +### Defining the Kapacitor task + +What does a leaderboard need to do? + +1. Get the most recent score per player per game. +1. Calculate the top X player scores per game. +1. Publish the results. +1. Store the results. + +To complete step one we need to buffer the incoming stream and return the most recent score update per player per game. +Our [TICKscript](/kapacitor/v1.4/tick/) will look like this: + +```javascript +var topPlayerScores = stream + |from() + .measurement('scores') + // Get the most recent score for each player per game. + // Not likely that a player is playing two games but just in case. + .groupBy('game', 'player') + |window() + // keep a buffer of the last 11s of scores + // just in case a player score hasn't updated in a while + .period(11s) + // Emit the current score per player every second. + .every(1s) + // Align the window boundaries to be on the second. + .align() + |last('value') +``` + +Place this script in a file called `top_scores.tick`. + +Now our `topPlayerScores` variable contains each player's most recent score. +Next to calculate the top scores per game we just need to group by game and run another map reduce job. +Let's keep the top 15 scores per game. +Add these lines to the `top_scores.tick` file. + +```javascript +// Calculate the top 15 scores per game +var topScores = topPlayerScores + |groupBy('game') + |top(15, 'last', 'player') +``` + +The `topScores` variable now contains the top 15 player's score per game. +All we need to be able to build our leaderboard. +Kapacitor can expose the scores over HTTP via the [HTTPOutNode](/kapacitor/v1.4/nodes/http_out_node/). +We will call our task `top_scores`; with the following addition the most recent scores will be available at +`http://localhost:9092/kapacitor/v1/tasks/top_scores/top_scores`. + +```javascript +// Expose top scores over the HTTP API at the 'top_scores' endpoint. +// Now your app can just request the top scores from Kapacitor +// and always get the most recent result. +// +// http://localhost:9092/kapacitor/v1/tasks/top_scores/top_scores +topScores + |httpOut('top_scores') +``` + +Finally we want to store the top scores over time so we can do in depth analysis to ensure the best game play. +But we do not want to store the scores every second as that is still too much data. +First we will sample the data and store scores only every 10 seconds. +Also let's do some basic analysis ahead of time since we already have a stream of all the data. +For now we will just do basic gap analysis where we will store the gap between the top player and the 15th player. +Add these lines to `top_scores.tick` to complete our task. + +```javascript +// Sample the top scores and keep a score once every 10s +var topScoresSampled = topScores + |sample(10s) + +// Store top fifteen player scores in InfluxDB. +topScoresSampled + |influxDBOut() + .database('game') + .measurement('top_scores') + +// Calculate the max and min of the top scores. +var max = topScoresSampled + |max('top') + +var min = topScoresSampled + |min('top') + +// Join the max and min streams back together and calculate the gap. +max + |join(min) + .as('max', 'min') + // Calculate the difference between the max and min scores. + // Rename the max and min fields to more friendly names 'topFirst', 'topLast'. + |eval(lambda: "max.max" - "min.min", lambda: "max.max", lambda: "min.min") + .as('gap', 'topFirst', 'topLast') + // Store the fields: gap, topFirst and topLast in InfluxDB. + |influxDBOut() + .database('game') + .measurement('top_scores_gap') +``` + +Since we are writing data back to InfluxDB create a database `game` for our results. + +{{< keep-url >}} +``` +curl -G 'http://localhost:8086/query?' --data-urlencode 'q=CREATE DATABASE game' +``` + +Here is the complete task TICKscript if you don't want to copy paste as much :) + +```javascript +dbrp "game"."autogen" + +// Define a result that contains the most recent score per player. +var topPlayerScores = stream + |from() + .measurement('scores') + // Get the most recent score for each player per game. + // Not likely that a player is playing two games but just in case. + .groupBy('game', 'player') + |window() + // keep a buffer of the last 11s of scores + // just in case a player score hasn't updated in a while + .period(11s) + // Emit the current score per player every second. + .every(1s) + // Align the window boundaries to be on the second. + .align() + |last('value') + +// Calculate the top 15 scores per game +var topScores = topPlayerScores + |groupBy('game') + |top(15, 'last', 'player') + +// Expose top scores over the HTTP API at the 'top_scores' endpoint. +// Now your app can just request the top scores from Kapacitor +// and always get the most recent result. +// +// http://localhost:9092/kapacitor/v1/tasks/top_scores/top_scores +topScores + |httpOut('top_scores') + +// Sample the top scores and keep a score once every 10s +var topScoresSampled = topScores + |sample(10s) + +// Store top fifteen player scores in InfluxDB. +topScoresSampled + |influxDBOut() + .database('game') + .measurement('top_scores') + +// Calculate the max and min of the top scores. +var max = topScoresSampled + |max('top') + +var min = topScoresSampled + |min('top') + +// Join the max and min streams back together and calculate the gap. +max + |join(min) + .as('max', 'min') + // calculate the difference between the max and min scores. + |eval(lambda: "max.max" - "min.min", lambda: "max.max", lambda: "min.min") + .as('gap', 'topFirst', 'topLast') + // store the fields: gap, topFirst, and topLast in InfluxDB. + |influxDBOut() + .database('game') + .measurement('top_scores_gap') +``` + +Define and enable our task to see it in action: + +```bash +kapacitor define top_scores -tick top_scores.tick +kapacitor enable top_scores +``` + +First let's check that the HTTP output is working. + +```bash +curl 'http://localhost:9092/kapacitor/v1/tasks/top_scores/top_scores' +``` + +You should have a JSON result of the top 15 players and their scores per game. +Hit the endpoint several times to see that the scores are updating once a second. + +Now, let's check InfluxDB to see our historical data. + +{{< keep-url >}} +```bash +curl \ + -G 'http://localhost:8086/query?db=game' \ + --data-urlencode 'q=SELECT * FROM top_scores WHERE time > now() - 5m GROUP BY game' + +curl \ + -G 'http://localhost:8086/query?db=game' \ + --data-urlencode 'q=SELECT * FROM top_scores_gap WHERE time > now() - 5m GROUP BY game' +``` + +Great! +The hard work is done. +All that remains is configuring the game server to send score updates to Kapacitor and update the spectator dashboard to pull scores from Kapacitor. diff --git a/content/chronograf/v1.9/guides/monitoring-influxenterprise-clusters.md b/content/chronograf/v1.9/guides/monitoring-influxenterprise-clusters.md new file mode 100644 index 000000000..1915d880a --- /dev/null +++ b/content/chronograf/v1.9/guides/monitoring-influxenterprise-clusters.md @@ -0,0 +1,291 @@ +--- +title: Monitor InfluxDB Enterprise clusters +description: Use Chronograf dashboards with an InfluxDB OSS server to measure and monitor InfluxDB Enterprise clusters. +aliases: + - /chronograf/v1.9/guides/monitor-an-influxenterprise-cluster/ +menu: + chronograf_1_9: + weight: 80 + parent: Guides + +--- + +[InfluxDB Enterprise](/{{< latest "enterprise_influxdb" >}}/) offers high availability and a highly scalable clustering solution for your time series data needs. +Use Chronograf to assess your cluster's health and to monitor the infrastructure behind your project. + +This guide offers step-by-step instructions for using Chronograf, [InfluxDB](/{{< latest "influxdb" "v1" >}}/), and [Telegraf](/{{< latest "telegraf" >}}/) to monitor data nodes in your InfluxDB Enterprise cluster. + +## Requirements + +You have a fully-functioning InfluxDB Enterprise cluster with authentication enabled. +See the InfluxDB Enterprise documentation for +[detailed setup instructions](/{{< latest "enterprise_influxdb" >}}/production_installation/). +This guide uses an InfluxData Enterprise cluster with three meta nodes and three data nodes; the steps are also applicable to other cluster configurations. + +InfluxData recommends using a separate server to store your monitoring data. +It is possible to store the monitoring data in your cluster and [connect the cluster to Chronograf](/chronograf/v1.9/troubleshooting/frequently-asked-questions/#how-do-i-connect-chronograf-to-an-influxenterprise-cluster), but, in general, your monitoring data should live on a separate server. + +You're working on an Ubuntu installation. +Chronograf and the other components of the TICK stack are supported on several operating systems and hardware architectures. Check out the [downloads page](https://portal.influxdata.com/downloads) for links to the binaries of your choice. + +## Architecture overview + +Before we begin, here's an overview of the final monitoring setup: + +![Architecture diagram](/img/chronograf/1-6-cluster-diagram.png) + +The diagram above shows an InfluxDB Enterprise cluster that consists of three meta nodes (M) and three data nodes (D). +Each data node has its own [Telegraf](/{{< latest "telegraf" >}}/) instance (T). + +Each Telegraf instance is configured to collect node CPU, disk, and memory data using the Telegraf [system stats](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/system) input plugin. +The Telegraf instances are also configured to send those data to a single [InfluxDB OSS](/{{< latest "influxdb" "v1" >}}/) instance that lives on a separate server. +When Telegraf sends data to InfluxDB, it automatically [tags](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#tag) the data with the hostname of the relevant data node. + +The InfluxDB OSS instance that stores the Telegraf data is connected to Chronograf. +Chronograf uses the hostnames in the Telegraf data to populate the Host List page and provide other hostname-specific information in the user interface. + +## Setup description + +### InfluxDB OSS setup + +#### Step 1: Download and install InfluxDB + +InfluxDB can be downloaded from the [InfluxData downloads page](https://portal.influxdata.com/downloads). + +#### Step 2: Enable authentication + +For security purposes, enable authentication in the InfluxDB [configuration file (influxdb.conf)](/{{< latest "influxdb" "v1" >}}/administration/config/), which is located in `/etc/influxdb/influxdb.conf`. + +In the `[http]` section of the configuration file, uncomment the `auth-enabled` option and set it to `true`: + +``` +[http] + # Determines whether HTTP endpoint is enabled. + # enabled = true + + # The bind address used by the HTTP service. + # bind-address = ":8086" + + # Determines whether HTTP authentication is enabled. + auth-enabled = true #💥 +``` + +#### Step 3: Start InfluxDB + +Next, start the InfluxDB process: + +``` +~# sudo systemctl start influxdb +``` + +#### Step 4: Create an admin user + +Create an [admin user](/{{< latest "influxdb" "v1" >}}/administration/authentication_and_authorization/#user-types-and-privileges) on your InfluxDB instance. +Because you enabled authentication, you must perform this step before moving on to the next section. +Run the command below to create an admin user, replacing `chronothan` and `supersecret` with your own username and password. +Note that the password requires single quotes. + +{{< keep-url >}} +``` +~# curl -XPOST "http://localhost:8086/query" --data-urlencode "q=CREATE USER chronothan WITH PASSWORD 'supersecret' WITH ALL PRIVILEGES" +``` + +A successful `CREATE USER` query returns a blank result: +``` +{"results":[{"statement_id":0}]} <--- Success! +``` + +### Telegraf setup + +Perform the following steps on each data node in your cluster. +You'll return to your InfluxDB instance at the end of this section. + +#### Step 1: Download and install Telegraf + +Telegraf can be downloaded from the [InfluxData downloads page](https://portal.influxdata.com/downloads). + +#### Step 2: Configure Telegraf + +Configure Telegraf to write monitoring data to your InfluxDB OSS instance. +The Telegraf configuration file is located in `/etc/telegraf/telegraf.conf`. + +First, in the `[[outputs.influxdb]]` section, set the `urls` option to the IP address and port of your InfluxDB OSS instance. +InfluxDB runs on port `8086` by default. +This step ensures that Telegraf writes data to your InfluxDB OSS instance. + +``` +[[outputs.influxdb]] + ## The full HTTP or UDP endpoint URL for your InfluxDB instance. + ## Multiple urls can be specified as part of the same cluster, + ## this means that only ONE of the urls will be written to each interval. + # urls = ["udp://localhost:8089"] # UDP endpoint example + urls = ["http://xxx.xx.xxx.xxx:8086"] #💥 +``` + +Next, in the same `[[outputs.influxdb]]` section, uncomment and set the `username` and `password` options to the username and password that you created in the [previous section](#step-4-create-an-admin-user). +Telegraf must be aware your username and password to successfully write data to your InfluxDB OSS instance. + +``` +[[outputs.influxdb]] + ## The full HTTP or UDP endpoint URL for your InfluxDB instance. + ## Multiple urls can be specified as part of the same cluster, + ## this means that only ONE of the urls will be written to each interval. + # urls = ["udp://localhost:8089"] # UDP endpoint example + urls = ["http://xxx.xx.xxx.xxx:8086"] # required + + [...] + + ## Write timeout (for the InfluxDB client), formatted as a string. + ## If not provided, will default to 5s. 0s means no timeout (not recommended). + timeout = "5s" + username = "chronothan" #💥 + password = "supersecret" #💥 +``` + +The [Telegraf System input plugin](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/system) is enabled by default and requires no additional configuration. +The input plugin automatically collects general statistics on system load, uptime, and the number of users logged in. +Enabled input plugins are configured in the `INPUT PLUGINS` section of the configuration file; for example, here's the section that controls the CPU data collection: + +``` +############################################################################### +# INPUT PLUGINS # +############################################################################### + +# Read metrics about cpu usage +[[inputs.cpu]] + ## Whether to report per-cpu stats or not + percpu = true + ## Whether to report total system cpu stats or not + totalcpu = true + ## If true, collect raw CPU time metrics. + collect_cpu_time = false +``` + +#### Step 3: Restart the Telegraf service + +Restart the Telegraf service so that your configuration changes take effect: + +**macOS** + +```sh +telegraf --config telegraf.conf +``` + +**Linux (sysvinit and upstart installations)** + +```sh +sudo service telegraf restart +``` + +**Linux (systemd installations)** + +```sh +systemctl restart telegraf +``` + +Repeat steps one through four for each data node in your cluster. + +#### Step 4: Confirm the Telegraf setup + +To verify Telegraf is successfully collecting and writing data, use one of the following methods to query your InfluxDB OSS instance: + +**InfluxDB CLI (`influx`)** + +```sh +$ influx +> SHOW TAG VALUES FROM cpu WITH KEY=host +``` + +**`curl`** + +Replace the `chronothan` and `supersecret` values with your actual username and password. + +{{< keep-url >}} +``` +~# curl -G "http://localhost:8086/query?db=telegraf&u=chronothan&p=supersecret&pretty=true" --data-urlencode "q=SHOW TAG VALUES FROM cpu WITH KEY=host" +``` + +The expected output is similar to the JSON code block below. +In this case, the `telegraf` database has three different [tag values](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#tag-value) for the `host` [tag key](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#tag-key): `data-node-01`, `data-node-02`, and `data-node-03`. +Those values match the hostnames of the three data nodes in the cluster; this means Telegraf is successfully writing monitoring data from those hosts to the InfluxDB OSS instance! +``` +{ + "results": [ + { + "statement_id": 0, + "series": [ + { + "name": "cpu", + "columns": [ + "key", + "value" + ], + "values": [ + [ + "host", + "data-node-01" + ], + [ + "host", + "data-node-02" + ], + [ + "host", + "data-node-03" + ] + ] + } + ] + } + ] +} +``` + +### Chronograf Setup + +#### Step 1: Download and install Chronograf + +Download and install Chronograf on the same server as the InfluxDB instance. +This is not a requirement; you may host Chronograf on a separate server. + +Chronograf can be downloaded from the [InfluxData downloads page](https://portal.influxdata.com/downloads). + +#### Step 2: Start Chronograf + +``` +~# sudo systemctl start chronograf +``` + +### Step 3: Connect Chronograf to the InfluxDB OSS instance + +To access Chronograf, go to http://localhost:8888. +The welcome page includes instructions for connecting Chronograf to that instance. + +![Connect Chronograf to InfluxDB](/img/chronograf/1-6-cluster-welcome.png) + +For the `Connection String`, enter the hostname or IP of your InfluxDB OSS instance, and be sure to include the default port: `8086`. +Next, name your data source; this can be anything you want. +Finally, enter your username and password and click `Add Source`. + +### Step 4: Explore the monitoring data in Chronograf + +Chronograf works with the Telegraf data in your InfluxDB OSS instance. +The `Host List` page shows your data node's hostnames, their statuses, CPU usage, load, and their configured applications. +In this case, you've only enabled the system stats input plugin so `system` is the single application that appears in the `Apps` column. + +![Host List page](/img/chronograf/1-6-cluster-hostlist.png) + +Click `system` to see the Chronograf canned dashboard for that application. +Keep an eye on your data nodes by viewing that dashboard for each hostname: + +![Pre-created dashboard](/img/chronograf/1-6-cluster-predash.gif) + +Next, check out the Data Explorer to create a customized graph with the monitoring data. +In the image below, the Chronograf query editor is used to visualize the idle CPU usage data for each data node: + +![Data Explorer](/img/chronograf/1-6-cluster-de.png) + +Create more customized graphs and save them to a dashboard on the Dashboard page in Chronograf. +See the [Creating Chronograf dashboards](/chronograf/v1.9/guides/create-a-dashboard/) guide for more information. + +That's it! You've successfully configured Telegraf to collect and write data, InfluxDB to store those data, and Chronograf to use those data for monitoring and visualization purposes. diff --git a/content/chronograf/v1.9/guides/presentation-mode.md b/content/chronograf/v1.9/guides/presentation-mode.md new file mode 100644 index 000000000..a4034e31d --- /dev/null +++ b/content/chronograf/v1.9/guides/presentation-mode.md @@ -0,0 +1,25 @@ +--- +title: View Chronograf dashboards in presentation mode +description: View dashboards in full screen using presentation mode. +menu: + chronograf_1_9: + name: View dashboards in presentation mode + weight: 130 + parent: Guides +--- + +Presentation mode allows you to view Chronograf in full screen, hiding the left and top navigation menus so only the cells appear. This mode might be helpful, for example, for stationary screens dedicated to monitoring visualizations. + +## Enter presentation mode manually +To enter presentation mode manually, click the icon in the upper right: + + + +To exit presentation mode, press `ESC`. + +## Use the URL query parameter +To load the dashboard in presentation mode, add URL query parameter `present=true` to your dashboard URL. For example, your URL might look like this: + +`http://example.com:8888/sources/1/dashboards/2?present=true` + +Note that if you use this option, you won't be able to exit presentation mode using `ESC`. diff --git a/content/chronograf/v1.9/guides/querying-data.md b/content/chronograf/v1.9/guides/querying-data.md new file mode 100644 index 000000000..8e82c8bad --- /dev/null +++ b/content/chronograf/v1.9/guides/querying-data.md @@ -0,0 +1,148 @@ +--- +title: Explore data in Chronograf +description: Query and visualize data in the Data Explorer. +menu: + chronograf_1_9: + name: Explore data in Chronograf + weight: 130 + parent: Guides +--- + +Explore and visualize your data in the **Data Explorer**. For both InfluxQL and Flux, Chronograf allows you to move seamlessly between using the builder or templates and manually editing the query; when possible, the interface automatically populates the builder with the information from your raw query. Choose between [visualization types](/chronograf/v1.9/guides/visualization-types/) for your query. + +To open the **Data Explorer**, click the **Explore** icon in the navigation bar: + + + +## Select local time or UTC (Coordinated Universal Time) + +- In the upper-right corner of the page, select the time to view metrics and events by clicking one of the following: + - **UTC** for Coordinated Universal Time + - **Local** for the local time reported by your browser + +{{% note %}} +**Note:** If your organization spans multiple time zones, we recommend using UTC (Coordinated Universal Time) to ensure that everyone sees metrics and events for the same time. +{{% /note %}} + +## Explore data with InfluxQL + +InfluxQL is a SQL-like query language you can use to interact with data in InfluxDB. For detailed tutorials and reference material, see our [InfluxQL documentation](/{{< latest "influxdb" "v1" >}}/query_language/). + +{{% note %}} +#### Limited InfluxQL support in InfluxDB Cloud and OSS 2.x +Chronograf interacts with **InfluxDB Cloud** and **InfluxDB OSS 2.x** through the +[v1 compatibility API](/influxdb/cloud/reference/api/influxdb-1x/). +The v1 compatibility API provides limited InfluxQL support. +For more information, see [InfluxQL support](/influxdb/cloud/query-data/influxql/#influxql-support). +{{% /note %}} + +1. Open the Data Explorer and click **Add a Query**. +2. To the right of the source dropdown above the graph placeholder, select **InfluxQL** as the source type. +3. Use the builder to select from your existing data and allow Chronograf to format the query for you. Alternatively, manually enter and edit a query. +4. You can also select from the dropdown list of **Metaquery Templates** that manage databases, retention policies, users, and more. + _See [Metaquery templates](#metaquery-templates)._ +5. To display the templated values in the query, click **Show template values**. +6. Click **Submit Query**. + +## Explore data with Flux + +Flux is InfluxData's new functional data scripting language designed for querying, analyzing, and acting on time series data. To learn more about Flux, see [Getting started with Flux](/{{< latest "influxdb" "v2" >}}/query-data/get-started). + +1. Open the Data Explorer and click **Add a Query**. +2. To the right of the source dropdown above the graph placeholder, select **Flux** as the source type. + The **Schema**, **Functions**, and **Script** panes appear. +3. Use the **Schema** pane to explore your available data. Click the **+** sign next to a bucket name to expand its content. +4. Use the **Functions** pane to view details about the available Flux functions. +5. Use the **Script** pane to enter your Flux query. + + * To get started with your query, click the **Script Wizard**. In the wizard, you can select a bucket, measurement, fields and an aggregate. + + + + For example, if you make the above selections, the wizard inserts the following script: + + ```js + from(bucket: "telegraf/autogen") + |> range(start: dashboardTime) + |> filter(fn: (r) => r._measurement == "cpu" and (r._field == "usage_system")) + |> window(every: autoInterval) + |> toFloat() + |> percentile(percentile: 0.95) + |> group(except: ["_time", "_start", "_stop", "_value"]) + ``` + * Alternatively, you can enter your entire script manually. + +6. Click **Run Script** in the top bar of the **Script** pane. You can then preview your graph in the above pane. + +## Visualize your query + +Select the **Visualization** tab at the top of the **Data Explorer**. For details about all of the available visualization options, see [Visualization types in Chronograf](/chronograf/v1.9/guides/visualization-types/). + +## Add queries to dashboards + +To add your query and graph to a dashboard: + +1. Click **Send to Dashboard** in the upper right. +2. In the **Target Dashboard(s)** dropdown, select at least one existing dashboard to send the cell to, or select **Send to a New Dashboard**. +3. Enter a name for the new cell and, if you created a new dashboard, the new dashboard. +4. If using an **InfluxQL** data source and you have multiple queries in the Data Explorer, +select which queries to send: + - **Active Query**: Send the currently viewed query. + - **All Queries**: Send all queries. +5. Click **Send to Dashboard(s)**. + +## Metaquery templates +Metaquery templates provide templated InfluxQL queries manage databases, retention policies, users, and more. +Choose from the following options in the **Metaquery Templates** dropdown list: + +###### Manage databases +- [Show Databases](/{{< latest "influxdb" "v1" >}}/query_language/explore-schema/#show-databases) +- [Create Database](/{{< latest "influxdb" "v1" >}}/query_language/manage-database/#create-database) +- [Drop Database](/{{< latest "influxdb" "v1" >}}/query_language/manage-database/#delete-a-database-with-drop-database) + +###### Measurements, Tags, and Fields +- [Show Measurements](/{{< latest "influxdb" "v1" >}}/query_language/explore-schema/#show-measurements) +- [Show Tag Keys](/{{< latest "influxdb" "v1" >}}/query_language/explore-schema/#show-tag-keys) +- [Show Tag Values](/{{< latest "influxdb" "v1" >}}/query_language/explore-schema/#show-tag-values) +- [Show Field Keys](/{{< latest "influxdb" "v1" >}}/query_language/explore-schema/#show-field-keys) + +###### Cardinality +- [Show Field Key Cardinality](/{{< latest "influxdb" "v1" >}}/query_language/spec/#show-field-key-cardinality) +- [Show Measurement Cardinality](/{{< latest "influxdb" "v1" >}}/query_language/spec/#show-measurement-cardinality) +- [Show Series Cardinality](/{{< latest "influxdb" "v1" >}}/query_language/spec/#show-series-cardinality) +- [Show Tag Key Cardinality](/{{< latest "influxdb" "v1" >}}/query_language/spec/#show-tag-key-cardinality) +- [Show Tag Values Cardinality](/{{< latest "influxdb" "v1" >}}/query_language/spec/#show-tag-values-cardinality) + +###### Manage retention policies +- [Show Retention Polices](/{{< latest "influxdb" "v1" >}}/query_language/explore-schema/#show-retention-policies) +- [Create Retention Policy](/{{< latest "influxdb" "v1" >}}/query_language/manage-database/#create-retention-policies-with-create-retention-policy) +- [Drop Retention Policy](/{{< latest "influxdb" "v1" >}}/query_language/manage-database/#delete-retention-policies-with-drop-retention-policy) + +###### Manage continuous queries +- [Show Continuous Queries](/{{< latest "influxdb" "v1" >}}/query_language/continuous_queries/#listing-continuous-queries) +- [Create Continuous Query](/{{< latest "influxdb" "v1" >}}/query_language/continuous_queries/#syntax) +- [Drop Continuous Query](/{{< latest "influxdb" "v1" >}}/query_language/continuous_queries/#deleting-continuous-queries) + +###### Manage users and permissions +- [Show Users](/{{< latest "influxdb" "v1" >}}/query_language/spec/#show-users) +- [Show Grants](/{{< latest "influxdb" "v1" >}}/query_language/spec/#show-grants) +- [Create User](/{{< latest "influxdb" "v1" >}}/query_language/spec/#create-user) +- [Create Admin User](/{{< latest "influxdb" "v1" >}}/administration/authentication_and_authorization/#admin-user-management) +- [Drop User](/{{< latest "influxdb" "v1" >}}/query_language/spec/#drop-user) + +###### Delete data +- [Drop Measurement](/{{< latest "influxdb" "v1" >}}/query_language/manage-database/#delete-measurements-with-drop-measurement) +- [Drop Series](/{{< latest "influxdb" "v1" >}}/query_language/manage-database/#drop-series-from-the-index-with-drop-series) +- [Delete](/{{< latest "influxdb" "v1" >}}/query_language/manage-database/#delete-series-with-delete) + +###### Analyze queries +- [Explain](/{{< latest "influxdb" "v1" >}}/query_language/spec/#explain) +- [Explain Analyze](/{{< latest "influxdb" "v1" >}}/query_language/spec/#explain-analyze) + +###### Inspect InfluxDB internal metrics +- [Show Stats](/{{< latest "influxdb" "v1" >}}/query_language/spec/#show-stats) +- [Show Diagnostics](/{{< latest "influxdb" "v1" >}}/query_language/spec/#show-diagnostics) +- [Show Subscriptions](/{{< latest "influxdb" "v1" >}}/query_language/spec/#show-subscriptions) +- [Show Queries](/{{< latest "influxdb" "v1" >}}/troubleshooting/query_management/#list-currently-running-queries-with-show-queries) +- [Show Shards](/{{< latest "influxdb" "v1" >}}/query_language/spec/#show-shards) +- [Show Shard Groups](/{{< latest "influxdb" "v1" >}}/query_language/spec/#show-shard-groups) diff --git a/content/chronograf/v1.9/guides/tickscript-logging.md b/content/chronograf/v1.9/guides/tickscript-logging.md new file mode 100644 index 000000000..e9c322199 --- /dev/null +++ b/content/chronograf/v1.9/guides/tickscript-logging.md @@ -0,0 +1,15 @@ +--- +title: Edit TICKscripts in Chronograf +description: View and edit TICKscript logs in Chronograf. +menu: + chronograf_1_9: + weight: 20 + parent: Guides +draft: true +--- + +TICKscript logs data to a log file for debugging purposes. + +Notes: +* TICKscript logs data to a log file for debugging purposes. We have a bunch of hosts which post data to an external endpoint. the payload is logged before being sent. +A feature to show the list of hosts , and an ability to see the logs for each of them. diff --git a/content/chronograf/v1.9/guides/using-precreated-dashboards.md b/content/chronograf/v1.9/guides/using-precreated-dashboards.md new file mode 100644 index 000000000..4826a4a88 --- /dev/null +++ b/content/chronograf/v1.9/guides/using-precreated-dashboards.md @@ -0,0 +1,482 @@ +--- +title: Use pre-created dashboards in Chronograf +description: > + Display metrics for popular third-party applications with preconfigured dashboards in Chronograf. +menu: + chronograf_1_9: + name: Use pre-created dashboards + weight: 10 + parent: Guides +--- + + +## Overview + +Pre-created dashboards are delivered with Chronograf depending on which Telegraf input plugins you have enabled and are available from the Host List page. These dashboards, which are built in and not editable, include cells with data visualizations for metrics that are relevant to data sources you are likely to be using. + +{{% note %}} +Note that these pre-created dashboards cannot be cloned or customized. They appear only as part of the Host List view and are associated with metrics gathered from a single host. Dashboard templates are also available and deliver a solid starting point for customizing your own unique dashboards based on the Telegraf plugins enabled and operate across one or more hosts. For details, see [Dashboard templates](/chronograf/v1.9/guides/create-a-dashboard/#dashboard-templates). +{{% /note %}} + +## Requirements + +The pre-created dashboards automatically appear in the Host List page to the right of hosts based on which Telegraf input plugins you have enabled. Check the list below for applications that you are interested in using and make sure that you have the required Telegraf input plugins enabled. + +## Use pre-created dashboards + +Pre-created dashboards are delivered in Chronograf installations and are ready to be used when you have the required Telegraf input plugins enabled. + +**To view a pre-created dashboard:** + +1. Open Chronograf in your web browser and click **Host List** in the navigation bar. +2. Select an application listed under **Apps**. By default, the system `app` should be listed next to a host listing. Other apps appear depending on the Telegraf input plugins that you have enabled. + The selected application appears showing pre-created cells, based on available measurements. + +## Create or edit dashboards + +Find a list of apps (pre-created dashboards) available to use with Chronograf below. For each app, you'll find: + +- Required Telegraf input plugins for the app +- JSON files included in the app +- Cell titles included in each JSON file + +The JSON files for apps are included in the `/usr/share/chronograf/canned` directory. Find information about the configuration option `--canned-path` on the [Chronograf configuration options](/chronograf/v1.9/administration/config-options/) page. + +Enable and disable apps in your Telegraf configuration file (by default, `/etc/telegraf/telegraf.conf`). See [Configuring Telegraf](/telegraf/v1.13/administration/configuration/) for details. + +## Apps (pre-created dashboards): + +* [apache](#apache) +* [consul](#consul) +* [docker](#docker) +* [elasticsearch](#elasticsearch) +* [haproxy](#haproxy) +* [iis](#iis) +* [influxdb](#influxdb) +* [kubernetes](#kubernetes) +* [memcached](#memcached-memcached) +* [mesos](#mesos) +* [mysql](#mysql) +* [nginx](#nginx) +* [nsq](#nsq) +* [phpfpm](#phpfpm) +* [ping](#ping) +* [postgresql](#postgresql) +* [rabbitmq](#rabbitmq) +* [redis](#redis) +* [riak](#riak) +* [system](#system) +* [varnish](#varnish) +* [win_system](#win-system) + +## apache + +**Required Telegraf plugin:** [Apache input plugin](/{{< latest "telegraf" >}}/plugins/#apache-http-server) + +`apache.json` + +* "Apache Bytes/Second" +* "Apache - Requests/Second" +* "Apache - Total Accesses" + +## consul + +**Required Telegraf plugin:** [Consul input plugin](/{{< latest "telegraf" >}}/plugins/#consul) + +`consul_http.json` + +* "Consul - HTTP Request Time (ms)" + +`consul_election.json` + +* "Consul - Leadership Election" + +`consul_cluster.json` + +* "Consul - Number of Agents" + +`consul_serf_events.json` + +* "Consul - Number of serf events" + +## docker + +**Required Telegraf plugin:** [Docker input plugin](/{{< latest "telegraf" >}}/plugins/#docker) + +`docker.json` + +* "Docker - Container CPU %" +* "Docker - Container Memory (MB)" +* "Docker - Containers" +* "Docker - Images" +* "Docker - Container State" + +`docker_blkio.json` + +* "Docker - Container Block IO" + +`docker_net.json` + +* "Docker - Container Network" + +## elasticsearch + +**Required Telegraf plugin:** [Elasticsearch input plugin](/{{< latest "telegraf" >}}/plugins/#elasticsearch) + +`elasticsearch.json` + +* "ElasticSearch - Query Throughput" +* "ElasticSearch - Open Connections" +* "ElasticSearch - Query Latency" +* "ElasticSearch - Fetch Latency" +* "ElasticSearch - Suggest Latency" +* "ElasticSearch - Scroll Latency" +* "ElasticSearch - Indexing Latency" +* "ElasticSearch - JVM GC Collection Counts" +* "ElasticSearch - JVM GC Latency" +* "ElasticSearch - JVM Heap Usage" + +## haproxy + +**Required Telegraf plugin:** [HAProxy input plugin](/{{< latest "telegraf" >}}/plugins/#haproxy) + +`haproxy.json` + + * "HAProxy - Number of Servers" + * "HAProxy - Sum HTTP 2xx" + * "HAProxy - Sum HTTP 4xx" + * "HAProxy - Sum HTTP 5xx" + * "HAProxy - Frontend HTTP Requests/Second" + * "HAProxy - Frontend Sessions/Second" + * "HAProxy - Frontend Session Usage %" + * "HAProxy - Frontend Security Denials/Second" + * "HAProxy - Frontend Request Errors/Second" + * "HAProxy - Frontend Bytes/Second" + * "HAProxy - Backend Average Response Time (ms)" + * "HAProxy - Backend Connection Errors/Second" + * "HAProxy - Backend Queued Requests/Second" + * "HAProxy - Backend Average Request Queue Time (ms)" + * "HAProxy - Backend Error Responses/Second" + +## iis + +**Required Telegraf plugin:** [Windows Performance Counters input plugin](/{{< latest "telegraf" >}}/plugins/#windows-performance-counters) + +`win_websvc.json` + + * "IIS - Service" + +## influxdb + +**Required Telegraf plugin:** [InfluxDB input plugin](/{{< latest "telegraf" >}}/plugins/#influxdb) + +`influxdb_database.json` + + * "InfluxDB - Cardinality" + +`influxdb_httpd.json` + + * "InfluxDB - Write HTTP Requests" + * "InfluxDB - Query Requests" + * "InfluxDB - Client Failures" + +`influxdb_queryExecutor.json` + + * "InfluxDB - Query Performance" + +`influxdb_write.json` + + * "InfluxDB - Write Points" + * "InfluxDB - Write Errors" + +## kubernetes + +`kubernetes_node.json` + +* "K8s - Node Millicores" +* "K8s - Node Memory Bytes" + +`kubernetes_pod_container.json` + +* "K8s - Pod Millicores" +* "K8s - Pod Memory Bytes" + +`kubernetes_pod_network.json` + +* "K8s - Pod TX Bytes/Second" +* "K8s - Pod RX Bytes/Second " + +`kubernetes_system_container.json` + +* "K8s - Kubelet Millicores" +* "K8s - Kubelet Memory Bytes" + +## Memcached (`memcached`) + +**Required Telegraf plugin:** [Memcached input plugin](/{{< latest "telegraf" >}}/plugins/#memcached) + +`memcached.json` + + * "Memcached - Current Connections" + * "Memcached - Get Hits/Second" + * "Memcached - Get Misses/Second" + * "Memcached - Delete Hits/Second" + * "Memcached - Delete Misses/Second" + * "Memcached - Incr Hits/Second" + * "Memcached - Incr Misses/Second" + * "Memcached - Current Items" + * "Memcached - Total Items" + * "Memcached - Bytes Stored" + * "Memcached - Bytes Written/Sec" + * "Memcached - Evictions/10 Seconds" + + +## mesos + +**Required Telegraf plugin:** [Mesos input plugin](/{{< latest "telegraf" >}}/plugins/#mesos) + +`mesos.json` + + * "Mesos Active Slaves" + * "Mesos Tasks Active" + * "Mesos Tasks" + * "Mesos Outstanding offers" + * "Mesos Available/Used CPUs" + * "Mesos Available/Used Memory" + * "Mesos Master Uptime" + + +## mongodb + +**Required Telegraf plugin:** [MongoDB input plugin](/{{< latest "telegraf" >}}/plugins/#mongodb) + +`mongodb.json` + + * "MongoDB - Read/Second" + * "MongoDB - Writes/Second" + * "MongoDB - Active Connections" + * "MongoDB - Reds/Writes Waiting in Queue" + * "MongoDB - Network Bytes/Second" + +## mysql + +**Required Telegraf plugin:** [MySQL input plugin](/{{< latest "telegraf" >}}/plugins/#mysql) + +`mysql.json` + + * "MySQL - Reads/Second" + * "MySQL - Writes/Second" + * "MySQL - Connections/Second" + * "MySQL - Connection Errors/Second" + +## nginx + +**Required Telegraf plugin:** [NGINX input plugin](/{{< latest "telegraf" >}}/plugins/#nginx) + +`nginx.json` + + * "NGINX - Client Connections" + * "NGINX - Client Errors" + * "NGINX - Client Requests" + * "NGINX - Active Client State" + +## nsq + +**Required Telegraf plugin:** [NSQ input plugin](/{{< latest "telegraf" >}}/plugins/#nsq) + +`nsq_channel.json` + + * "NSQ - Channel Client Count" + * "NSQ - Channel Messages Count" + +`nsq_server.json` + + * "NSQ - Topic Count" + * "NSQ - Server Count" + +`nsq_topic.json` + + * "NSQ - Topic Messages" + * "NSQ - Topic Messages on Disk" + * "NSQ - Topic Ingress" + * "NSQ topic egress" + +## phpfpm + +**Required Telegraf plugin:** [PHPfpm input plugin](/{{< latest "telegraf" >}}/plugins/#php-fpm) + +`phpfpm.json` + + * "phpfpm - Accepted Connections" + * "phpfpm - Processes" + * "phpfpm - Slow Requests" + * "phpfpm - Max Children Reached" + + +## ping + +**Required Telegraf plugin:** [Ping input plugin](/{{< latest "telegraf" >}}/plugins/#ping) + +`ping.json` + + * "Ping - Packet Loss Percent" + * "Ping - Response Times (ms)" + +## postgresql + +**Required Telegraf plugin:** [PostgreSQL input plugin](/{{< latest "telegraf" >}}/plugins/#postgresql) + +`postgresql.json` + + * "PostgreSQL - Rows" + * "PostgreSQL - QPS" + * "PostgreSQL - Buffers" + * "PostgreSQL - Conflicts/Deadlocks" + +## rabbitmq + +**Required Telegraf plugin:** [RabbitMQ input plugin](/{{< latest "telegraf" >}}/plugins/#rabbitmq) + +`rabbitmq.json` + + * "RabbitMQ - Overview" + * "RabbitMQ - Published/Delivered per second" + * "RabbitMQ - Acked/Unacked per second" + + +## redis + +**Required Telegraf plugin:** [Redis input plugin](/{{< latest "telegraf" >}}/plugins/#redis) + + +`redis.json` + + * "Redis - Connected Clients" + * "Redis - Blocked Clients" + * "Redis - CPU" + * "Redis - Memory" + +## riak + +**Required Telegraf plugin:** [Riak input plugin](/{{< latest "telegraf" >}}/plugins/#riak) + + +`riak.json` + + * "Riak - Toal Memory Bytes" + * "Riak - Object Byte Size" + * "Riak - Number of Siblings/Minute" + * "Riak - Latency (ms)" + * "Riak - Reads and Writes/Minute" + * "Riak - Active Connections" + * "Riak - Read Repairs/Minute" + +## system + + The `system` application includes metrics that require all of the listed plugins. If any of the following plugins aren't enabled, the metrics associated with the plugins will not display data. + +### cpu + +**Required Telegraf plugin:** [CPU input plugin](/{{< latest "telegraf" >}}/plugins/#cpu) + +`cpu.json` + + * "CPU Usage" + +### disk + +`disk.json` + +**Required Telegraf plugin:** [Disk input plugin](/{{< latest "telegraf" >}}/plugins/#disk) + + * "System - Disk used %" + +### diskio + +**Required Telegraf plugin:** [DiskIO input plugin](/{{< latest "telegraf" >}}/plugins/#diskio) + +`diskio.json` + + * "System - Disk MB/s" +* + +### mem + +**Required Telegraf plugin:** [Mem input plugin](/{{< latest "telegraf" >}}/plugins/#mem) + +`mem.json` + + * "System - Memory Gigabytes Used" + +### net + +**Required Telegraf plugin:** [Net input plugin](/{{< latest "telegraf" >}}/plugins/#net) + +`net.json` + + * "System - Network Mb/s" + * "System - Network Error Rate" + +### netstat + +**Required Telegraf plugin:** [Netstat input plugin](/{{< latest "telegraf" >}}/plugins/#netstat) + +`netstat.json` + + * "System - Open Sockets" + * "System - Sockets Created/Second" + +### processes + +**Required Telegraf plugin:** [Processes input plugin](/{{< latest "telegraf" >}}/plugins/#processes) + +`processes.json` + + * "System - Total Processes" + +### procstat + +**Required Telegraf plugin:** [Procstat input plugin](/{{< latest "telegraf" >}}/plugins/#procstat) + +`procstat.json` + + * "Processes - Resident Memory (MB)" + * "Processes – CPU Usage %" + +### system + +**Required Telegraf plugin:** [Procstat input plugin](/{{< latest "telegraf" >}}/plugins/#procstat) + +`load.json` + + * "System Load" + +## varnish + +**Required Telegraf plugin:** [Varnish](/{{< latest "telegraf" >}}/plugins/#varnish) + +`varnish.json` + + * "Varnish - Cache Hits/Misses" + + +## win_system + +**Required Telegraf plugin:** [Windows Performance Counters input plugin](/{{< latest "telegraf" >}}/plugins/#windows-performance-counters) + +`win_cpu.json` + + * "System - CPU Usage" + +`win_mem.json` + + * "System - Available Bytes" + +`win_net.json` + + * "System - TX Bytes/Second" + * "RX Bytes/Second" + +`win_system.json` + + * "System - Load" diff --git a/content/chronograf/v1.9/guides/visualization-types.md b/content/chronograf/v1.9/guides/visualization-types.md new file mode 100644 index 000000000..0676370ef --- /dev/null +++ b/content/chronograf/v1.9/guides/visualization-types.md @@ -0,0 +1,278 @@ +--- +title: Visualization types in Chronograf +descriptions: > + Chronograf provides multiple visualization types to visualize your data in a format that makes to the most sense for your use case. +menu: + chronograf_1_9: + name: Visualization types + weight: 40 + parent: Guides +--- + +Chronograf's dashboard views support the following visualization types, which can be selected in the **Visualization Type** selection view of the [Data Explorer](/chronograf/v1.9/guides/querying-data). + +[Visualization Type selector](/img/chronograf/1-6-viz-types-selector.png) + +Each of the available visualization types and available user controls are described below. + +* [Line Graph](#line-graph) +* [Stacked Graph](#stacked-graph) +* [Step-Plot Graph](#step-plot-graph) +* [Single Stat](#single-stat) +* [Line Graph + Single Stat](#line-graph-single-stat) +* [Bar Graph](#bar-graph) +* [Gauge](#gauge) +* [Table](#table) +* [Note](#note) + +For information on adding and displaying annotations in graph views, see [Adding annotations to Chronograf views](/chronograf/v1.9/guides/annotations/). + +### Line Graph + +The **Line Graph** view displays a time series in a line graph. + +![Line Graph selector](/img/chronograf/1-6-viz-line-graph-selector.png) + +#### Line Graph Controls + +![Line Graph Controls](/img/chronograf/1-6-viz-line-graph-controls.png) + +Use the **Line Graph Controls** to specify the following: + +* **Title**: y-axis title. Enter title, if using a custom title. + - **auto**: Enable or disable auto-setting. +* **Min**: Minimum y-axis value. + - **auto**: Enable or disable auto-setting. +* **Max**: Maximum y-axis value. + - **auto**: Enable or disable auto-setting. +* **Y-Value's Prefix**: Prefix to be added to y-value. +* **Y-Value's Suffix**: Suffix to be added to y-value. +* **Y-Value's Format**: Toggle between **K/M/B** (Thousand/Million/Billion) and **K/M/G** (Kilo/Mega/Giga). +* **Scale**: Toggle between **Linear** and **Logarithmic**. +* **Static Legend**: Toggle between **Show** and **Hide**. + +#### Line Graph example + +![Line Graph example](/img/chronograf/1-6-viz-line-graph-example.png) + + +### Stacked Graph + +The **Stacked Graph** view displays multiple time series bars as segments stacked on top of each other. + +![Stacked Graph selector](/img/chronograf/1-6-viz-stacked-graph-selector.png) + +#### Stacked Graph Controls + +![Stacked Graph Controls](/img/chronograf/1-6-viz-stacked-graph-controls.png) + +Use the **Stacked Graph Controls** to specify the following: + +* **Title**: y-axis title. Enter title, if using a custom title. + - **auto**: Enable or disable auto-setting. +* **Min**: Minimum y-axis value. + - **auto**: Enable or disable auto-setting. +* **Max**: Maximum y-axis value. + - **auto**: Enable or disable auto-setting. +* **Y-Value's Prefix**: Prefix to be added to y-value. +* **Y-Value's Suffix**: Suffix to be added to y-value. +* **Y-Value's Format**: Toggle between **K/M/B** (Thousand/Million/Billion) and **K/M/G** (Kilo/Mega/Giga). +* **Scale**: Toggle between **Linear** and **Logarithmic**. +* **Static Legend**: Toggle between **Show** and **Hide**. + + +#### Stacked Graph example + +![Stacked Graph example](/img/chronograf/1-6-viz-stacked-graph-example.png) + + +### Step-Plot Graph + +The **Step-Plot Graph** view displays a time series in a staircase graph. + +![Step-Plot Graph selector](/img/chronograf/1-6-viz-step-plot-graph-selector.png) + +#### Step-Plot Graph Controls + +![Step-Plot Graph Controls](/img/chronograf/1-6-viz-step-plot-graph-controls.png) + +Use the **Step-Plot Graph Controls** to specify the following: + +* **Title**: y-axis title. Enter title, if using a custom title. + - **auto**: Enable or disable auto-setting. +* **Min**: Minimum y-axis value. + - **auto**: Enable or disable auto-setting. +* **Max**: Maximum y-axis value. + - **auto**: Enable or disable auto-setting. +* **Y-Value's Prefix**: Prefix to be added to y-value. +* **Y-Value's Suffix**: Suffix to be added to y-value. +* **Y-Value's Format**: Toggle between **K/M/B** (Thousand/Million/Billion) and **K/M/G** (Kilo/Mega/Giga). +* **Scale**: Toggle between **Linear** and **Logarithmic**. + +#### Step-Plot Graph example + +![Step-Plot Graph example](/img/chronograf/1-6-viz-step-plot-graph-example.png) + + +### Bar Graph + +The **Bar Graph** view displays the specified time series using a bar chart. + +To select this view, click the Bar Graph selector icon. + +![Bar Graph selector](/img/chronograf/1-6-viz-bar-graph-selector.png) + +#### Bar Graph Controls + +![Bar Graph Controls](/img/chronograf/1-6-viz-bar-graph-controls.png) + +Use the **Bar Graph Controls** to specify the following: + +* **Title**: y-axis title. Enter title, if using a custom title. + - **auto**: Enable or disable auto-setting. +* **Min**: Minimum y-axis value. + - **auto**: Enable or disable auto-setting. +* **Max**: Maximum y-axis value. + - **auto**: Enable or disable auto-setting. +* **Y-Value's Prefix**: Prefix to be added to y-value. +* **Y-Value's Suffix**: Suffix to be added to y-value. +* **Y-Value's Format**: Toggle between **K/M/B** (Thousand/Million/Billion) and **K/M/G** (Kilo/Mega/Giga). +* **Scale**: Toggle between **Linear** and **Logarithmic**. + +#### Bar Graph example + +![Bar Graph example](/img/chronograf/1-6-viz-bar-graph-example.png) + + +### Line Graph + Single Stat + +The **Line Graph + Single Stat** view displays the specified time series in a line graph and overlays the single most recent value as a large numeric value. + +To select this view, click the **Line Graph + Single Stat** view option. + +![Line Graph + Single Stat selector](/img/chronograf/1-6-viz-line-graph-single-stat-selector.png) + +#### Line Graph + Single Stat Controls + +![Line Graph + Single Stat Controls](/img/chronograf/1-6-viz-line-graph-single-stat-controls.png) + +Use the **Line Graph + Single Stat Controls** to specify the following: + +* **Title**: y-axis title. Enter title, if using a custom title. + - **auto**: Enable or disable auto-setting. +* **Min**: Minimum y-axis value. + - **auto**: Enable or disable auto-setting. +* **Max**: Maximum y-axis value. + - **auto**: Enable or disable auto-setting. +* **Y-Value's Prefix**: Prefix to be added to y-value. +* **Y-Value's Suffix**: Suffix to be added to y-value. +* **Y-Value's Format**: Toggle between **K/M/B** (Thousand/Million/Billion) and **K/M/G** (Kilo/Mega/Giga). +* **Scale**: Toggle between **Linear** and **Logarithmic**. + +#### Line Graph + Single Stat example + +![Line Graph + Single Stat example](/img/chronograf/1-6-viz-line-graph-single-stat-example.png) + +### Single Stat + +The **Single Stat** view displays the most recent value of the specified time series as a numerical value. + +![Single Stat view](/img/chronograf/1-6-viz-single-stat-selector.png) + +If a cell's query includes a [`GROUP BY` tag](/{{< latest "influxdb" "v1" >}}/query_language/explore-data/#group-by-tags) clause, Chronograf sorts the different [series](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#series) lexicographically and shows the most recent [field value](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#field-value) associated with the first series. +For example, if a query groups by the `name` [tag key](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#tag-key) and `name` has two [tag values](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#tag-value) (`chronelda` and `chronz`), Chronograf shows the most recent field value associated with the `chronelda` series. + +If a cell's query includes more than one [field key](/{{< latest "influxdb" "v1" >}}/concepts/glossary/#field-key) in the [`SELECT` clause](/{{< latest "influxdb" "v1" >}}/query_language/explore-data/#select-clause), Chronograf returns the most recent field value associated with the first field key in the `SELECT` clause. +For example, if a query's `SELECT` clause is `SELECT "chronogiraffe","chronelda"`, Chronograf shows the most recent field value associated with the `chronogiraffe` field key. + +#### Single Stat Controls + +Use the **Single Stat Controls** panel to specify one or more thresholds: + +* **Add Threshold**: Button to add a new threshold. +* **Base Color**: Select a base, or background, color from the selection list. + * Color options: Ruby, Fire, Curacao, Tiger, Pineapple, Thunder, Honeydew, Rainforest, Viridian, Ocean, Pool, Laser (default), Planet, Star, Comet, Pepper, Graphite, White, and Castle. +* **Prefix**: Prefix. For example, `%`, `MPH`, etc. +* **Suffix**: Suffix. For example, `%`, `MPH`, etc. +* Threshold Coloring: Toggle between **Background** and **Text** + + +### Gauge + +The **Gauge** view displays the single value most recent value for a time series in a gauge view. + +To select this view, click the Gauge selector icon. + +![Gauge selector](/img/chronograf/1-6-viz-gauge-selector.png) + +#### Gauge Controls + +![Gauge Controls](/img/chronograf/1-6-viz-gauge-controls.png) + +Use the **Gauge Controls** to specify the following: + +* **Add Threshold**: Click button to add a threshold. +* **Min**: Minimum value for the threshold. + - Select color to display. Selection list options include: Laser (default), Ruby, Fire, Curacao, Tiger, Pineapple, Thunder, and Honeydew. +* **Max**: Maximum value for the threshold. + - Select color to display. Selection list options include: Laser (default), Ruby, Fire, Curacao, Tiger, Pineapple, Thunder, and Honeydew. +* **Prefix**: Prefix. For example, `%`, `MPH`, etc. +* **Suffix**: Suffix. For example, `%`, `MPH`, etc. + +#### Gauge example + +![Gauge example](/img/chronograf/1-6-viz-gauge-example.png) + +### Table + +The **Table** panel displays the results of queries in a tabular view, which is sometimes easier to analyze than graph views of data. + +![Table selector](/img/chronograf/1-6-viz-table-selector.png) + +#### Table Controls + +![Table Controls](/img/chronograf/1-6-viz-table-controls.png) + +Use the **Table Controls** to specify the following: + +- **Default Sort Field**: Select the default sort field. Default is **time**. +- **Decimal Places**: Enter the number of decimal places. Default (empty field) is **unlimited**. +- **Time Axis**: Select **Vertical** or **Horizontal**. +- **Time Format**: Select the time format. + - Options include: `MM/DD/YYYY HH:mm:ss` (default), `MM/DD/YYYY HH:mm:ss.SSS`, `YYYY-MM-DD HH:mm:ss`, `HH:mm:ss`, `HH:mm:ss.SSS`, `MMMM D, YYYY HH:mm:ss`, `dddd, MMMM D, YYYY HH:mm:ss`, or `Custom`. +- **Lock First Column**: Lock the first column so that the listings are always visible. Threshold settings do not apply in the first column when locked. +- **Customize Field**: + - **time**: Enter a new name to rename. + - [additional]: Enter name for each additional column. + - Change the order of the column by dragging to the desired position. +- **Thresholds** + + {{% note %}} +**Note:** Threshold settings apply to any cells with values, except when they appear in the first column and **Lock First Column** is enabled. + {{% /note %}} + + - **Add Threshold** (button): Click to add a threshold. + - **Base Color**: Select a base, or background, color from the selection list. + - Color options: Ruby, Fire, Curacao, Tiger, Pineapple, Thunder, Honeydew, Rainforest, Viridian, Ocean, Pool, Laser (default), Planet, Star, Comet, Pepper, Graphite, White, and Castle. + +#### Table view example + +![Table example](/img/chronograf/1-6-viz-table-example.png) + +### Note + +The **Note** panel displays Markdown-formatted text with your graph. + +![Note selector](/img/chronograf/1-7-viz-note-selector.png) + +#### Note Controls + +![Note Controls](/img/chronograf/1-7-viz-note-controls.png) + +Enter your text in the **Add a Note** panel, using Markdown to format the text. + +Enable the **Display note in cell when query returns no results** option to display the note text in the cell instead of `No Results`. + +#### Note view example + +![Note example](/img/chronograf/1-7-viz-note-example.png) diff --git a/content/chronograf/v1.9/guides/write-to-influxdb.md b/content/chronograf/v1.9/guides/write-to-influxdb.md new file mode 100644 index 000000000..15f1e1848 --- /dev/null +++ b/content/chronograf/v1.9/guides/write-to-influxdb.md @@ -0,0 +1,102 @@ +--- +title: Write data to InfluxDB +description: + Use Chronograf to write data to InfluxDB. Upload line protocol into the UI, use the + InfluxQL `INTO` clause, or use the Flux `to()` function to write data back to InfluxDB. +menu: + chronograf_1_9: + name: Write data to InfluxDB + parent: Guides +weight: 140 +--- + +Use Chronograf to write data to InfluxDB. +Choose from the following methods: + +- [Upload line protocol through the Chronograf UI](#upload-line-protocol-through-the-chronograf-ui) +- [Use the InfluxQL `INTO` clause in a query](#use-the-influxql-into-clause-in-a-query) +- [Use the Flux `to()` function in a query](#use-the-flux-to-function-in-a-query) + +## Upload line protocol through the Chronograf UI + +1. Select **{{< icon "data-explorer" >}} Explore** in the left navigation bar. +2. Click **Write Data** in the top right corner of the Data Explorer. + + {{< img-hd src="/img/chronograf/1-9-write-data.png" alt="Write data to InfluxDB with Chronograf" />}} + +3. Select the **database** _(if an InfluxQL data source is selected)_ or + **database and retention policy** _(if a Flux data source is selected)_ to write to. + + {{< img-hd src="/img/chronograf/1-9-write-db-rp.png" alt="Select database and retention policy to write to" />}} + +4. Select one of the following methods for uploading [line protocol](/{{< latest "influxdb" "v1" >}}/write_protocols/line_protocol_tutorial/): + + - **Upload File**: Upload a file containing line protocol to write to InfluxDB. + Either drag and drop a file into the file uploader or click to use your + operating systems file selector and choose a file to upload. + - **Manual Entry**: Manually enter line protocol to write to InfluxDB. + +5. Select the timestamp precision of your line protocol. + Chronograf supports the following units: + + - `s` (seconds) + - `ms` (milliseconds) + - `u` (microseconds) + - `ns` (nanoseconds) + + {{< img-hd src="/img/chronograf/1-9-write-precision.png" alt="Select write precision in Chronograf" />}} + +5. Click **Write**. + +## Use the InfluxQL `INTO` clause in a query +To write data back to InfluxDB with an InfluxQL query, include the +[`INTO` clause](/{{< latest "influxdb" "v1" >}}/query_language/explore-data/#the-into-clause) +in your query: + +1. Select **{{< icon "data-explorer" >}} Explore** in the left navigation bar. +2. Select **InfluxQL** as your data source type. +3. Write an InfluxQL query that includes the `INTO` clause. Specify the database, + retention policy, and measurement to write to. For example: + + ```sql + SELECT * + INTO "mydb"."autogen"."example-measurement" + FROM "example-db"."example-rp"."example-measurement" + GROUP BY * + ``` + +4. Click **Submit Query**. + +{{% note %}} +#### Use InfluxQL to write to InfluxDB 2.x or InfluxDB Cloud +To use InfluxQL to write to an **InfluxDB 2.x** or **InfluxDB Cloud** instance, +[configure database and retention policy mappings](/{{< latest "influxdb" >}}/upgrade/v1-to-v2/manual-upgrade/#create-dbrp-mappings) +and ensure the current [InfluxDB connection](/chronograf/v1.9/administration/creating-connections/#manage-influxdb-connections-using-the-chronograf-ui) +includes the appropriate connection credentials. +{{% /note %}} + +## Use the Flux `to()` function in a query +To write data back to InfluxDB with an InfluxQL query, include the +[`INTO` clause](/{{< latest "influxdb" "v1" >}}/query_language/explore-data/#the-into-clause) +in your query: + +1. Select **{{< icon "data-explorer" >}} Explore** in the left navigation bar. +2. Select **Flux** as your data source type. + + {{% note %}} +To query InfluxDB with Flux, [enable Flux](/{{< latest "influxdb" "v1" >}}/flux/installation/) +in your InfluxDB configuration. + {{% /note %}} + +3. Write an Flux query that includes the `to()` function. + Provide the database and retention policy to write to. + Use the `db-name/rp-name` syntax: + + ```js + from(bucket: "example-db/example-rp") + |> range(start: -30d) + |> filter(fn: (r) => r._measurement == "example-measurement") + |> to(bucket: "mydb/autogen") + ``` + +4. Click **Run Script**. diff --git a/content/chronograf/v1.9/introduction/_index.md b/content/chronograf/v1.9/introduction/_index.md new file mode 100644 index 000000000..a24d13f62 --- /dev/null +++ b/content/chronograf/v1.9/introduction/_index.md @@ -0,0 +1,14 @@ +--- +title: Introduction to Chronograf +description: > + An introduction to Chronograf, the user interface and data visualization component for the InfluxData Platform. Includes documentation on getting started, installation, and downloading. + +menu: + chronograf_1_9: + name: Introduction + weight: 20 +--- + +Follow the links below to get acquainted with Chronograf: + +{{< children >}} diff --git a/content/chronograf/v1.9/introduction/downloading.md b/content/chronograf/v1.9/introduction/downloading.md new file mode 100644 index 000000000..09efacec5 --- /dev/null +++ b/content/chronograf/v1.9/introduction/downloading.md @@ -0,0 +1,10 @@ +--- +title: Download Chronograf +menu: + chronograf_1_9: + name: Download + weight: 10 + parent: Introduction +--- + +Download the latest Chronograf release at the [InfluxData download page](https://portal.influxdata.com/downloads). diff --git a/content/chronograf/v1.9/introduction/getting-started.md b/content/chronograf/v1.9/introduction/getting-started.md new file mode 100644 index 000000000..b21aa8b02 --- /dev/null +++ b/content/chronograf/v1.9/introduction/getting-started.md @@ -0,0 +1,29 @@ +--- +title: Get started with Chronograf +description: > + Overview of data visualization, alerting, and infrastructure monitoring features available in Chronograf. +aliases: + - /chronograf/v1.9/introduction/getting_started/ +menu: + chronograf_1_9: + name: Get started + weight: 30 + parent: Introduction +--- + +## Overview +Chronograf allows you to quickly see data you have stored in InfluxDB so you can build robust queries and alerts. After your administrator has set up Chronograf as described in [Installing Chronograf](/chronograf/v1.9/introduction/installation), get started with key features using the guides below. + +### Data visualization +* Investigate your data by building queries using the [Data Explorer](/chronograf/v1.9/guides/querying-data/). +* Use [pre-created dashboards](/chronograf/v1.9/guides/using-precreated-dashboards/) to monitor your application data or [create your own dashboards](/chronograf/v1.9/guides/create-a-dashboard/). +* Customize dashboards using [template variables](/chronograf/v1.9/guides/dashboard-template-variables/). + +### Alerting +* [Create alert rules](/chronograf/v1.9/guides/create-alert-rules/) to generate threshold, relative, and deadman alerts on your data. +* [View all active alerts](/chronograf/v1.9/guides/create-alert-rules/#step-2-view-the-alerts) on an alert dashboard. +* Use [alert endpoints](/chronograf/v1.9/guides/configuring-alert-endpoints/) in Chronograf to send alert messages to specific URLs and applications. + +### Infrastructure monitoring +* [View all hosts](/chronograf/v1.9/guides/monitoring-influxenterprise-clusters/#step-4-explore-the-monitoring-data-in-chronograf) and their statuses in your infrastructure. +* [Use pre-created dashboards](/chronograf/v1.9/guides/using-precreated-dashboards/) to monitor your applications. diff --git a/content/chronograf/v1.9/introduction/installation.md b/content/chronograf/v1.9/introduction/installation.md new file mode 100644 index 000000000..5473f7e7a --- /dev/null +++ b/content/chronograf/v1.9/introduction/installation.md @@ -0,0 +1,96 @@ +--- +title: Install Chronograf +description: Download and install Chronograf. +menu: + chronograf_1_9: + name: Install + weight: 20 + parent: Introduction +--- + +This page describes how to download and install Chronograf. + +### Content + +* [TICK overview](#tick-overview) +* [Download and install](#download-and-install) +* [Connect to your InfluxDB instance or InfluxDB Enterprise cluster](#connect-chronograf-to-your-influxdb-instance-or-influxdb-enterprise-cluster) +* [Connect to Kapacitor](#connect-chronograf-to-kapacitor) + +## TICK overview + +Chronograf is the user interface for InfluxData's [TICK stack](https://www.influxdata.com/time-series-platform/). + +## Download and install + +The latest Chronograf builds are available on InfluxData's [Downloads page](https://portal.influxdata.com/downloads). + +1. Choose the download link for your operating system. + + {{% note %}} +If your download includes a TAR package, save the underlying datastore `chronograf-v1.db` in directory outside of where you start Chronograf. This preserves and references your existing datastore, including configurations and dashboards, when you download future versions. + {{% /note %}} + +2. Install Chronograf, replacing `` with the appropriate version: + + {{% tabs-wrapper %}} + {{% tabs %}} +[macOS](#) +[Ubuntu & Debian](#) +[RedHat & CentOS](#) + {{% /tabs %}} + {{% tab-content %}} +```sh +tar zxvf chronograf-_darwin_amd64.tar.gz +``` + {{% /tab-content %}} + + {{% tab-content %}} +```sh +sudo dpkg -i chronograf__amd64.deb +``` + {{% /tab-content %}} + + {{% tab-content %}} +```sh +sudo yum localinstall chronograf-.x86_64.rpm +``` + {{% /tab-content %}} + {{% /tabs-wrapper %}} + +3. Start Chronograf: + + ```sh + chronograf + ``` + +## Connect Chronograf to your InfluxDB instance or InfluxDB Enterprise cluster + +1. In a browser, navigate to [localhost:8888](http://localhost:8888). +2. Provide the following details: + - **Connection String**: InfluxDB hostname or IP, and port (default port is `8086`). + - **Connection Name**: Connection name. + - **Username** and **Password**: If you've enabled + [InfluxDB authentication](/{{< latest "influxdb" "v1" >}}/administration/authentication_and_authorization), + provide your InfluxDB username and password. Otherwise, leave blank. + + {{% note %}} +To ensure distinct permissions can be applied, Chronograf user accounts and +credentials should be different than InfluxDB credentials. +For example, you may want to set up Chronograf to run as a service account +with read-only permissions to InfluxDB. For more information, see how to +[manage InfluxDB users in Chronograf](/chronograf/v1.9/administration/managing-influxdb-users/) +and [manage Chronograf users](/chronograf/v1.9/administration/managing-chronograf-users/). + {{% /note %}} + + - **Telegraf Database Name**: _(Optional)_ Telegraf database name. + Default name is `telegraf`. +3. Click **Add Source**. + +## Connect Chronograf to Kapacitor + +1. In Chronograf, click the configuration (wrench) icon in the sidebar menu, then select **Add Config** in the **Active Kapacitator** column. +2. In the **Kapacitor URL** field, enter the hostname or IP of the machine that Kapacitor is running on. Be sure to include Kapacitor's default port: `9092`. +3. Enter a name for your connection. +4. Leave the **Username** and **Password** fields blank unless you've specifically enabled authorization in Kapacitor. +5. Click **Connect**. diff --git a/content/chronograf/v1.9/tools/_index.md b/content/chronograf/v1.9/tools/_index.md new file mode 100644 index 000000000..fb3d9b75e --- /dev/null +++ b/content/chronograf/v1.9/tools/_index.md @@ -0,0 +1,13 @@ +--- +title: Chronograf Tools +description: > + Chronograf provides command line tools designed to aid in managing and working with Chronograf from the command line. +menu: + chronograf_1_9: + name: Tools + weight: 40 +--- + +Chronograf provides command line tools designed to aid in managing and working with Chronograf from the command line. The following command line interfaces (CLIs) are available: + +{{< children hlevel="h2" >}} diff --git a/content/chronograf/v1.9/tools/chronoctl/_index.md b/content/chronograf/v1.9/tools/chronoctl/_index.md new file mode 100644 index 000000000..58db42a86 --- /dev/null +++ b/content/chronograf/v1.9/tools/chronoctl/_index.md @@ -0,0 +1,26 @@ +--- +title: chronoctl +description: > + The `chronoctl` command line interface (CLI) includes commands to interact with an instance of Chronograf's data store. +menu: + chronograf_1_9: + name: chronoctl + parent: Tools + weight: 10 +--- + +The `chronoctl` command line interface (CLI) includes commands to interact with an instance of Chronograf's data store. + +## Usage +``` +chronoctl [command] +chronoctl [flags] +``` + +## Commands + +| Command | Description | +|:------- |:----------- | +| [add-superadmin](/chronograf/v1.9/tools/chronoctl/add-superadmin/) | Create a new user with superadmin status | +| [list-users](/chronograf/v1.9/tools/chronoctl/list-users) | List all users in the Chronograf data store | +| [migrate](/chronograf/v1.9/tools/chronoctl/migrate) | Migrate your Chronograf configuration store | diff --git a/content/chronograf/v1.9/tools/chronoctl/add-superadmin.md b/content/chronograf/v1.9/tools/chronoctl/add-superadmin.md new file mode 100644 index 000000000..b72588c04 --- /dev/null +++ b/content/chronograf/v1.9/tools/chronoctl/add-superadmin.md @@ -0,0 +1,28 @@ +--- +title: chronoctl add-superadmin +description: > + The `add-superadmin` command creates a new user with superadmin status. +menu: + chronograf_1_9: + name: chronoctl add-superadmin + parent: chronoctl + weight: 20 +--- + +The `add-superadmin` command creates a new user with superadmin status. + +## Usage +``` +chronoctl add-superadmin [flags] +``` + +## Flags + +| Flag | | Description | Input type | +|:---- |:----------------- | :---------------------------------------------------------------------------------------------------- | :--------: | +| `-b` | `--bolt-path` | Full path to boltDB file (e.g. `./chronograf-v1.db`)" env:"BOLT_PATH" default:"chronograf-v1.db" | string | +| `-i` | `--id` | User ID for an existing user | uint64 | +| `-n` | `--name` | User's name. Must be Oauth-able email address or username. | | +| `-p` | `--provider` | Name of the Auth provider (e.g. Google, GitHub, auth0, or generic) | string | +| `-s` | `--scheme` | Authentication scheme that matches auth provider (default:oauth2) | string | +| `-o` | `--orgs` | A comma-separated list of organizations that the user should be added to (default:"default") | string | diff --git a/content/chronograf/v1.9/tools/chronoctl/list-users.md b/content/chronograf/v1.9/tools/chronoctl/list-users.md new file mode 100644 index 000000000..be43a76b5 --- /dev/null +++ b/content/chronograf/v1.9/tools/chronoctl/list-users.md @@ -0,0 +1,23 @@ +--- +title: chronoctl list-users +description: > + The `list-users` command lists all users in the Chronograf data store. + +menu: + chronograf_1_9: + name: chronoctl list-users + parent: chronoctl + weight: 30 +--- + +The `list-users` command lists all users in the Chronograf data store. + +## Usage +``` +chronoctl list-users [flags] +``` + +## Flags +| Flag | | Description | Input type | +| :---- |:----------- | :------------------------------------------------------------ | :--------: | +| `-b` | `--bolt-path` | Full path to boltDB file (e.g. `./chronograf-v1.db`)" env:"BOLT_PATH" (default:chronograf-v1.db) | string | diff --git a/content/chronograf/v1.9/tools/chronoctl/migrate.md b/content/chronograf/v1.9/tools/chronoctl/migrate.md new file mode 100644 index 000000000..eec096c45 --- /dev/null +++ b/content/chronograf/v1.9/tools/chronoctl/migrate.md @@ -0,0 +1,46 @@ +--- +title: chronoctl migrate +description: > + The `migrate` command allows you to migrate your Chronograf configuration store. +menu: + chronograf_1_9: + name: chronoctl migrate + parent: chronoctl + weight: 40 +--- + +The `migrate` command lets you migrate your Chronograf configuration store. + +By default, Chronograf is delivered with BoltDB as a data store. For information on migrating from BoltDB to an etcd cluster as a data store, +see [Migrating to a Chronograf HA configuration](/chronograf/v1.9/administration/migrate-to-high-availability). + +## Usage +``` +chronoctl migrate [flags] +``` + +## Flags +| Flag | | Description | Input type | +|:---- |:--- |:----------- |:----------: | +| `-f` | `--from` | Full path to BoltDB file or etcd (e.g. `bolt:///path/to/chronograf-v1.db` or `etcd://user:pass@localhost:2379` (default: `chronograf-v1.db`) | string | +| `-t` | `--to` | Full path to BoltDB file or etcd (e.g. `bolt:///path/to/chronograf-v1.db` or `etcd://user:pass@localhost:2379` (default: `etcd://localhost:2379`) | string | + +#### Provide etcd authentication credentials +If authentication is enabled on `etcd`, use the standard URI basic +authentication format to define a username and password. For example: + +```sh +etcd://username:password@localhost:2379 +``` + +#### Provide etcd TLS credentials +If TLS is enabled on `etcd`, provide your TLS certificate credentials using +the following query parameters in your etcd URL: + +- **cert**: Path to client certificate file or PEM file +- **key**: Path to client key file +- **ca**: Path to trusted CA certificates + +```sh +etcd://127.0.0.1:2379?cert=/tmp/client.crt&key=/tst/client.key&ca=/tst/ca.crt +``` diff --git a/content/chronograf/v1.9/tools/chronograf-cli/_index.md b/content/chronograf/v1.9/tools/chronograf-cli/_index.md new file mode 100644 index 000000000..906ff2605 --- /dev/null +++ b/content/chronograf/v1.9/tools/chronograf-cli/_index.md @@ -0,0 +1,145 @@ +--- +title: chronograf CLI +description: > + The `chronograf` command line interface (CLI) includes options to manage many aspects of Chronograf security. +menu: + chronograf_1_9: + name: chronograf CLI + parent: Tools + weight: 10 + +--- + +The `chronograf` command line interface (CLI) includes options to manage Chronograf security. + +## Usage +``` +chronograf [flags] +``` + +## Chronograf service flags + +| Flag | Description | Env. Variable | +|:-----------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------|:---------------------| +| `--host` | IP the Chronograf service listens on. By default, `0.0.0.0` | `$HOST` | +| `--port` | Port the Chronograf service listens on for insecure connections. By default, `8888` | `$PORT` | +| `-b`,`--bolt-path` | File path to the BoltDB file. By default, `./chronograf-v1.db` | `$BOLT_PATH` | +| `-c`,`--canned-path` | File path to the directory of canned dashboard files. By default, `/usr/share/chronograf/canned` | `$CANNED_PATH` | +| `--resources-path` | Path to directory of canned dashboards, sources, Kapacitor connections, and organizations. By default, `/usr/share/chronograf/resources` | `$RESOURCES_PATH` | +| `-p`, `--basepath` | URL path prefix under which all Chronograf routes will be mounted. | `$BASE_PATH` | +| `--status-feed-url` | URL of JSON feed to display as a news feed on the client status page. By default, `https://www.influxdata.com/feed/json` | `$STATUS_FEED_URL` | +| `-v`, `--version` | Displays the version of the Chronograf service | | +| `-h`, `--host-page-disabled` | Disables the hosts page | `$HOST_PAGE_DISABLED`| + +## InfluxDB connection flags + +| Flag | Description | Env. Variable | +| :-------------------- | :-------------------------------------------------------------------------------------- | :------------------- | +| `--influxdb-url` | InfluxDB URL, including the protocol, IP address, and port | `$INFLUXDB_URL` | +| `--influxdb-username` | InfluxDB username | `$INFLUXDB_USERNAME` | +| `--influxdb-password` | InfluxDB password | `$INFLUXDB_PASSWORD` | +| `--influxdb-org` | InfluxDB 2.x or InfluxDB Cloud organization name | `$INFLUXDB_ORG` | +| `--influxdb-token` | InfluxDB 2.x or InfluxDB Cloud [authentication token](/influxdb/cloud/security/tokens/) | `$INFLUXDB_TOKEN` | + +## Kapacitor connection flags + +| Flag | Description | Env. Variable | +|:-----------------------|:-------------------------------------------------------------------------------|:----------------------| +| `--kapacitor-url` | Location of your Kapacitor instance, including `http://`, IP address, and port | `$KAPACITOR_URL` | +| `--kapacitor-username` | Username for your Kapacitor instance | `$KAPACITOR_USERNAME` | +| `--kapacitor-password` | Password for your Kapacitor instance | `$KAPACITOR_PASSWORD` | + +## TLS (Transport Layer Security) flags + +| Flag | Description | Env. Variable | +|:--------- |:------------------------------------------------------------ |:--------------------| +| `--cert` | File path to PEM-encoded public key certificate | `$TLS_CERTIFICATE` | +| `--key` | File path to private key associated with given certificate | `$TLS_PRIVATE_KEY` | +| `--tls-ciphers` | Comma-separated list of supported cipher suites. Use `help` to print available ciphers. | `$TLS_CIPHERS` | +| `--tls-min-version` | Minimum version of the TLS protocol that will be negotiated. (default: 1.2) | `$TLS_MIN_VERSION` | +| `--tls-max-version` | Maximum version of the TLS protocol that will be negotiated. | `$TLS_MAX_VERSION` | + +## Other service option flags + +| Flag | Description | Env. Variable | +| :--------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------- | +| `--custom-auto-refresh` | Add custom auto-refresh options using semicolon separated list of label=milliseconds pairs | `$CUSTOM-AUTO-REFRESH | +| `--custom-link` | Add a custom link to Chronograf user menu options using `:` syntax. For multiple custom links, include multiple flags. | | +| `-d`, `--develop` | Run the Chronograf service in developer mode | | +| `-h`, `--help` | Display command line help for Chronograf | | +| `-l`, `--log-level` | Set the logging level. Valid values include `info` (default), `debug`, and `error` | `$LOG_LEVEL` | +| `-r`, `--reporting-disabled` | Disable reporting of usage statistics. Usage statistics reported once every 24 hours include: `OS`, `arch`, `version`, `cluster_id`, and `uptime`. | `$REPORTING_DISABLED` | + +## Authentication option flags + +### General authentication flags + +| Flag | Description | Env. Variable | +| :--------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------- | +| `-t`, `--token-secret` | Secret for signing tokens | `$TOKEN_SECRET` | +| `--auth-duration` | Total duration, in hours, of cookie life for authentication. Default value is `720h`. | `$AUTH_DURATION` | +| `--public-url` | Public URL required to access Chronograf using a web browser. For example, if you access Chronograf using the default URL, the public URL value would be `http://localhost:8888`. Required for Google OAuth 2.0 authentication. Used for Auth0 and some generic OAuth 2.0 authentication providers. | `$PUBLIC_URL` | +| `—-htpasswd` | Path to password file for use with HTTP basic authentication. See [NGINX documentation](https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-http-basic-authentication/) for more on password files. | `$HTPASSWD` | + +### GitHub-specific OAuth 2.0 authentication flags + +| Flag | Description | Env. Variable | +| :----------------------------- | :------------------------------------------------------------------------------------------------------------------------------------- | :------------------ | +| `--github-url` | Github base URL. Default is `https://github.com`. {{< req "Required if using Github Enterprise" >}} | `$GH_URL` | +| `-i`, `--github-client-id` | GitHub client ID value for OAuth 2.0 support | `$GH_CLIENT_ID` | +| `-s`, `--github-client-secret` | GitHub client secret value for OAuth 2.0 support | `$GH_CLIENT_SECRET` | +| `-o`, `--github-organization` | Restricts authorization to users from specified Github organizations. To add more than one organization, add multiple flags. Optional. | `$GH_ORGS` | + +### Google-specific OAuth 2.0 authentication flags + +| Flag | Description | Env. Variable | +|:-------------------------|:--------------------------------------------------------------------------------|:------------------------| +| `--google-client-id` | Google client ID value for OAuth 2.0 support | `$GOOGLE_CLIENT_ID` | +| `--google-client-secret` | Google client secret value for OAuth 2.0 support | `$GOOGLE_CLIENT_SECRET` | +| `--google-domains` | Restricts authorization to users from specified Google email domain. To add more than one domain, add multiple flags. Optional. | `$GOOGLE_DOMAINS` | + + +### Auth0-specific OAuth 2.0 authentication flags + +| Flag | Description | Env. Variable | +|:------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------| +| `--auth0-domain` | Subdomain of your Auth0 client. Available on the configuration page for your Auth0 client. | `$AUTH0_DOMAIN` | +| `--auth0-client-id` | Auth0 client ID value for OAuth 2.0 support | `$AUTH0_CLIENT_ID` | +| `--auth0-client-secret` | Auth0 client secret value for OAuth 2.0 support | `$AUTH0_CLIENT_SECRET` | +| `--auth0-organizations` | Restricts authorization to users specified Auth0 organization. To add more than one organization, add multiple flags. Optional. Organizations are set using an organization key in the user’s `app_metadata`. | `$AUTH0_ORGS` | + +### Heroku-specific OAuth 2.0 authentication flags + +| Flag | Description | Env. Variable | +|:------------------------|:-----------------------------------------------------------------------------------------|:--------------------| +| `--heroku-client-id` | Heroku client ID value for OAuth 2.0 support | `$HEROKU_CLIENT_ID` | +| `--heroku-secret` | Heroku secret for OAuth 2.0 support | `$HEROKU_SECRET` | +| `--heroku-organization` | Restricts authorization to users from specified Heroku organization. To add more than one organization, add multiple flags. Optional. | `$HEROKU_ORGS` | + +### Generic OAuth 2.0 authentication flags + +| Flag | Description | Env. Variable | +| :------------------------ | :----------------------------------------------------------------------------- | :----------------------- | +| `--generic-name` | Generic OAuth 2.0 name presented on the login page | `$GENERIC_NAME` | +| `--generic-client-id` | Generic OAuth 2.0 client ID value. Can be used for a custom OAuth 2.0 service. | `$GENERIC_CLIENT_ID` | +| `--generic-client-secret` | Generic OAuth 2.0 client secret value | `$GENERIC_CLIENT_SECRET` | +| `--generic-scopes` | Scopes requested by provider of web client | `$GENERIC_SCOPES` | +| `--generic-domains` | Email domain required for user email addresses | `$GENERIC_DOMAINS` | +| `--generic-auth-url` | Authorization endpoint URL for the OAuth 2.0 provider | `$GENERIC_AUTH_URL` | +| `--generic-token-url` | Token endpoint URL for the OAuth 2.0 provider | `$GENERIC_TOKEN_URL` | +| `--generic-api-url` | URL that returns OpenID UserInfo-compatible information | `$GENERIC_API_URL` | +| `--oauth-no-pkce` | Disable OAuth PKCE | `$OAUTH_NO_PKCE` | + +### etcd flags + +| Flag | Description | Env. Variable | +| :----------------------- | :--------------------------------------------------------------------------------------------------------- | :---------------------- | +| `-e`, `--etcd-endpoints` | etcd endpoint URL (include multiple flags for multiple endpoints) | `$ETCD_ENDPOINTS` | +| `--etcd-username` | etcd username | `$ETCD_USERNAME` | +| `--etcd-password` | etcd password | `$ETCD_PASSWORD` | +| `--etcd-dial-timeout` | Total time to wait before timing out while connecting to etcd endpoints (0 means no timeout, default: -1s) | `$ETCD_DIAL_TIMEOUT` | +| `--etcd-request-timeout` | Total time to wait before timing out the etcd view or update (0 means no timeout, default: -1s) | `$ETCD_REQUEST_TIMEOUT` | +| `--etcd-cert` | Path to PEM encoded TLS public key certificate for use with TLS | `$ETCD_CERTIFICATE` | +| `--etcd-key` | Path to private key associated with given certificate for use with TLS | `$ETCD_PRIVATE_KEY` | +| `--etcd-root-ca` | Path to root CA certificate for TLS verification | `$ETCD-ROOT-CA | + diff --git a/content/chronograf/v1.9/troubleshooting/_index.md b/content/chronograf/v1.9/troubleshooting/_index.md new file mode 100644 index 000000000..2ea33ed71 --- /dev/null +++ b/content/chronograf/v1.9/troubleshooting/_index.md @@ -0,0 +1,12 @@ +--- +title: Troubleshoot Chronograf +Description: Troubleshoot Chronograf. +menu: + chronograf_1_9: + name: Troubleshoot + weight: 50 +--- + +Follow the link below to access Chronograf's FAQ. + +{{< children hlevel="h2" >}} diff --git a/content/chronograf/v1.9/troubleshooting/frequently-asked-questions.md b/content/chronograf/v1.9/troubleshooting/frequently-asked-questions.md new file mode 100644 index 000000000..af0b96078 --- /dev/null +++ b/content/chronograf/v1.9/troubleshooting/frequently-asked-questions.md @@ -0,0 +1,23 @@ +--- +title: Chronograf frequently asked questions (FAQs) +description: Common issues with Chronograf +menu: + chronograf_1_9: + name: Frequently asked questions (FAQs) + weight: 10 + parent: Troubleshoot +--- + +## How do I connect Chronograf to an InfluxDB Enterprise cluster? + +The connection details form requires additional information when connecting Chronograf to an [InfluxDB Enterprise cluster](/{{< latest "enterprise_influxdb" >}}/). + +When you enter the InfluxDB HTTP bind address in the `Connection String` input, Chronograf automatically checks if that InfluxDB instance is a data node. +If it is a data node, Chronograf automatically adds the `Meta Service Connection URL` input to the connection details form. +Enter the HTTP bind address of one of your cluster's meta nodes into that input and Chronograf takes care of the rest. + +![Cluster connection details](/img/chronograf/1-6-faq-cluster-connection.png) + +Note that the example above assumes that you do not have authentication enabled. +If you have authentication enabled, the form requires username and password information. +For more details about monitoring an InfluxDB Enterprise cluster, see the [Monitor an InfluxDB Enterprise Cluster](/chronograf/v1.9/guides/monitoring-influxenterprise-clusters/) guide. diff --git a/content/enterprise_influxdb/v1.5/administration/backup-and-restore.md b/content/enterprise_influxdb/v1.5/administration/backup-and-restore.md index a9b100a80..80d210e80 100644 --- a/content/enterprise_influxdb/v1.5/administration/backup-and-restore.md +++ b/content/enterprise_influxdb/v1.5/administration/backup-and-restore.md @@ -18,8 +18,8 @@ The primary use cases for backup and restore are: * Debugging * Restoring clusters to a consistent state -InfluxDB Enterprise supports backing up and restoring data in a cluster, a single database, a single database and retention policy, and -single [shard](/influxdb/v1.5/concepts/glossary/#shard). +InfluxDB Enterprise supports backing up and restoring data in a cluster, +a single database and retention policy, and single [shard](/influxdb/v1.5/concepts/glossary/#shard). > **Note:** You can use the [new `backup` and `restore` utilities in InfluxDB OSS 1.5](/influxdb/v1.5/administration/backup_and_restore/) to: > * Restore InfluxDB Enterprise 1.5 backup files to InfluxDB OSS 1.5. diff --git a/content/enterprise_influxdb/v1.6/administration/backup-and-restore.md b/content/enterprise_influxdb/v1.6/administration/backup-and-restore.md index 9bc7436b6..abc1b87b6 100644 --- a/content/enterprise_influxdb/v1.6/administration/backup-and-restore.md +++ b/content/enterprise_influxdb/v1.6/administration/backup-and-restore.md @@ -17,8 +17,8 @@ The primary use cases for backup and restore are: * Debugging * Restoring clusters to a consistent state -InfluxDB Enterprise supports backing up and restoring data in a cluster, a single database, a single database and retention policy, and -single [shard](/influxdb/v1.6/concepts/glossary/#shard). +InfluxDB Enterprise supports backing up and restoring data in a cluster, +a single database and retention policy, and single [shard](/influxdb/v1.6/concepts/glossary/#shard). > **Note:** You can use the [new `backup` and `restore` utilities in InfluxDB OSS 1.5](/influxdb/v1.5/administration/backup_and_restore/) to: > * Restore InfluxDB Enterprise 1.5 backup files to InfluxDB OSS 1.5. diff --git a/content/enterprise_influxdb/v1.8/about-the-project/release-notes-changelog.md b/content/enterprise_influxdb/v1.8/about-the-project/release-notes-changelog.md index 08ef65830..690fde078 100644 --- a/content/enterprise_influxdb/v1.8/about-the-project/release-notes-changelog.md +++ b/content/enterprise_influxdb/v1.8/about-the-project/release-notes-changelog.md @@ -12,7 +12,11 @@ menu: ## v1.8.6 [2021-05-21] {{% warn %}} -**Fine-grained authorization security update.** If you're on InfluxDB Enterprise {{< latest-patch >}}, we recommend immediately upgrading to this release. An issue was reported in {{< latest-patch >}} where grants with specified permissions for users were not enforced. Versions prior to InfluxDB Enterprise {{< latest-patch >}} are not affected. This security update ensures that only users with sufficient permissions can read and write to a measurement. +**Fine-grained authorization security update.** +If using **InfluxDB Enterprise 1.8.5**, we strongly recommend upgrading to **InfluxDB Enterprise 1.8.6** immediately. +1.8.5 does not correctly enforce grants with specified permissions for users. +Versions prior to InfluxDB Enterprise 1.8.5 are not affected. +1.8.6 ensures that only users with sufficient permissions can read and write to a measurement. {{% /warn %}} ### Features @@ -32,9 +36,9 @@ menu: - Previously, the Anti-Entropy service would loop trying to copy an empty shard to a data node missing that shard. Now, an empty shard is successfully created on a new node. - Check for previously ignored errors in `DiffIterator.Next()`. Update to check before possible function exit and ensure handles are closed on error in digest diffs. -## v{{< latest-patch >}} [2020-04-20] +## v1.8.5 [2020-04-20] -The InfluxDB Enterprise {{< latest-patch >}} release builds on the InfluxDB OSS {{< latest-patch >}} release. +The InfluxDB Enterprise v1.8.5 release builds on the InfluxDB OSS v1.8.5 release. For details on changes incorporated from the InfluxDB OSS release, see [InfluxDB OSS release notes](/influxdb/v1.8/about_the_project/releasenotes-changelog/#v185-2021-04-20). diff --git a/content/enterprise_influxdb/v1.8/administration/backup-and-restore.md b/content/enterprise_influxdb/v1.8/administration/backup-and-restore.md index b2a42498f..e7e4c7457 100644 --- a/content/enterprise_influxdb/v1.8/administration/backup-and-restore.md +++ b/content/enterprise_influxdb/v1.8/administration/backup-and-restore.md @@ -34,9 +34,13 @@ Depending on the volume of data to be protected and your application requirement ## Backup and restore utilities -InfluxDB Enterprise supports backing up and restoring data in a cluster, a single database, a single database and retention policy, and single shards. Most InfluxDB Enterprise applications can use the backup and restore utilities. +InfluxDB Enterprise supports backing up and restoring data in a cluster, +a single database and retention policy, and single shards. +Most InfluxDB Enterprise applications can use the backup and restore utilities. -Use the `backup` and `restore` utilities to back up and restore between `influxd` instances with the same versions or with only minor version differences. For example, you can backup from 1.7.3 and restore on 1.8.6. +Use the `backup` and `restore` utilities to back up and restore between `influxd` +instances with the same versions or with only minor version differences. +For example, you can backup from 1.7.3 and restore on 1.8.6. ### Backup utility diff --git a/content/enterprise_influxdb/v1.9/_index.md b/content/enterprise_influxdb/v1.9/_index.md new file mode 100644 index 000000000..ba842d182 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/_index.md @@ -0,0 +1,41 @@ +--- +title: InfluxDB Enterprise 1.9 documentation +description: > + Documentation for InfluxDB Enterprise, which adds clustering, high availability, fine-grained authorization, and more to InfluxDB OSS. +aliases: + - /enterprise/v1.9/ +menu: + enterprise_influxdb_1_9: + name: InfluxDB Enterprise v1.9 +weight: 1 +--- + +InfluxDB Enterprise provides a time series database designed to handle high write and query loads and offers highly scalable clusters on your infrastructure with a management UI. Use for DevOps monitoring, IoT sensor data, and real-time analytics. Check out the key features that make InfluxDB Enterprise a great choice for working with time series data. + +If you're interested in working with InfluxDB Enterprise, visit +[InfluxPortal](https://portal.influxdata.com/) to sign up, get a license key, +and get started! + +## Key features + +- High performance datastore written specifically for time series data. High ingest speed and data compression. +- Provides high availability across your cluster and eliminates a single point of failure. +- Written entirely in Go. Compiles into a single binary with no external dependencies. +- Simple, high performing write and query HTTP APIs. +- Plugin support for other data ingestion protocols such as Graphite, collectd, and OpenTSDB. +- Expressive SQL-like query language tailored to easily query aggregated data. +- Continuous queries automatically compute aggregate data to make frequent queries more efficient. +- Tags let you index series for fast and efficient queries. +- Retention policies efficiently auto-expire stale data. + +## Next steps + +- [Install and deploy](/enterprise_influxdb/v1.9/install-and-deploy/) +- Review key [concepts](/enterprise_influxdb/v1.9/concepts/) +- [Get started](/enterprise_influxdb/v1.9/introduction/getting-started/) + + diff --git a/content/enterprise_influxdb/v1.9/about-the-project/_index.md b/content/enterprise_influxdb/v1.9/about-the-project/_index.md new file mode 100644 index 000000000..0a7940ba1 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/about-the-project/_index.md @@ -0,0 +1,14 @@ +--- +title: About the project +description: > + Release notes, licenses, and third-party software details for InfluxDB Enterprise. +menu: + enterprise_influxdb_1_9_ref: + weight: 10 +--- + +{{< children hlevel="h2" >}} + +## Commercial license + +InfluxDB Enterprise is available with a commercial license. [Contact sales for more information](https://www.influxdata.com/contact-sales/). diff --git a/content/enterprise_influxdb/v1.9/about-the-project/release-notes-changelog.md b/content/enterprise_influxdb/v1.9/about-the-project/release-notes-changelog.md new file mode 100644 index 000000000..090e7a45b --- /dev/null +++ b/content/enterprise_influxdb/v1.9/about-the-project/release-notes-changelog.md @@ -0,0 +1,971 @@ +--- +title: InfluxDB Enterprise 1.9 release notes +description: > + Important changes and what's new in each version InfluxDB Enterprise. +menu: + enterprise_influxdb_1_9_ref: + name: Release notes + weight: 10 + parent: About the project +--- + +## v1.9.2 [2021-06-17] + +The release of InfluxDB Enterprise 1.9 is different from previous InfluxDB Enterprise releases +in that there is no corresponding InfluxDB OSS release. +(InfluxDB 1.8.x will continue to receive maintenance updates.) + +### Features +- Upgrade to Go 1.15.10. +- Support user-defined *node labels*. + Node labels let you assign arbitrary key-value pairs to meta and data nodes in a cluster. + For instance, an operator might want to label nodes with the availability zone in which they're located. +- Improve performance of `SHOW SERIES CARDINALITY` and `SHOW SERIES CARDINALITY from ` InfluxQL queries. + These queries now return a `cardinality estimation` column header where before they returned `count`. +- Improve diagnostics for license problems. + Add [license expiration date](/enterprise_influxdb/v1.9/features/clustering-features/#entitlements) to `debug/vars` metrics. +- Add improved [ingress metrics](/enterprise_influxdb/v1.9/administration/config-data-nodes/#ingress-metric-by-measurement-enabled--false) to track points written by measurement and by login. + Allow for collection of statistics regarding points, values, and new series written per measurement and by login. + This data is collected and exposed at the data node level. + With these metrics you can, for example: + aggregate the write requests across the entire cluster, + monitor the growth of series within a measurement, + and track what user credentials are being used to write data. +- Support authentication for Kapacitor via LDAP. +- Support for [configuring Flux query resource usage](/enterprise_influxdb/v1.9/administration/config-data-nodes/#flux-controller) (concurrency, memory, etc.). +- Upgrade to [Flux v0.113.0](/influxdb/v2.0/reference/release-notes/flux/#v01130-2021-04-21). +- Update Prometheus remote protocol to allow streamed reading. +- Improve performance of sorted merge iterator. +- Add arguments to Flux `to` function. +- Add meancount aggregation for WindowAggregate pushdown. +- Optimize series iteration in TSI. +- Add `WITH KEY` to `SHOW TAG KEYS`. + +### Bug fixes +- `show databases` now checks read and write permissions. +- Anti-entropy: Update `tsm1.BlockCount()` call to match signature. +- Remove extraneous nil check from points writer. +- Ensure a newline is printed after a successful copy during [restoration](/enterprise_influxdb/v1.9/administration/backup-and-restore/). +- Make `entropy show` expiration times consistent with `show-shards`. +- Properly shutdown multiple HTTP servers. +- Allow CORS in v2 compatibility endpoints. +- Address staticcheck warnings SA4006, ST1006, S1039, and S1020. +- Fix Anti-Entropy looping endlessly with empty shard. +- Disable MergeFiltersRule until it is more stable. +- Fix data race and validation in cache ring. +- Return error on nonexistent shard ID. +- Add `User-Agent` to allowed CORS headers. +- Fix variables masked by a declaration. +- Fix key collisions when serializing `/debug/vars`. +- Fix temporary directory search bug. +- Grow tag index buffer if needed. +- Use native type for summation in new meancount iterator. +- Fix consistent error for missing shard. +- Properly read payload in `snapshotter`. +- Fix help text for `influx_inspect`. +- Allow `PATCH` in CORS. +- Fix `GROUP BY` returning multiple results per group in some circumstances. +- Add option to authenticate Prometheus remote read. +- Fix FGA enablement. +- Fix "snapshot in progress" error during backup. +- Fix cursor requests (`[start, stop]` instead of `[start, stop)`). +- Exclude stop time from array cursors. +- Fix Flux regression in buckets query. +- Fix redundant registration for Prometheus collector metrics. +- Re-add Flux CLI. +- Use non-nil `context.Context` value in client. + +### Other changes + +- Remove `influx_stress` tool (deprecated since version 1.2). + Instead, use [`inch`](https://github.com/influxdata/inch) + or [`influx-stress`](https://github.com/influxdata/influx-stress) (not to be confused with `influx_stress`). + +{{% note %}} +**Note:** InfluxDB Enterprise 1.9.0 and 1.9.1 were not released. +Bug fixes intended for 1.9.0 and 1.9.1 were rolled into InfluxDB Enterprise 1.9.2. +{{% /note %}} + +## v1.8.6 [2021-05-21] + +{{% warn %}} +**Fine-grained authorization security update.** +If using **InfluxDB Enterprise 1.8.5**, we strongly recommend upgrading to **InfluxDB Enterprise 1.8.6** immediately. +1.8.5 does not correctly enforce grants with specified permissions for users. +Versions prior to InfluxDB Enterprise 1.8.5 are not affected. +1.8.6 ensures that only users with sufficient permissions can read and write to a measurement. +{{% /warn %}} + +### Features + +- **Enhanced Anti-Entropy (AE) logging**: When the [debug logging level](/enterprise_influxdb/v1.8/administration/config-data-nodes/#logging-settings) is set (`level="debug"`) in the data node configuration, the Anti-Entropy service reports reasons a shard is not idle, including: + - active Cache compactions + - active Level (Zero, One, Two) compactions + - active Full compactions + - active TSM Optimization compactions + - cache size is nonzero + - shard is not fully compacted +- **Enhanced `copy-shard` logging**. Add information to log messages in `copy-shard` functions and additional error tests. + +### Bug fixes + +- Use the proper TLS configuration when a meta node makes an remote procedure call (RPC) to a data node. Addresses RPC call issues using the following influxd-ctl commands: `copy-shard` `copy-shard-status` `kill-copy-shard` `remove-shard` +- Previously, the Anti-Entropy service would loop trying to copy an empty shard to a data node missing that shard. Now, an empty shard is successfully created on a new node. +- Check for previously ignored errors in `DiffIterator.Next()`. Update to check before possible function exit and ensure handles are closed on error in digest diffs. + +## v1.8.5 [2020-04-20] + +The InfluxDB Enterprise v1.8.5 release builds on the InfluxDB OSS v1.8.5 release. +For details on changes incorporated from the InfluxDB OSS release, see +[InfluxDB OSS release notes](/influxdb/v1.8/about_the_project/releasenotes-changelog/#v185-2021-04-20). + +### Bug fixes + +- Resolve TSM backup "snapshot in progress" error. +- SHOW DATABASES now only shows databases that the user has either read or write access to +- `influxd_ctl entropy show` now shows shard expiry times consistent with `influxd_ctl show-shards` +- Add labels to the values returned in SHOW SHARDS output to clarify the node ID and TCP address. +- Always forward repairs to the next data node (even if the current data node does not have to take action for the repair). + +## v1.8.4 [2020-02-08] + +The InfluxDB Enterprise 1.8.4 release builds on the InfluxDB OSS 1.8.4 release. +For details on changes incorporated from the InfluxDB OSS release, see +[InfluxDB OSS release notes](/influxdb/v1.8/about_the_project/releasenotes-changelog/#v1-8-4-unreleased). + + > **Note:** InfluxDB Enterprise 1.8.3 was not released. Bug fixes intended for 1.8.3 were rolled into InfluxDB Enterprise 1.8.4. + +### Features + +#### Update your InfluxDB Enterprise license without restarting data nodes + +Add the ability to [renew or update your license key or file](/enterprise_influxdb/v1.8/administration/renew-license/) without restarting data nodes. +### Bug fixes + +- Wrap TCP mux–based HTTP server with a function that adds custom headers. +- Correct output for `influxd-ctl show shards`. +- Properly encode/decode `control.Shard.Err`. + +## v1.8.2 [2020-08-24] + +The InfluxDB Enterprise 1.8.2 release builds on the InfluxDB OSS 1.8.2 and 1.8.1 releases. +Due to a defect in InfluxDB OSS 1.8.1, InfluxDB Enterprise 1.8.1 was not released. +This release resolves the defect and includes the features and bug fixes listed below. +For details on changes incorporated from the InfluxDB OSS release, see +[InfluxDB OSS release notes](/influxdb/v1.8/about_the_project/releasenotes-changelog/). + +### Features + +#### Hinted handoff improvements + +- Allow out-of-order writes. This change adds a configuration option `allow-out-of-order-writes` to the `[cluster]` section of the data node configuration file. This setting defaults to `false` to match the existing behavior. There are some important operational considerations to review before turning this on. But, the result is enabling this option reduces the time required to drain the hinted handoff queue and increase throughput during recovery. See [allow-out-of-order-writes](/enterprise_influxdb/v1.8/administration/config-data-nodes#allow-out-of-order-false) for more detail. +- Make the number of pending writes configurable. This change adds a configuration option in the `[hinted-handoff]` section called `max-pending-writes`, which defaults to `1024`. See [max-pending-writes](/enterprise_influxdb/v1.8/administration/config-data-nodes#max-pending-writes-1024) for more detail. +- Update the hinted handoff queue to ensure various entries to segment files occur atomically. Prior to this change, entries were written to disk in three separate writes (len, data, offset). If the process stopped in the middle of any of those writes, the hinted handoff segment file was left in an invalid state. +- In certain scenarios, the hinted-handoff queue would fail to drain. Upon node startup, the queue segment files are now verified and truncated if any are corrupted. Some additional logging has been added when a node starts writing to the hinted handoff queue as well. + +#### `influxd-ctl` CLI improvements + +- Add a verbose flag to [`influxd-ctl show-shards`](/enterprise_influxdb/v1.8/administration/cluster-commands/#show-shards). This option provides more information about each shard owner, including the state (hot/cold), last modified date and time, and size on disk. + +### Bug fixes + +- Resolve a cluster read service issue that caused a panic. Previously, if no tags keys or values were read, the cluster read service returned a nil cursor. Now, an empty cursor is returned. +- LDAP configuration: `GroupSearchBaseDNs`, `SearchFilter`, `GroupMembershipSearchFilter`, and `GroupSearchFilter` values in the LDAP section of the configuration file are now all escaped. +- Eliminate orphaned, temporary directories when an error occurs during `processCreateShardSnapshotRequest()` and provide useful log information regarding the reason a temporary directory is created. + +## v1.8 [2020-04-27] + +The InfluxDB Enterprise 1.8 release builds on the InfluxDB OSS 1.8 release. +For details on changes incorporated from the InfluxDB OSS release, see +[InfluxDB OSS release notes](/influxdb/v1.8/about_the_project/releasenotes-changelog/). + +### Features + +#### **Back up meta data only** + +- Add option to back up **meta data only** (users, roles, databases, continuous queries, and retention policies) using the new `-strategy` flag and `only meta` option: `influx ctl backup -strategy only meta `. + + > **Note:** To restore a meta data backup, use the `restore -full` command and specify your backup manifest: `influxd-ctl restore -full `. + +For more information, see [Perform a metastore only backup](/enterprise_influxdb/v1.8/administration/backup-and-restore/#perform-a-metastore-only-backup). + +#### **Incremental and full backups** + +- Add `incremental` and `full` backup options to the new `-strategy` flag in `influx ctl backup`: + - `influx ctl backup -strategy incremental` + - `influx ctl backup -strategy full` + +For more information, see the [`influxd-ctl backup` syntax](/enterprise_influxdb/v1.8/administration/backup-and-restore/#syntax). + +### Bug fixes + +- Update the Anti-Entropy (AE) service to ignore expired shards. + +## v1.7.10 [2020-02-07] + +The InfluxDB Enterprise 1.7.10 release builds on the InfluxDB OSS 1.7.10 release. +For details on changes incorporated from the InfluxDB OSS release, see +[InfluxDB OSS release notes](/influxdb/v1.7/about_the_project/releasenotes-changelog/). + +### Features +- Log when meta state file cannot be opened. + +### Bugfixes +- Update `MaxShardGroupID` on meta update. +- Don't reassign shard ownership when removing a data node. + +## v1.7.9 [2019-10-27] + +The InfluxDB Enterprise 1.7.9 release builds on the InfluxDB OSS 1.7.9 release. +For details on changes incorporated from the InfluxDB OSS release, see +[InfluxDB OSS release notes](/influxdb/v1.7/about_the_project/releasenotes-changelog/). + +### Release notes +- This release is built using Go 1.12.10 which eliminates the + [HTTP desync vulnerability](https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn). + +### Bug fixes +- Move `tsdb store open` to beginning of server initialization. +- Enable Meta client and Raft to use verified TLS. +- Fix RPC pool TLS configuration. +- Update example configuration file with new authorization options. + +## 1.7.8 [2019-09-03] + +{{% warn %}} +InfluxDB now rejects all non-UTF-8 characters. +To successfully write data to InfluxDB, use only UTF-8 characters in +database names, measurement names, tag sets, and field sets. +InfluxDB Enterprise customers can contact InfluxData support for more information. +{{% /warn %}} + +The InfluxDB Enterprise 1.7.8 release builds on the InfluxDB OSS 1.7.8 release. +For details on changes incorporated from the InfluxDB OSS release, see [InfluxDB OSS release notes](/influxdb/v1.7/about_the_project/releasenotes-changelog/). + +### Bug fixes +- Clarified `influxd-ctl` error message when the Anti-Entropy (AE) service is disabled. +- Ensure invalid, non-UTF-8 data is removed from hinted handoff. +- Added error messages for `INFLUXDB_LOGGING_LEVEL` if misconfigured. +- Added logging when data nodes connect to meta service. + +### Features +- The Flux Technical Preview has advanced to version [0.36.2](/flux/v0.36/). + +## 1.7.7 [2019-07-12] + +The InfluxDB Enterprise 1.7.7 release builds on the InfluxDB OSS 1.7.7 release. For details on changes incorporated from the InfluxDB OSS release, see [InfluxDB OSS release notes](/influxdb/v1.7/about_the_project/releasenotes-changelog/). + +### Known issues + +- The Flux Technical Preview was not advanced and remains at version 0.24.0. Next month's maintenance release will update the preview. +- After upgrading, customers have experienced an excessively large output additional lines due to a `Println` statement introduced in this release. For a possible workaround, see https://github.com/influxdata/influxdb/issues/14265#issuecomment-508875853. Next month's maintenance release will address this issue. + +### Features + +- Adds TLS to RPC calls. If verifying certificates, uses the TLS setting in the configuration passed in with -config. + +### Bug fixes + +- Ensure retry-rate-limit configuration value is used for hinted handoff. +- Always forward AE repair to next node. +- Improve hinted handoff metrics. + +## 1.7.6 [2019-05-07] + +This InfluxDB Enterprise release builds on the InfluxDB OSS 1.7.6 release. For details on changes incorporated from the InfluxDB OSS release, see [InfluxDB OSS release notes](/influxdb/v1.7/about_the_project/releasenotes-changelog/). + +### Bug fixes + +- Reverts v1.7.5 InfluxQL regressions that removed parentheses and resulted in operator precedence causing changing results in complex queries and regular expressions. + +## 1.7.5 [2019-03-26] + +{{% warn %}} + +**If you are currently on this release, roll back to v1.7.4 until a fix is available.** + +After upgrading to this release, some customers have experienced regressions, +including parentheses being removed resulting in operator precedence causing changing results +in complex queries and regular expressions. + +Examples: + +- Complex WHERE clauses with parentheses. For example, `WHERE d > 100 AND (c = 'foo' OR v = 'bar'`). +- Conditions not including parentheses caysubg operator precedence to return `(a AND b) OR c` instead of `a AND (b OR c)` + +{{% /warn %}} + +This InfluxDB Enterprise release builds on the InfluxDB OSS 1.7.5 release. For details on changes incorporated from the InfluxDB OSS release, see [InfluxDB OSS release notes](/influxdb/v1.7/about_the_project/releasenotes-changelog/). + +### Features + +- Add `influx_tools` utility (for internal support use) to be part of the packaging. + +### Bug fixes + +- Anti-Entropy: fix `contains no .tsm files` error. +- `fix(cluster)`: account for nil result set when writing read response. + +## 1.7.4 [2019-02-13] + +This InfluxDB Enterprise release builds on the InfluxDB OSS 1.7.4 release. For details on changes incorporated from the InfluxDB OSS release, see [InfluxDB OSS release notes](/influxdb/v1.7/about_the_project/releasenotes-changelog/). + +### Bug fixes + +- Use `systemd` for Amazon Linux 2. + +## 1.7.3 [2019-01-11] + +This InfluxDB Enterprise release builds on the InfluxDB OSS 1.7.3 release. For details on changes incorporated from the InfluxDB OSS release, see the [InfluxDB OSS release notes](/influxdb/v1.7/about_the_project/releasenotes-changelog/). + +### Important update [2019-02-13] + +If you have not installed this release, then install the 1.7.4 release. + +**If you are currently running this release, then upgrade to the 1.7.4 release as soon as possible.** + +- A critical defect in the InfluxDB 1.7.3 release was discovered and our engineering team fixed the issue in the 1.7.4 release. Out of high concern for your data and projects, upgrade to the 1.7.4 release as soon as possible. + - **Critical defect:** Shards larger than 16GB are at high risk for data loss during full compaction. The full compaction process runs when a shard go "cold" – no new data is being written into the database during the time range specified by the shard. + - **Post-mortem analysis:** InfluxData engineering is performing a post-mortem analysis to determine how this defect was introduced. Their discoveries will be shared in a blog post. + +- A small percentage of customers experienced data node crashes with segmentation violation errors. We fixed this issue in 1.7.4. + +### Breaking changes +- Fix invalid UTF-8 bytes preventing shard opening. Treat fields and measurements as raw bytes. + +### Features + +#### Anti-entropy service disabled by default + +Prior to v.1.7.3, the anti-entropy (AE) service was enabled by default. When shards create large digests with lots of time ranges (10s of thousands), some customers experienced significant performance issues, including CPU usage spikes. If your shards include a small number of time ranges (most have 1 to 10, some have up to several hundreds) and you can benefit from the AE service, then you can enable AE and watch to see if performance is significantly impacted. + +- Add user authentication and authorization support for Flux HTTP requests. +- Add support for optionally logging Flux queries. +- Add support for LDAP StartTLS. +- Flux 0.7 support. +- Implement TLS between data nodes. +- Update to Flux 0.7.1. +- Add optional TLS support to meta node Raft port. +- Anti-Entropy: memoize `DistinctCount`, `min`, & `max` time. +- Update influxdb dep for subquery auth update. + +### Bug fixes + +- Update sample configuration. + +## 1.6.6 [2019-02-28] +------------------- + +This release only includes the InfluxDB OSS 1.6.6 changes (no Enterprise-specific changes). + +## 1.6.5 [2019-01-10] + +This release builds off of the InfluxDB OSS 1.6.0 through 1.6.5 releases. For details about changes incorporated from InfluxDB OSS releases, see [InfluxDB OSS release notes](/influxdb/v1.7/about_the_project/releasenotes-changelog/). + +## 1.6.4 [2018-10-23] + +This release builds off of the InfluxDB OSS 1.6.0 through 1.6.4 releases. For details about changes incorporated from InfluxDB OSS releases, see the [InfluxDB OSS release notes](/influxdb/v1.7/about_the_project/releasenotes-changelog/). + +### Breaking changes + +#### Require `internal-shared-secret` if meta auth enabled + +If `[meta] auth-enabled` is set to `true`, the `[meta] internal-shared-secret` value must be set in the configuration. +If it is not set, an error will be logged and `influxd-meta` will not start. + +* Previously, authentication could be enabled without setting an `internal-shared-secret`. The security risk was that an unset (empty) value could be used for the `internal-shared-secret`, seriously weakening the JWT authentication used for internode communication. + +#### Review production installation configurations + +The [Production Installation](/enterprise_influxdb/v1.7/production_installation/) +documentation has been updated to fix errors in configuration settings, including changing `shared-secret` to `internal-shared-secret` and adding missing steps for configuration settings of data nodes and meta nodes. All Enterprise users should review their current configurations to ensure that the configuration settings properly enable JWT authentication for internode communication. + +The following summarizes the expected settings for proper configuration of JWT authentication for internode communication: + +##### Data node configuration files (`influxdb.conf`) + +**[http] section** + +* `auth-enabled = true` + - Enables authentication. Default value is false. + +**[meta] section** + +- `meta-auth-enabled = true` + - Must match for meta nodes' `[meta] auth-enabled` settings. +- `meta-internal-shared-secret = ""` + - Must be the same pass phrase on all meta nodes' `[meta] internal-shared-secret` settings. + - Used by the internal API for JWT authentication. Default value is `""`. + - A long pass phrase is recommended for stronger security. + +##### Meta node configuration files (`meta-influxdb.conf`) + +**[meta]** section + +- `auth-enabled = true` + - Enables authentication. Default value is `false` . +- `internal-shared-secret = ""` + - Must same pass phrase on all data nodes' `[meta] meta-internal-shared-secret` + settings. + - Used by the internal API for JWT authentication. Default value is +`""`. + - A long pass phrase is recommended for better security. + +>**Note:** To provide encrypted internode communication, you must enable HTTPS. Although the JWT signature is encrypted, the the payload of a JWT token is encoded, but is not encrypted. + +### Bug fixes + +- Only map shards that are reported ready. +- Fix data race when shards are deleted and created concurrently. +- Reject `influxd-ctl update-data` from one existing host to another. +- Require `internal-shared-secret` if meta auth enabled. + +## 1.6.2 [08-27-2018] + +This release builds off of the InfluxDB OSS 1.6.0 through 1.6.2 releases. For details about changes incorporated from InfluxDB OSS releases, see the [InfluxDB OSS release notes](/influxdb/v1.7/about_the_project/releasenotes-changelog/). + +### Features + +- Update Go runtime to `1.10`. +- Provide configurable TLS security options. +- Add LDAP functionality for authorization and authentication. +- Anti-Entropy (AE): add ability to repair shards. +- Anti-Entropy (AE): improve swagger doc for `/status` endpoint. +- Include the query task status in the show queries output. + +#### Bug fixes + +- TSM files not closed when shard is deleted. +- Ensure shards are not queued to copy if a remote node is unavailable. +- Ensure the hinted handoff (hh) queue makes forward progress when segment errors occur. +- Add hinted handoff (hh) queue back pressure. + +## 1.5.4 [2018-06-21] + +This release builds off of the InfluxDB OSS 1.5.4 release. Please see the [InfluxDB OSS release notes](/influxdb/v1.5/about_the_project/releasenotes-changelog/) for more information about the InfluxDB OSS release. + +## 1.5.3 [2018-05-25] + +This release builds off of the InfluxDB OSS 1.5.3 release. Please see the [InfluxDB OSS release notes](/influxdb/v1.5/about_the_project/releasenotes-changelog/) for more information about the InfluxDB OSS release. + +### Features + +* Include the query task status in the show queries output. +* Add hh writeBlocked counter. + +### Bug fixes + +* Hinted-handoff: enforce max queue size per peer node. +* TSM files not closed when shard deleted. + + +## v1.5.2 [2018-04-12] + +This release builds off of the InfluxDB OSS 1.5.2 release. Please see the [InfluxDB OSS release notes](/influxdb/v1.5/about_the_project/releasenotes-changelog/) for more information about the InfluxDB OSS release. + +### Bug fixes + +* Running backup snapshot with client's retryWithBackoff function. +* Ensure that conditions are encoded correctly even if the AST is not properly formed. + +## v1.5.1 [2018-03-20] + +This release builds off of the InfluxDB OSS 1.5.1 release. There are no Enterprise-specific changes. +Please see the [InfluxDB OSS release notes](/influxdb/v1.7/about_the_project/releasenotes-changelog/) for more information about the InfluxDB OSS release. + +## v1.5.0 [2018-03-06] + +> ***Note:*** This release builds off of the 1.5 release of InfluxDB OSS. Please see the [InfluxDB OSS release +> notes](/influxdb/v1.5/about_the_project/releasenotes-changelog/) for more information about the InfluxDB OSS release. + +For highlights of the InfluxDB 1.5 release, see [What's new in InfluxDB 1.5](/influxdb/v1.5/about_the_project/whats_new/). + +### Breaking changes + +The default logging format has been changed. See [Logging and tracing in InfluxDB](/influxdb/v1.6/administration/logs/) for details. + +### Features + +* Add `LastModified` fields to shard RPC calls. +* As of OSS 1.5 backup/restore interoperability is confirmed. +* Make InfluxDB Enterprise use OSS digests. +* Move digest to its own package. +* Implement distributed cardinality estimation. +* Add logging configuration to the configuration files. +* Add AE `/repair` endpoint and update Swagger doc. +* Update logging calls to take advantage of structured logging. +* Use actual URL when logging anonymous stats start. +* Fix auth failures on backup/restore. +* Add support for passive nodes +* Implement explain plan for remote nodes. +* Add message pack format for query responses. +* Teach show tag values to respect FGA +* Address deadlock in meta server on 1.3.6 +* Add time support to `SHOW TAG VALUES` +* Add distributed `SHOW TAG KEYS` with time support + +### Bug fixes + +* Fix errors occurring when policy or shard keys are missing from the manifest when limited is set to true. +* Fix spurious `rpc error: i/o deadline exceeded` errors. +* Elide `stream closed` error from logs and handle `io.EOF` as remote iterator interrupt. +* Discard remote iterators that label their type as unknown. +* Do not queue partial write errors to hinted handoff. +* Segfault in `digest.merge` +* Meta Node CPU pegged on idle cluster. +* Data race on `(meta.UserInfo).acl)` +* Fix wildcard when one shard has no data for a measurement with partial replication. +* Add `X-Influxdb-Build` to http response headers so users can identify if a response is from an InfluxDB OSS or InfluxDB Enterprise service. +* Ensure that permissions cannot be set on non-existent databases. +* Switch back to using `cluster-tracing` config option to enable meta HTTP request logging. +* `influxd-ctl restore -newdb` can't restore data. +* Close connection for remote iterators after EOF to avoid writer hanging indefinitely. +* Data race reading `Len()` in connection pool. +* Use InfluxData fork of `yamux`. This update reduces overall memory usage when streaming large amounts of data. +* Fix group by marshaling in the IteratorOptions. +* Meta service data race. +* Read for the interrupt signal from the stream before creating the iterators. +* Show retention policies requires the `createdatabase` permission +* Handle UTF files with a byte order mark when reading the configuration files. +* Remove the pidfile after the server has exited. +* Resend authentication credentials on redirect. +* Updated yamux resolves race condition when SYN is successfully sent and a write timeout occurs. +* Fix no license message. + +## v1.3.9 [2018-01-19] + +### Upgrading -- for users of the TSI preview + +If you have been using the TSI preview with 1.3.6 or earlier 1.3.x releases, you will need to follow the upgrade steps to continue using the TSI preview. Unfortunately, these steps cannot be executed while the cluster is operating -- +so it will require downtime. + +### Bugfixes + +* Elide `stream closed` error from logs and handle `io.EOF` as remote iterator interrupt. +* Fix spurious `rpc error: i/o deadline exceeded` errors +* Discard remote iterators that label their type as unknown. +* Do not queue `partial write` errors to hinted handoff. + +## v1.3.8 [2017-12-04] + +### Upgrading -- for users of the TSI preview + +If you have been using the TSI preview with 1.3.6 or earlier 1.3.x releases, you will need to follow the upgrade steps to continue using the TSI preview. Unfortunately, these steps cannot be executed while the cluster is operating -- so it will require downtime. + +### Bugfixes + +- Updated `yamux` resolves race condition when SYN is successfully sent and a write timeout occurs. +- Resend authentication credentials on redirect. +- Fix wildcard when one shard has no data for a measurement with partial replication. +- Fix spurious `rpc error: i/o deadline exceeded` errors. + +## v1.3.7 [2017-10-26] + +### Upgrading -- for users of the TSI preview + +The 1.3.7 release resolves a defect that created duplicate tag values in TSI indexes See Issues +[#8995](https://github.com/influxdata/influxdb/pull/8995), and [#8998](https://github.com/influxdata/influxdb/pull/8998). +However, upgrading to 1.3.7 cause compactions to fail, see [Issue #9025](https://github.com/influxdata/influxdb/issues/9025). +We will provide a utility that will allow TSI indexes to be rebuilt, +resolving the corruption possible in releases prior to 1.3.7. If you are using the TSI preview, +**you should not upgrade to 1.3.7 until this utility is available**. +We will update this release note with operational steps once the utility is available. + +#### Bugfixes + + - Read for the interrupt signal from the stream before creating the iterators. + - Address Deadlock issue in meta server on 1.3.6 + - Fix logger panic associated with anti-entropy service and manually removed shards. + +## v1.3.6 [2017-09-28] + +### Bugfixes + +- Fix "group by" marshaling in the IteratorOptions. +- Address meta service data race condition. +- Fix race condition when writing points to remote nodes. +- Use InfluxData fork of yamux. This update reduces overall memory usage when streaming large amounts of data. + Contributed back to the yamux project via: https://github.com/hashicorp/yamux/pull/50 +- Address data race reading Len() in connection pool. + +## v1.3.5 [2017-08-29] + +This release builds off of the 1.3.5 release of OSS InfluxDB. +Please see the OSS [release notes](/influxdb/v1.3/about_the_project/releasenotes-changelog/#v1-3-5-2017-08-29) for more information about the OSS releases. + +## v1.3.4 [2017-08-23] + +This release builds off of the 1.3.4 release of OSS InfluxDB. Please see the [OSS release notes](/influxdb/v1.3/about_the_project/releasenotes-changelog/) for more information about the OSS releases. + +### Bugfixes + +- Close connection for remote iterators after EOF to avoid writer hanging indefinitely + +## v1.3.3 [2017-08-10] + +This release builds off of the 1.3.3 release of OSS InfluxDB. Please see the [OSS release notes](/influxdb/v1.3/about_the_project/releasenotes-changelog/) for more information about the OSS releases. + +### Bugfixes + +- Connections are not closed when `CreateRemoteIterator` RPC returns no iterators, resolved memory leak + +## v1.3.2 [2017-08-04] + +### Bug fixes + +- `influxd-ctl restore -newdb` unable to restore data. +- Improve performance of `SHOW TAG VALUES`. +- Show a subset of config settings in `SHOW DIAGNOSTICS`. +- Switch back to using cluster-tracing config option to enable meta HTTP request logging. +- Fix remove-data error. + +## v1.3.1 [2017-07-20] + +#### Bug fixes + +- Show a subset of config settings in SHOW DIAGNOSTICS. +- Switch back to using cluster-tracing config option to enable meta HTTP request logging. +- Fix remove-data error. + +## v1.3.0 [2017-06-21] + +### Configuration Changes + +#### `[cluster]` Section + +* `max-remote-write-connections` is deprecated and can be removed. +* NEW: `pool-max-idle-streams` and `pool-max-idle-time` configure the RPC connection pool. + See `config.sample.toml` for descriptions of these new options. + +### Removals + +The admin UI is removed and unusable in this release. The `[admin]` configuration section will be ignored. + +#### Features + +- Allow non-admin users to execute SHOW DATABASES +- Add default config path search for influxd-meta. +- Reduce cost of admin user check for clusters with large numbers of users. +- Store HH segments by node and shard +- Remove references to the admin console. +- Refactor RPC connection pool to multiplex multiple streams over single connection. +- Report RPC connection pool statistics. + +#### Bug fixes + +- Fix security escalation bug in subscription management. +- Certain permissions should not be allowed at the database context. +- Make the time in `influxd-ctl`'s `copy-shard-status` argument human readable. +- Fix `influxd-ctl remove-data -force`. +- Ensure replaced data node correctly joins meta cluster. +- Delay metadata restriction on restore. +- Writing points outside of retention policy does not return error +- Decrement internal database's replication factor when a node is removed. + +## v1.2.5 [2017-05-16] + +This release builds off of the 1.2.4 release of OSS InfluxDB. +Please see the OSS [release notes](/influxdb/v1.3/about_the_project/releasenotes-changelog/#v1-2-4-2017-05-08) for more information about the OSS releases. + +#### Bug fixes + +- Fix issue where the [`ALTER RETENTION POLICY` query](/influxdb/v1.3/query_language/database_management/#modify-retention-policies-with-alter-retention-policy) does not update the default retention policy. +- Hinted-handoff: remote write errors containing `partial write` are considered droppable. +- Fix the broken `influxd-ctl remove-data -force` command. +- Fix security escalation bug in subscription management. +- Prevent certain user permissions from having a database-specific scope. +- Reduce the cost of the admin user check for clusters with large numbers of users. +- Fix hinted-handoff remote write batching. + +## v1.2.2 [2017-03-15] + +This release builds off of the 1.2.1 release of OSS InfluxDB. +Please see the OSS [release notes](https://github.com/influxdata/influxdb/blob/1.2/CHANGELOG.md#v121-2017-03-08) for more information about the OSS release. + +### Configuration Changes + +The following configuration changes may need to changed before [upgrading](/enterprise_influxdb/v1.3/administration/upgrading/) to 1.2.2 from prior versions. + +#### shard-writer-timeout + +We've removed the data node's `shard-writer-timeout` configuration option from the `[cluster]` section. +As of version 1.2.2, the system sets `shard-writer-timeout` internally. +The configuration option can be removed from the [data node configuration file](/enterprise_influxdb/v1.3/administration/configuration/#data-node-configuration). + +#### retention-autocreate + +In versions 1.2.0 and 1.2.1, the `retention-autocreate` setting appears in both the meta node and data node configuration files. +To disable retention policy auto-creation, users on version 1.2.0 and 1.2.1 must set `retention-autocreate` to `false` in both the meta node and data node configuration files. + +In version 1.2.2, we’ve removed the `retention-autocreate` setting from the data node configuration file. +As of version 1.2.2, users may remove `retention-autocreate` from the data node configuration file. +To disable retention policy auto-creation, set `retention-autocreate` to `false` in the meta node configuration file only. + +This change only affects users who have disabled the `retention-autocreate` option and have installed version 1.2.0 or 1.2.1. + +#### Bug fixes + +##### Backup and Restore +
+ +- Prevent the `shard not found` error by making [backups](/enterprise_influxdb/v1.3/guides/backup-and-restore/#backup) skip empty shards +- Prevent the `shard not found` error by making [restore](/enterprise_influxdb/v1.3/guides/backup-and-restore/#restore) handle empty shards +- Ensure that restores from an incremental backup correctly handle file paths +- Allow incremental backups with restrictions (for example, they use the `-db` or `rp` flags) to be stores in the same directory +- Support restores on meta nodes that are not the raft leader + +##### Hinted handoff +
+ +- Fix issue where dropped writes were not recorded when the [hinted handoff](/enterprise_influxdb/v1.3/concepts/clustering/#hinted-handoff) queue reached the maximum size +- Prevent the hinted handoff from becoming blocked if it encounters field type errors + +##### Other +
+ +- Return partial results for the [`SHOW TAG VALUES` query](/influxdb/v1.3/query_language/schema_exploration/#show-tag-values) even if the cluster includes an unreachable data node +- Return partial results for the [`SHOW MEASUREMENTS` query](/influxdb/v1.3/query_language/schema_exploration/#show-measurements) even if the cluster includes an unreachable data node +- Prevent a panic when the system files to process points +- Ensure that cluster hostnames can be case insensitive +- Update the `retryCAS` code to wait for a newer snapshot before retrying +- Serialize access to the meta client and meta store to prevent raft log buildup +- Remove sysvinit package dependency for RPM packages +- Make the default retention policy creation an atomic process instead of a two-step process +- Prevent `influxd-ctl`'s [`join` argument](/enterprise_influxdb/v1.3/features/cluster-commands/#join) from completing a join when the command also specifies the help flag (`-h`) +- Fix the `influxd-ctl`'s [force removal](/enterprise_influxdb/v1.3/features/cluster-commands/#remove-meta) of meta nodes +- Update the meta node and data node sample configuration files + +## v1.2.1 [2017-01-25] + +#### Cluster-specific Bugfixes + +- Fix panic: Slice bounds out of range + Fix how the system removes expired shards. +- Remove misplaced newlines from cluster logs + +## v1.2.0 [2017-01-24] + +This release builds off of the 1.2.0 release of OSS InfluxDB. +Please see the OSS [release notes](https://github.com/influxdata/influxdb/blob/1.2/CHANGELOG.md#v120-2017-01-24) for more information about the OSS release. + +### Upgrading + +* The `retention-autocreate` configuration option has moved from the meta node configuration file to the [data node configuration file](/enterprise_influxdb/v1.3/administration/configuration/#retention-autocreate-true). +To disable the auto-creation of retention policies, set `retention-autocreate` to `false` in your data node configuration files. +* The previously deprecated `influxd-ctl force-leave` command has been removed. The replacement command to remove a meta node which is never coming back online is [`influxd-ctl remove-meta -force`](/enterprise_influxdb/v1.3/features/cluster-commands/). + +#### Cluster-specific Features + +- Improve the meta store: any meta store changes are done via a compare and swap +- Add support for [incremental backups](/enterprise_influxdb/v1.3/guides/backup-and-restore/) +- Automatically remove any deleted shard groups from the data store +- Uncomment the section headers in the default [configuration file](/enterprise_influxdb/v1.3/administration/configuration/) +- Add InfluxQL support for [subqueries](/influxdb/v1.3/query_language/data_exploration/#subqueries) + +#### Cluster-specific Bugfixes + +- Update dependencies with Godeps +- Fix a data race in meta client +- Ensure that the system removes the relevant [user permissions and roles](/enterprise_influxdb/v1.3/features/users/) when a database is dropped +- Fix a couple typos in demo [configuration file](/enterprise_influxdb/v1.3/administration/configuration/) +- Make optional the version protobuf field for the meta store +- Remove the override of GOMAXPROCS +- Remove an unused configuration option (`dir`) from the backend +- Fix a panic around processing remote writes +- Return an error if a remote write has a field conflict +- Drop points in the hinted handoff that (1) have field conflict errors (2) have [`max-values-per-tag`](/influxdb/v1.3/administration/config/#max-values-per-tag-100000) errors +- Remove the deprecated `influxd-ctl force-leave` command +- Fix issue where CQs would stop running if the first meta node in the cluster stops +- Fix logging in the meta httpd handler service +- Fix issue where subscriptions send duplicate data for [Continuous Query](/influxdb/v1.3/query_language/continuous_queries/) results +- Fix the output for `influxd-ctl show-shards` +- Send the correct RPC response for `ExecuteStatementRequestMessage` + +## v1.1.5 [2017-04-28] + +### Bug fixes + +- Prevent certain user permissions from having a database-specific scope. +- Fix security escalation bug in subscription management. + +## v1.1.3 [2017-02-27] + +This release incorporates the changes in the 1.1.4 release of OSS InfluxDB. +Please see the OSS [changelog](https://github.com/influxdata/influxdb/blob/v1.1.4/CHANGELOG.md) for more information about the OSS release. + +### Bug fixes + +- Delay when a node listens for network connections until after all requisite services are running. This prevents queries to the cluster from failing unnecessarily. +- Allow users to set the `GOMAXPROCS` environment variable. + +## v1.1.2 [internal] + +This release was an internal release only. +It incorporates the changes in the 1.1.3 release of OSS InfluxDB. +Please see the OSS [changelog](https://github.com/influxdata/influxdb/blob/v1.1.3/CHANGELOG.md) for more information about the OSS release. + +## v1.1.1 [2016-12-06] + +This release builds off of the 1.1.1 release of OSS InfluxDB. +Please see the OSS [release notes](https://github.com/influxdata/influxdb/blob/1.1/CHANGELOG.md#v111-2016-12-06) for more information about the OSS release. + +This release is built with Go (golang) 1.7.4. +It resolves a security vulnerability reported in Go (golang) version 1.7.3 which impacts all +users currently running on the macOS platform, powered by the Darwin operating system. + +#### Cluster-specific bug fixes + +- Fix hinted-handoff issue: Fix record size larger than max size + If a Hinted Handoff write appended a block that was larger than the maximum file size, the queue would get stuck because the maximum size was not updated. When reading the block back out during processing, the system would return an error because the block size was larger than the file size -- which indicates a corrupted block. + +## v1.1.0 [2016-11-14] + +This release builds off of the 1.1.0 release of InfluxDB OSS. +Please see the OSS [release notes](https://github.com/influxdata/influxdb/blob/1.1/CHANGELOG.md#v110-2016-11-14) for more information about the OSS release. + +### Upgrading + +* The 1.1.0 release of OSS InfluxDB has some important [configuration changes](https://github.com/influxdata/influxdb/blob/1.1/CHANGELOG.md#configuration-changes) that may affect existing clusters. +* The `influxd-ctl join` command has been renamed to `influxd-ctl add-meta`. If you have existing scripts that use `influxd-ctl join`, they will need to use `influxd-ctl add-meta` or be updated to use the new cluster setup command. + +#### Cluster setup + +The `influxd-ctl join` command has been changed to simplify cluster setups. To join a node to a cluster, you can run `influxd-ctl join `, and we will attempt to detect and add any meta or data node process running on the hosts automatically. The previous `join` command exists as `add-meta` now. If it's the first node of a cluster, the meta address argument is optional. + +#### Logging + +Switches to journald logging for on systemd systems. Logs are no longer sent to `/var/log/influxdb` on systemd systems. + +#### Cluster-specific features + +- Add a configuration option for setting gossiping frequency on data nodes +- Allow for detailed insight into the Hinted Handoff queue size by adding `queueBytes` to the hh\_processor statistics +- Add authentication to the meta service API +- Update Go (golang) dependencies: Fix Go Vet and update circle Go Vet command +- Simplify the process for joining nodes to a cluster +- Include the node's version number in the `influxd-ctl show` output +- Return and error if there are additional arguments after `influxd-ctl show` + Fixes any confusion between the correct command for showing detailed shard information (`influxd-ctl show-shards`) and the incorrect command (`influxd-ctl show shards`) + +#### Cluster-specific bug fixes + +- Return an error if getting latest snapshot takes longer than 30 seconds +- Remove any expired shards from the `/show-shards` output +- Respect the [`pprof-enabled` configuration setting](/enterprise_influxdb/v1.3/administration/configuration/#pprof-enabled-true) and enable it by default on meta nodes +- Respect the [`pprof-enabled` configuration setting](/enterprise_influxdb/v1.3/administration/configuration/#pprof-enabled-true-1) on data nodes +- Use the data reference instead of `Clone()` during read-only operations for performance purposes +- Prevent the system from double-collecting cluster statistics +- Ensure that the Meta API redirects to the cluster leader when it gets the `ErrNotLeader` error +- Don't overwrite cluster users with existing OSS InfluxDB users when migrating an OSS instance into a cluster +- Fix a data race in the raft store +- Allow large segment files (> 10MB) in the Hinted Handoff +- Prevent `copy-shard` from retrying if the `copy-shard` command was killed +- Prevent a hanging `influxd-ctl add-data` command by making data nodes check for meta nodes before they join a cluster + +## v1.0.4 [2016-10-19] + +#### Cluster-specific bug fixes + +- Respect the [Hinted Handoff settings](/enterprise_influxdb/v1.3/administration/configuration/#hinted-handoff) in the configuration file +- Fix expanding regular expressions when all shards do not exist on node that's handling the request + +## v1.0.3 [2016-10-07] + +#### Cluster-specific bug fixes + +- Fix a panic in the Hinted Handoff: `lastModified` + +## v1.0.2 [2016-10-06] + +This release builds off of the 1.0.2 release of OSS InfluxDB. Please see the OSS [release notes](https://github.com/influxdata/influxdb/blob/1.0/CHANGELOG.md#v102-2016-10-05) for more information about the OSS release. + +#### Cluster-specific bug fixes + +- Prevent double read-lock in the meta client +- Fix a panic around a corrupt block in Hinted Handoff +- Fix issue where `systemctl enable` would throw an error if the symlink already exists + +## v1.0.1 [2016-09-28] + +This release builds off of the 1.0.1 release of OSS InfluxDB. +Please see the OSS [release notes](https://github.com/influxdata/influxdb/blob/1.0/CHANGELOG.md#v101-2016-09-26) +for more information about the OSS release. + +#### Cluster-specific bug fixes + +* Balance shards correctly with a restore +* Fix a panic in the Hinted Handoff: `runtime error: invalid memory address or nil pointer dereference` +* Ensure meta node redirects to leader when removing data node +* Fix a panic in the Hinted Handoff: `runtime error: makeslice: len out of range` +* Update the data node configuration file so that only the minimum configuration options are uncommented + +## v1.0.0 [2016-09-07] + +This release builds off of the 1.0.0 release of OSS InfluxDB. +Please see the OSS [release notes](https://github.com/influxdata/influxdb/blob/1.0/CHANGELOG.md#v100-2016-09-07) for more information about the OSS release. + +Breaking Changes: + +* The keywords `IF`, `EXISTS`, and `NOT` were removed for this release. This means you no longer need to specify `IF NOT EXISTS` for `DROP DATABASE` or `IF EXISTS` for `CREATE DATABASE`. Using these keywords will return a query error. +* `max-series-per-database` was added with a default of 1M but can be disabled by setting it to `0`. Existing databases with series that exceed this limit will continue to load, but writes that would create new series will fail. + +### Hinted handoff + +A number of changes to hinted handoff are included in this release: + +* Truncating only the corrupt block in a corrupted segment to minimize data loss +* Immediately queue writes in hinted handoff if there are still writes pending to prevent inconsistencies in shards +* Remove hinted handoff queues when data nodes are removed to eliminate manual cleanup tasks + +### Performance + +* `SHOW MEASUREMENTS` and `SHOW TAG VALUES` have been optimized to work better for multiple nodes and shards +* `DROP` and `DELETE` statements run in parallel and more efficiently and should not leave the system in an inconsistent state + +### Security + +The Cluster API used by `influxd-ctl` can not be protected with SSL certs. + +### Cluster management + +Data nodes that can no longer be restarted can now be forcefully removed from the cluster using `influxd-ctl remove-data -force `. This should only be run if a grace removal is not possible. + +Backup and restore has been updated to fix issues and refine existing capabilities. + +#### Cluster-specific features + +- Add the Users method to control client +- Add a `-force` option to the `influxd-ctl remove-data` command +- Disable the logging of `stats` service queries +- Optimize the `SHOW MEASUREMENTS` and `SHOW TAG VALUES` queries +- Update the Go (golang) package library dependencies +- Minimize the amount of data-loss in a corrupted Hinted Handoff file by truncating only the last corrupted segment instead of the entire file +- Log a write error when the Hinted Handoff queue is full for a node +- Remove Hinted Handoff queues on data nodes when the target data nodes are removed from the cluster +- Add unit testing around restore in the meta store +- Add full TLS support to the cluster API, including the use of self-signed certificates +- Improve backup/restore to allow for partial restores to a different cluster or to a database with a different database name +- Update the shard group creation logic to be balanced +- Keep raft log to a minimum to prevent replaying large raft logs on startup + +#### Cluster-specific bug fixes + +- Remove bad connections from the meta executor connection pool +- Fix a panic in the meta store +- Fix a panic caused when a shard group is not found +- Fix a corrupted Hinted Handoff +- Ensure that any imported OSS admin users have all privileges in the cluster +- Ensure that `max-select-series` is respected +- Handle the `peer already known` error +- Fix Hinted handoff panic around segment size check +- Drop Hinted Handoff writes if they contain field type inconsistencies + +
+# Web Console + +## DEPRECATED: Enterprise Web Console + +The Enterprise Web Console has officially been deprecated and will be eliminated entirely by the end of 2017. +No additional features will be added and no additional bug fix releases are planned. + +For browser-based access to InfluxDB Enterprise, [Chronograf](/{{< latest "chronograf" >}}/introduction) is now the recommended tool to use. diff --git a/content/enterprise_influxdb/v1.9/about-the-project/third-party.md b/content/enterprise_influxdb/v1.9/about-the-project/third-party.md new file mode 100644 index 000000000..8cc3f30d3 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/about-the-project/third-party.md @@ -0,0 +1,51 @@ +--- +title: Third party software +description: > + InfluxData products contain third-party software that is copyrighted, + patented, or otherwise legally protected software of third parties + incorporated in InfluxData products. +menu: + enterprise_influxdb_1_9_ref: + name: Third party software + weight: 20 + parent: About the project +--- + +InfluxData products contain third party software, which means the copyrighted, +patented, or otherwise legally protected software of third parties that is +incorporated in InfluxData products. + +Third party suppliers make no representation nor warranty with respect to +such third party software or any portion thereof. +Third party suppliers assume no liability for any claim that might arise with +respect to such third party software, nor for a +customer’s use of or inability to use the third party software. + +InfluxDB Enterprise 1.9 includes the following third party software components, which are maintained on a version by version basis. + +| Component | License | Integration | +| :-------- | :-------- | :-------- | +| [ASN1 BER Encoding / Decoding Library for the GO programming language (go-asn1-ber/ans1-ber)](https://github.com/go-asn1-ber/asn1-ber) | [MIT](https://opensource.org/licenses/MIT) | Statically linked | +| [Cobra is a commander for modern Go CLI interactions (spf13/cobra)](https://github.com/spf13/cobra) | [BSD 2-Clause](https://opensource.org/licenses/BSD-2-Clause) | Statically linked | +| [A golang registry for global request variables (gorilla/context)](https://github.com/gorilla/context) | [BSD 3-Clause](https://opensource.org/licenses/BSD-3-Clause) | Statically linked | +| [FlatBuffers: Memory Efficient Serialization Library (google/flatbuffers)](https://github.com/google/flatbuffers) | [Apache License 2.0](https://opensource.org/licenses/Apache-2.0) | Statically linked | +| [Flux is a lightweight scripting language for querying databases (like InfluxDB) and working with data (influxdata/flux)](https://github.com/influxdata/flux) | [Apache License 2.0](https://opensource.org/licenses/Apache-2.0) | Statically linked | +| [GoConvey is a yummy Go testing tool for gophers (glycerine/goconvey)](https://github.com/glycerine/goconvey) | [MIT](https://opensource.org/licenses/MIT) | Statically linked | +| [An immutable radix tree implementation in Golang (hashicorp/go-immutable-radix)](https://github.com/hashicorp/go-immutable-radix)| [Mozilla Public License 2.0](https://opensource.org/licenses/MPL-2.0) | Statically linked | +| [Some helpful packages for writing Go apps (markbates/going)](https://github.com/markbates/going)| [MIT](https://opensource.org/licenses/MIT) | Statically linked | +| [Golang LRU cache implements a fixed-size thread safe LRU cache (hashicorp/golang-lru)](https://github.com/hashicorp/golang-lru) |[Mozilla Public License 2.0](https://opensource.org/licenses/MPL-2.0) | Statically linked | +| [Codec - a high performance and feature-rich Idiomatic encode/decode and rpc library for msgpack and Binc (hashicorp/go-msgpack)](https://github.com/hashicorp/go-msgpack)| [BSD 3-Clause](https://opensource.org/licenses/BSD-3-Clause) | Statically linked | +| [A Golang library for exporting performance and runtime metrics to external metrics systems, i.e. statsite, statsd (armon/go-metrics)](https://github.com/armon/go-metrics) | [MIT](https://opensource.org/licenses/MIT) | Statically linked | +| [Generates UUID-format strings using purely high quality random bytes (hashicorp/go-uuid)](https://github.com/hashicorp/go-uuid) | [Mozilla Public License 2.0](https://opensource.org/licenses/MPL-2.0) | Statically linked | +| [Collection of useful handlers for Go net/http package (gorilla/handlers)](https://github.com/gorilla/handlers) | [BSD 2-Clause](https://opensource.org/licenses/BSD-2-Clause) | Statically linked | +| [Golang implementation of JavaScript Object (dvsekhvalnov/jose2go)](https://github.com/dvsekhvalnov/jose2go) | [MIT](https://opensource.org/licenses/MIT) | Statically linked | +| [Basic LDAP v3 functionality for the Go programming language (go-ldap/ldap)](https://github.com/go-ldap/ldap) | [MIT](https://opensource.org/licenses/MIT) | Statically linked | +| [Basic LDAP v3 functionality for the Go programming language (mark-rushakoff/ldapserver)](https://github.com/mark-rushakoff/ldapserver) | [BSD 3-Clause](https://opensource.org/licenses/BSD-3-Clause) | Statically linked | +| [A powerful URL router and dispatcher for golang (gorilla/mux)](https://github.com/gorilla/mux) | [BSD 2-Clause](https://opensource.org/licenses/BSD-2-Clause) | Statically linked | +| [pkcs7 implements parsing and creating signed and enveloped messages (fullsailor/pkcs7)](https://github.com/fullsailor/pkcs7) | [MIT](https://opensource.org/licenses/MIT) | Statically linked | +| [Pretty printing for Go values (kr/pretty)](https://github.com/kr/pretty) | [MIT](https://opensource.org/licenses/MIT) | Statically linked |Statically linked| +|[Go language implementation of the Raft consensus protocol (hashicorp/raft)](https://github.com/hashicorp/raft) | [Mozilla Public License 2.0](https://opensource.org/licenses/MPL-2.0) | Statically linked | +| [Raft backend implementation using BoltDB (hashicorp/raft-boltdb)](https://github.com/hashicorp/raft-boltdb) | [Mozilla Public License 2.0](https://opensource.org/licenses/MPL-2.0) | Statically linked | +| [General purpose extensions to golang's database/sql (jmoiron/sqlx)](https://github.com/jmoiron/sqlx) | [MIT](https://opensource.org/licenses/MIT) | Statically linked |Statically linked| +| [Miscellaneous functions for formatting text (kr/text)](https://github.com/kr/text) | [MIT](https://opensource.org/licenses/MIT) | Statically linked | +| [Golang connection multiplexing library (hashicorp/yamux)](https://github.com/hashicorp/yamux/) | [Mozilla Public License 2.0](https://opensource.org/licenses/MPL-2.0) | Statically linked | diff --git a/content/enterprise_influxdb/v1.9/administration/_index.md b/content/enterprise_influxdb/v1.9/administration/_index.md new file mode 100644 index 000000000..d4ce25cb1 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/administration/_index.md @@ -0,0 +1,10 @@ +--- +title: Administer InfluxDB Enterprise +description: Configuration, security, and logging in InfluxDB enterprise. +menu: + enterprise_influxdb_1_9: + name: Administration + weight: 70 +--- + +{{< children >}} diff --git a/content/enterprise_influxdb/v1.9/administration/anti-entropy/_index.md b/content/enterprise_influxdb/v1.9/administration/anti-entropy/_index.md new file mode 100644 index 000000000..4dc0dae35 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/administration/anti-entropy/_index.md @@ -0,0 +1,348 @@ +--- +title: Use Anti-Entropy service in InfluxDB Enterprise +description: The Anti-Entropy service monitors and repairs shards in InfluxDB. +aliases: + - /enterprise_influxdb/v1.9/guides/Anti-Entropy/ +menu: + enterprise_influxdb_1_9: + name: Use Anti-entropy service + weight: 60 + parent: Administration +--- + +{{% warn %}} +Prior to InfluxDB Enterprise 1.7.2, the Anti-Entropy (AE) service was enabled by default. When shards create digests with lots of time ranges (10s of thousands), some customers have experienced significant performance issues, including CPU usage spikes. If your shards include a small number of time ranges (most have 1 to 10, some have up to several hundreds) and you can benefit from the AE service, enable AE and monitor it closely to see if your performance is adversely impacted. +{{% /warn %}} + +## Introduction + +Shard entropy refers to inconsistency among shards in a shard group. +This can be due to the "eventually consistent" nature of data stored in InfluxDB +Enterprise clusters or due to missing or unreachable shards. +The Anti-Entropy (AE) service ensures that each data node has all the shards it +owns according to the metastore and that all shards in a shard group are consistent. +Missing shards are automatically repaired without operator intervention while +out-of-sync shards can be manually queued for repair. +This topic covers how the Anti-Entropy service works and some of the basic situations where it takes effect. + +## Concepts + +The Anti-Entropy service is a component of the `influxd` service available on each of your data nodes. Use this service to ensure that each data node has all of the shards that the metastore says it owns and ensure all shards in a shard group are in sync. +If any shards are missing, the Anti-Entropy service will copy existing shards from other shard owners. +If data inconsistencies are detected among shards in a shard group, [invoke the Anti-Entropy service](#command-line-tools-for-managing-entropy) and queue the out-of-sync shards for repair. +In the repair process, the Anti-Entropy service will sync the necessary updates from other shards +within a shard group. + +By default, the service performs consistency checks every 5 minutes. This interval can be modified in the [`anti-entropy.check-interval`](/enterprise_influxdb/v1.9/administration/config-data-nodes/#check-interval-5m) configuration setting. + +The Anti-Entropy service can only address missing or inconsistent shards when +there is at least one copy of the shard available. +In other words, as long as new and healthy nodes are introduced, a replication +factor of 2 can recover from one missing or inconsistent node; +a replication factor of 3 can recover from two missing or inconsistent nodes, and so on. +A replication factor of 1, which is not recommended, cannot be recovered by the Anti-Entropy service. + +## Symptoms of entropy + +The Anti-Entropy service automatically detects and fixes missing shards, but shard inconsistencies +must be [manually detected and queued for repair](#detecting-and-repairing-entropy). +There are symptoms of entropy that, if seen, would indicate an entropy repair is necessary. + +### Different results for the same query + +When running queries against an InfluxDB Enterprise cluster, each query may be routed to a different data node. +If entropy affects data within the queried range, the same query will return different +results depending on which node the query runs against. + +_**Query attempt 1**_ + +```sql +SELECT mean("usage_idle") WHERE time > '2018-06-06T18:00:00Z' AND time < '2018-06-06T18:15:00Z' GROUP BY time(3m) FILL(0) + +name: cpu +time mean +---- ---- +1528308000000000000 99.11867392974537 +1528308180000000000 99.15410822137049 +1528308360000000000 99.14927494363032 +1528308540000000000 99.1980535465783 +1528308720000000000 99.18584290492262 +``` + +_**Query attempt 2**_ + +```sql +SELECT mean("usage_idle") WHERE time > '2018-06-06T18:00:00Z' AND time < '2018-06-06T18:15:00Z' GROUP BY time(3m) FILL(0) + +name: cpu +time mean +---- ---- +1528308000000000000 99.11867392974537 +1528308180000000000 0 +1528308360000000000 0 +1528308540000000000 0 +1528308720000000000 99.18584290492262 +``` + +The results indicate that data is missing in the queried time range and entropy is present. + +### Flapping dashboards + +A "flapping" dashboard means data visualizations change when data is refreshed +and pulled from a node with entropy (inconsistent data). +It is the visual manifestation of getting [different results from the same query](#different-results-for-the-same-query). + +Flapping dashboard + +## Technical details + +### Detecting entropy + +The Anti-Entropy service runs on each data node and periodically checks its shards' statuses +relative to the next data node in the ownership list. +The service creates a "digest" or summary of data in the shards on the node. + +For example, assume there are two data nodes in your cluster: `node1` and `node2`. +Both `node1` and `node2` own `shard1` so `shard1` is replicated across each. + +When a status check runs, `node1` will ask `node2` when `shard1` was last modified. +If the reported modification time differs from the previous check, then +`node1` asks `node2` for a new digest of `shard1`, checks for differences (performs a "diff") between the `shard1` digest for `node2` and the local `shard1` digest. +If a difference exists, `shard1` is flagged as having entropy. + +### Repairing entropy + +If during a status check a node determines the next node is completely missing a shard, +it immediately adds the missing shard to the repair queue. +A background routine monitors the queue and begins the repair process as new shards are added to it. +Repair requests are pulled from the queue by the background process and repaired using a `copy shard` operation. + +> Currently, shards that are present on both nodes but contain different data are not automatically queued for repair. +> A user must make the request via `influxd-ctl entropy repair `. +> For more information, see [Detecting and repairing entropy](#detecting-and-repairing-entropy) below. + +Using `node1` and `node2` from the [earlier example](#detecting-entropy), `node1` asks `node2` for a digest of `shard1`. +`node1` diffs its own local `shard1` digest and `node2`'s `shard1` digest, +then creates a new digest containing only the differences (the diff digest). +The diff digest is used to create a patch containing only the data `node2` is missing. +`node1` sends the patch to `node2` and instructs it to apply it. +Once `node2` finishes applying the patch, it queues a repair for `shard1` locally. + +The "node-to-node" shard repair continues until it runs on every data node that owns the shard in need of repair. + +### Repair order + +Repairs between shard owners happen in a deterministic order. +This doesn't mean repairs always start on node 1 and then follow a specific node order. +Repairs are viewed at the shard level. +Each shard has a list of owners and the repairs for a particular shard will happen +in a deterministic order among its owners. + +When the Anti-Entropy service on any data node receives a repair request for a shard, it determines which +owner node is the first in the deterministic order and forwards the request to that node. +The request is now queued on the first owner. + +The first owner's repair processor pulls it from the queue, detects the differences +between the local copy of the shard with the copy of the same shard on the next +owner in the deterministic order, then generates a patch from that difference. +The first owner then makes an RPC call to the next owner instructing it to apply +the patch to its copy of the shard. + +Once the next owner has successfully applied the patch, it adds that shard to the Anti-Entropy repair queue. +A list of "visited" nodes follows the repair through the list of owners. +Each owner will check the list to detect when the repair has cycled through all owners, +at which point the repair is finished. + +### Hot shards + +The Anti-Entropy service does its best to avoid hot shards (shards that are currently receiving writes) +because they change quickly. +While write replication between shard owner nodes (with a +[replication factor](/enterprise_influxdb/v1.9/concepts/glossary/#replication-factor) +greater than 1) typically happens in milliseconds, this slight difference is +still enough to cause the appearance of entropy where there is none. + +Because the Anti-Entropy service repairs only cold shards, unexpected effects can occur. +Consider the following scenario: + +1. A shard goes cold. +2. Anti-Entropy detects entropy. +3. Entropy is reported by the [Anti-Entropy `/status` API](/enterprise_influxdb/v1.9/administration/anti-entropy-api/#get-status) or with the `influxd-ctl entropy show` command. +4. Shard takes a write, gets compacted, or something else causes it to go hot. + _These actions are out of Anti-Entropy's control._ +5. A repair is requested, but is ignored because the shard is now hot. + +In this example, you would have to periodically request a repair of the shard +until it either shows as being in the queue, being repaired, or no longer in the list of shards with entropy. + +## Configuration + +The configuration settings for the Anti-Entropy service are described in [Anti-Entropy settings](/enterprise_influxdb/v1.9/administration/config-data-nodes#anti-entropy) section of the data node configuration. + +To enable the Anti-Entropy service, change the default value of the `[anti-entropy].enabled = false` setting to `true` in the `influxdb.conf` file of each of your data nodes. + +## Command line tools for managing entropy + +>**Note:** The Anti-Entropy service is disabled by default and must be enabled before using these commands. + +The `influxd-ctl entropy` command enables you to manage entropy among shards in a cluster. +It includes the following subcommands: + +#### `show` + +Lists shards that are in an inconsistent state and in need of repair as well as +shards currently in the repair queue. + +```bash +influxd-ctl entropy show +``` + +#### `repair` + +Queues a shard for repair. +It requires a Shard ID which is provided in the [`show`](#show) output. + +```bash +influxd-ctl entropy repair +``` + +Repairing entropy in a shard is an asynchronous operation. +This command will return quickly as it only adds a shard to the repair queue. +Queuing shards for repair is idempotent. +There is no harm in making multiple requests to repair the same shard even if +it is already queued, currently being repaired, or not in need of repair. + +#### `kill-repair` + +Removes a shard from the repair queue. +It requires a Shard ID which is provided in the [`show`](#show) output. + +```bash +influxd-ctl entropy kill-repair +``` + +This only applies to shards in the repair queue. +It does not cancel repairs on nodes that are in the process of being repaired. +Once a repair has started, requests to cancel it are ignored. + +> Stopping a entropy repair for a **missing** shard operation is not currently supported. +> It may be possible to stop repairs for missing shards with the +> [`influxd-ctl kill-copy-shard`](/enterprise_influxdb/v1.9/tools/influxd-ctl/#kill-copy-shard) command. + +## InfluxDB Anti-Entropy API + +The Anti-Entropy service uses an API for managing and monitoring entropy. +Details on the available API endpoints can be found in [The InfluxDB Anti-Entropy API](/enterprise_influxdb/v1.9/administration/anti-entropy-api). + +## Use cases + +Common use cases for the Anti-Entropy service include detecting and repairing entropy, replacing unresponsive data nodes, replacing data nodes for upgrades and maintenance, and eliminating entropy in active shards. + +### Detecting and repairing entropy + +Periodically, you may want to see if shards in your cluster have entropy or are +inconsistent with other shards in the shard group. +Use the `influxd-ctl entropy show` command to list all shards with detected entropy: + +```bash +influxd-ctl entropy show + +Entropy +========== +ID Database Retention Policy Start End Expires Status +21179 statsdb 1hour 2017-10-09 00:00:00 +0000 UTC 2017-10-16 00:00:00 +0000 UTC 2018-10-22 00:00:00 +0000 UTC diff +25165 statsdb 1hour 2017-11-20 00:00:00 +0000 UTC 2017-11-27 00:00:00 +0000 UTC 2018-12-03 00:00:00 +0000 UTC diff +``` + +Then use the `influxd-ctl entropy repair` command to add the shards with entropy +to the repair queue: + +```bash +influxd-ctl entropy repair 21179 + +Repair Shard 21179 queued + +influxd-ctl entropy repair 25165 + +Repair Shard 25165 queued +``` + +Check on the status of the repair queue with the `influxd-ctl entropy show` command: + +```bash +influxd-ctl entropy show + +Entropy +========== +ID Database Retention Policy Start End Expires Status +21179 statsdb 1hour 2017-10-09 00:00:00 +0000 UTC 2017-10-16 00:00:00 +0000 UTC 2018-10-22 00:00:00 +0000 UTC diff +25165 statsdb 1hour 2017-11-20 00:00:00 +0000 UTC 2017-11-27 00:00:00 +0000 UTC 2018-12-03 00:00:00 +0000 UTC diff + +Queued Shards: [21179 25165] +``` + +### Replacing an unresponsive data node + +If a data node suddenly disappears due to a catastrophic hardware failure or for any other reason, as soon as a new data node is online, the Anti-Entropy service will copy the correct shards to the new replacement node. The time it takes for the copying to complete is determined by the number of shards to be copied and how much data is stored in each. + +_View the [Replacing Data Nodes](/enterprise_influxdb/v1.9/guides/replacing-nodes/#replace-data-nodes-in-an-influxdb-enterprise-cluster) documentation for instructions on replacing data nodes in your InfluxDB Enterprise cluster._ + +### Replacing a machine that is running a data node + +Perhaps you are replacing a machine that is being decommissioned, upgrading hardware, or something else entirely. +The Anti-Entropy service will automatically copy shards to the new machines. + +Once you have successfully run the `influxd-ctl update-data` command, you are free +to shut down the retired node without causing any interruption to the cluster. +The Anti-Entropy process will continue copying the appropriate shards from the +remaining replicas in the cluster. + +### Fixing entropy in active shards + +In rare cases, the currently active shard, or the shard to which new data is +currently being written, may find itself with inconsistent data. +Because the Anti-Entropy process can't write to hot shards, you must stop writes to the new +shard using the [`influxd-ctl truncate-shards` command](/enterprise_influxdb/v1.9/tools/influxd-ctl/#truncate-shards), +then add the inconsistent shard to the entropy repair queue: + +```bash +# Truncate hot shards +influxd-ctl truncate-shards + +# Show shards with entropy +influxd-ctl entropy show + +Entropy +========== +ID Database Retention Policy Start End Expires Status +21179 statsdb 1hour 2018-06-06 12:00:00 +0000 UTC 2018-06-06 23:44:12 +0000 UTC 2018-12-06 00:00:00 +0000 UTC diff + +# Add the inconsistent shard to the repair queue +influxd-ctl entropy repair 21179 +``` + +## Troubleshooting + +### Queued repairs are not being processed + +The primary reason a repair in the repair queue isn't being processed is because +it went "hot" after the repair was queued. +The Anti-Entropy service only repairs cold shards or shards that are not currently being written to. +If the shard is hot, the Anti-Entropy service will wait until it goes cold again before performing the repair. + +If the shard is "old" and writes to it are part of a backfill process, you simply +have to wait until the backfill process is finished. If the shard is the active +shard, run `truncate-shards` to stop writes to active shards. This process is +outlined [above](#fixing-entropy-in-active-shards). + +### Anti-Entropy log messages + +Below are common messages output by Anti-Entropy along with what they mean. + +#### `Checking status` + +Indicates that the Anti-Entropy process has begun the [status check process](#detecting-entropy). + +#### `Skipped shards` + +Indicates that the Anti-Entropy process has skipped a status check on shards because they are currently [hot](#hot-shards). diff --git a/content/enterprise_influxdb/v1.9/administration/anti-entropy/anti-entropy-api.md b/content/enterprise_influxdb/v1.9/administration/anti-entropy/anti-entropy-api.md new file mode 100644 index 000000000..d0c3b3419 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/administration/anti-entropy/anti-entropy-api.md @@ -0,0 +1,238 @@ +--- +title: InfluxDB Anti-Entropy API +description: > + Monitor and repair shards on InfluxDB Enterprise data nodes the InfluxDB Anti-Entropy API. +menu: + enterprise_influxdb_1_9: + name: Anti-entropy API + weight: 70 + parent: Use Anti-entropy service +aliases: + - /enterprise_influxdb/v1.9/administration/anti-entropy-api/ +--- + +>**Note:** The Anti-Entropy API is available from the meta nodes and is only available when the Anti-Entropy service is enabled in the data node configuration settings. For information on the configuration settings, see +> [Anti-Entropy settings](/enterprise_influxdb/v1.9/administration/config-data-nodes/#anti-entropy-ae-settings). + +Use the [Anti-Entropy service](/enterprise_influxdb/v1.9/administration/anti-entropy) in InfluxDB Enterprise to monitor and repair entropy in data nodes and their shards. To access the Anti-Entropy API and work with this service, use [`influx-ctl entropy`](/enterprise_influxdb/v1.9/tools/influxd-ctl/#entropy) (also available on meta nodes). + +The base URL is: + +```text +http://localhost:8086/shard-repair +``` + +## GET `/status` + +### Description + +Lists shards that are in an inconsistent state and in need of repair. + +### Parameters + +| Name | Located in | Description | Required | Type | +| ---- | ---------- | ----------- | -------- | ---- | +| `local` | query | Limits status check to local shards on the data node handling this request | No | boolean | + +### Responses + +#### Headers + +| Header name | Value | +|-------------|--------------------| +| `Accept` | `application/json` | + +#### Status codes + +| Code | Description | Type | +| ---- | ----------- | ------ | +| `200` | `Successful operation` | object | + +### Examples + +#### cURL request + +```bash +curl -X GET "http://localhost:8086/shard-repair/status?local=true" -H "accept: application/json" +``` + +#### Request URL + +```text +http://localhost:8086/shard-repair/status?local=true + +``` + +### Responses + +Example of server response value: + +```json +{ + "shards": [ + { + "id": "1", + "database": "ae", + "retention_policy": "autogen", + "start_time": "-259200000000000", + "end_time": "345600000000000", + "expires": "0", + "status": "diff" + }, + { + "id": "3", + "database": "ae", + "retention_policy": "autogen", + "start_time": "62640000000000000", + "end_time": "63244800000000000", + "expires": "0", + "status": "diff" + } + ], + "queued_shards": [ + "3", + "5", + "9" + ], + "processing_shards": [ + "3", + "9" + ] +} +``` + +## POST `/repair` + +### Description + +Queues the specified shard for repair of the inconsistent state. + +### Parameters + +| Name | Located in | Description | Required | Type | +| ---- | ---------- | ----------- | -------- | ---- | +| `id` | query | ID of shard to queue for repair | Yes | integer | + +### Responses + +#### Headers + +| Header name | Value | +| ----------- | ----- | +| `Accept` | `application/json` | + +#### Status codes + +| Code | Description | +| ---- | ----------- | +| `204` | `Successful operation` | +| `400` | `Bad request` | +| `500` | `Internal server error` | + +### Examples + +#### cURL request + +```bash +curl -X POST "http://localhost:8086/shard-repair/repair?id=1" -H "accept: application/json" +``` + +#### Request URL + +```text +http://localhost:8086/shard-repair/repair?id=1 +``` + +## POST `/cancel-repair` + +### Description + +Removes the specified shard from the repair queue on nodes. + +### Parameters + +| Name | Located in | Description | Required | Type | +| ---- | ---------- | ----------- | -------- | ---- | +| `id` | query | ID of shard to remove from repair queue | Yes | integer | +| `local` | query | Only remove shard from repair queue on node receiving the request | No | boolean | + +### Responses + +#### Headers + +| Header name | Value | +|-------------|--------------------| +| `Accept` | `application/json` | + +#### Status codes + +| Code | Description | +| ---- | ----------- | +| `204` | `Successful operation` | +| `400` | `Bad request` | +| `500` | `Internal server error` | + +### Examples + +#### cURL request + +```bash +curl -X POST "http://localhost:8086/shard-repair/cancel-repair?id=1&local=false" -H "accept: application/json" +``` + +#### Request URL + +```text +http://localhost:8086/shard-repair/cancel-repair?id=1&local=false +``` + +## Models + +### ShardStatus + +| Name | Type | Required | +| ---- | ---- | -------- | +| `id` | string | No | +| `database` | string | No | +| `retention_policy` | string | No | +| `start_time` | string | No | +| `end_time` | string | No | +| `expires` | string | No | +| `status` | string | No | + +### Examples + + +```json +{ + "shards": [ + { + "id": "1", + "database": "ae", + "retention_policy": "autogen", + "start_time": "-259200000000000", + "end_time": "345600000000000", + "expires": "0", + "status": "diff" + }, + { + "id": "3", + "database": "ae", + "retention_policy": "autogen", + "start_time": "62640000000000000", + "end_time": "63244800000000000", + "expires": "0", + "status": "diff" + } + ], + "queued_shards": [ + "3", + "5", + "9" + ], + "processing_shards": [ + "3", + "9" + ] +} +``` diff --git a/content/enterprise_influxdb/v1.9/administration/authentication_and_authorization.md b/content/enterprise_influxdb/v1.9/administration/authentication_and_authorization.md new file mode 100644 index 000000000..10153c364 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/administration/authentication_and_authorization.md @@ -0,0 +1,500 @@ +--- +title: Authentication and authorization in InfluxDB Enterprise +description: > + Set up and manage authentication and authorization in InfluxDB Enterprise. +menu: + enterprise_influxdb_1_9: + name: Manage authentication and authorization + weight: 20 + parent: Administration +--- + +This document covers setting up and managing authentication and authorization in InfluxDB Enterprise. + +- [Authentication](#authentication) + - [Set up Authentication](#set-up-authentication) + - [Authenticate Requests](#authenticate-requests) +- [Authorization](#authorization) + - [User Types and Privileges](#user-types-and-privileges) + - [User Management Commands](#user-management-commands) +- [HTTP Errors](#authentication-and-authorization-http-errors) + +{{% note %}} +Authentication and authorization should not be relied upon to prevent access and protect data from malicious actors. +If additional security or compliance features are desired, InfluxDB Enterprise should be run behind a third-party service. +If InfluxDB Enterprise is being deployed on a publicly accessible endpoint, we strongly recommend authentication be enabled. Otherwise the data will be +publicly available to any unauthenticated user. +{{% /note %}} + +## Authentication + +The InfluxDB API and the [`influx` CLI](/enterprise_influxdb/v1.9/tools/influx-cli/), +which connects to the database using the API, +include built-in authentication based on user credentials. +When you enable authentication, InfluxDB Enterprise only executes HTTP requests that are sent with valid credentials. + +{{% note %}} +Authentication only occurs at the HTTP request scope. +Plugins do not currently have the ability to authenticate requests and service +endpoints (for example, Graphite, collectd, etc.) are not authenticated. +{{% /note %}} + +### Set up authentication + +1. **Create at least one [admin user](#admin-users)**. + See the [authorization section](#authorization) for how to create an admin user. + + {{% note %}} +If you enable authentication and have no users, InfluxDB Enterprise will **not** enforce authentication +and will only accept the [query](#user-management-commands) that creates a new admin user. + {{% /note %}} + + InfluxDB Enterprise will enforce authentication once there is an admin user. + +2. **Enable authentication in your configuration file** + by setting the `auth-enabled` option to `true` in the `[http]` section: + + ```toml + [http] + enabled = true + bind-address = ":8086" + auth-enabled = true # Set to true + log-enabled = true + write-tracing = false + pprof-enabled = true + pprof-auth-enabled = true + debug-pprof-enabled = false + ping-auth-enabled = true + https-enabled = true + https-certificate = "/etc/ssl/influxdb.pem" + ``` + + {{% note %}} +If `pprof-enabled` is set to `true`, set `pprof-auth-enabled` and `ping-auth-enabled` +to `true` to require authentication on profiling and ping endpoints. + {{% /note %}} + +3. **Restart InfluxDB Enterprise**. + Once restarted, InfluxDB Enterprise checks user credentials on every request and only + processes requests that have valid credentials for an existing user. + +### Authenticate requests + +#### Authenticate with the InfluxDB API + +There are two options for authenticating with the [InfluxDB API](/enterprise_influxdb/v1.9/tools/api/). + +If you authenticate with both Basic Authentication **and** the URL query parameters, +the user credentials specified in the query parameters take precedence. +The queries in the following examples assume that the user is an [admin user](#admin-users). +See the section on [authorization](#authorization) for the different user types, their privileges, and more on user management. + +> **Note:** InfluxDB Enterprise redacts passwords when you enable authentication. + +##### Authenticate with Basic Authentication +```bash +curl -G http://localhost:8086/query \ + -u todd:influxdb4ever \ + --data-urlencode "q=SHOW DATABASES" +``` + +##### Authenticate with query parameters in the URL or request body +Set `u` as the username and `p` as the password. + +###### Credentials as query parameters +```bash +curl -G "http://localhost:8086/query?u=todd&p=influxdb4ever" \ + --data-urlencode "q=SHOW DATABASES" +``` + +###### Credentials in the request body +```bash +curl -G http://localhost:8086/query \ + --data-urlencode "u=todd" \ + --data-urlencode "p=influxdb4ever" \ + --data-urlencode "q=SHOW DATABASES" +``` + +#### Authenticate with the CLI + +There are three options for authenticating with the [CLI](/influxdb/v1.8/tools/shell/). + +##### Authenticate with environment variables +Use the `INFLUX_USERNAME` and `INFLUX_PASSWORD` environment variables to provide +authentication credentials to the `influx` CLI. + +```bash +export INFLUX_USERNAME=todd +export INFLUX_PASSWORD=influxdb4ever +echo $INFLUX_USERNAME $INFLUX_PASSWORD +todd influxdb4ever + +influx +Connected to http://localhost:8086 version 1.4.x +InfluxDB shell 1.4.x +``` + +##### Authenticate with CLI flags +Use the `-username` and `-password` flags to provide authentication credentials +to the `influx` CLI. + +```bash +influx -username todd -password influxdb4ever +Connected to http://localhost:8086 version 1.4.x +InfluxDB shell 1.4.x +``` + +##### Authenticate with credentials in the influx shell +Start the `influx` shell and run the `auth` command. +Enter your username and password when prompted. + +```bash +> influx +Connected to http://localhost:8086 version 1.4.x +InfluxDB shell 1.8.x +> auth +username: todd +password: +> +``` + +#### Authenticate using JWT tokens +For a more secure alternative to using passwords, include JWT tokens with requests to the InfluxDB API. +This is currently only possible through the [InfluxDB HTTP API](/influxdb/v1.8/tools/api/). + +1. [Add a shared secret in your InfluxDB configuration file](#add-a-shared-secret-in-your-influxdb-configuration-file) +2. [Generate your JWT token](#generate-your-jwt-token) +3. [Include the token in HTTP requests](#include-the-token-in-http-requests) + +##### Add a shared secret in your InfluxDB Enterprise configuration file +InfluxDB Enterprise uses the shared secret to encode the JWT signature. +By default, `shared-secret` is set to an empty string, in which case no JWT authentication takes place. +Add a custom shared secret in your [InfluxDB configuration file](/influxdb/v1.8/administration/config/#shared-secret). +The longer the secret string, the more secure it is: + +```toml +[http] + shared-secret = "my super secret pass phrase" +``` + +Alternatively, to avoid keeping your secret phrase as plain text in your InfluxDB configuration file, set the value with the `INFLUXDB_HTTP_SHARED_SECRET` environment variable. + + +##### Generate your JWT token +Use an authentication service to generate a secure token using your InfluxDB username, an expiration time, and your shared secret. +There are online tools, such as [https://jwt.io/](https://jwt.io/), that will do this for you. + +The payload (or claims) of the token must be in the following format: + +```json +{ + "username": "myUserName", + "exp": 1516239022 +} +``` + +- **username** - The name of your InfluxDB user. +- **exp** - The expiration time of the token in UNIX epoch time. + For increased security, keep token expiration periods short. + For testing, you can manually generate UNIX timestamps using [https://www.unixtimestamp.com/index.php](https://www.unixtimestamp.com/index.php). + +Encode the payload using your shared secret. +You can do this with either a JWT library in your own authentication server or by hand at [https://jwt.io/](https://jwt.io/). + +The generated token follows this format: `
..` + +##### Include the token in HTTP requests +Include your generated token as part of the ``Authorization`` header in HTTP requests. +Use the ``Bearer`` authorization scheme: + +``` +Authorization: Bearer +``` +{{% note %}} +Only unexpired tokens will successfully authenticate. +Be sure your token has not expired. +{{% /note %}} + +###### Example query request with JWT authentication +```bash +curl -G "http://localhost:8086/query?db=demodb" \ + --data-urlencode "q=SHOW DATABASES" \ + --header "Authorization: Bearer
.." +``` + +## Authenticate Telegraf requests to InfluxDB + +Authenticating [Telegraf](/{{< latest "telegraf" >}}/) requests to an InfluxDB instance with +authentication enabled requires some additional steps. +In the Telegraf configuration file (`/etc/telegraf/telegraf.conf`), uncomment +and edit the `username` and `password` settings. + +```toml +############################################################################### +# OUTPUT PLUGINS # +############################################################################### + +# ... + +[[outputs.influxdb]] + # ... + username = "example-username" # Provide your username + password = "example-password" # Provide your password + +# ... +``` + +Restart Telegraf and you're all set! + +## Authorization + +Authorization is only enforced once you've [enabled authentication](#set-up-authentication). +By default, authentication is disabled, all credentials are silently ignored, and all users have all privileges. + +### User types and privileges + +#### Admin users +Admin users have `READ` and `WRITE` access to all databases and full access to the following administrative queries: + +##### Database management +- `CREATE DATABASE` +- `DROP DATABASE` +- `DROP SERIES` +- `DROP MEASUREMENT` +- `CREATE RETENTION POLICY` +- `ALTER RETENTION POLICY` +- `DROP RETENTION POLICY` +- `CREATE CONTINUOUS QUERY` +- `DROP CONTINUOUS QUERY` + +For more information about these commands, see [Database management](/influxdb/v1.8/query_language/manage-database/) and +[Continuous queries](/influxdb/v1.8/query_language/continuous_queries/). + +##### User management +- Admin user management + - [`CREATE USER`](#user-management-commands) + - [`GRANT ALL PRIVILEGES`](#grant-administrative-privileges-to-an-existing-user) + - [`REVOKE ALL PRIVILEGES`](#revoke-administrative-privileges-from-an-admin-user) + - [`SHOW USERS`](#show-all-existing-users-and-their-admin-status) +- Non-admin user management: + - [`CREATE USER`](#user-management-commands) + - [`GRANT [READ,WRITE,ALL]`](#grant-read-write-or-all-database-privileges-to-an-existing-user) + - [`REVOKE [READ,WRITE,ALL]`](#revoke-read-write-or-all-database-privileges-from-an-existing-user) +- General user management: + - [`SET PASSWORD`](#reset-a-users-password) + - [`DROP USER`](#drop-a-user) + +See [below](#user-management-commands) for a complete discussion of the user management commands. + +#### Non-admin users +Non-admin users can have one of the following three privileges per database: + +- `READ` +- `WRITE` +- `ALL` (both `READ` and `WRITE` access) + +`READ`, `WRITE`, and `ALL` privileges are controlled per user per database. A new non-admin user has no access to any database until they are specifically [granted privileges to a database](#grant-read-write-or-all-database-privileges-to-an-existing-user) by an admin user. +Non-admin users can [`SHOW`](/influxdb/v1.8/query_language/explore-schema/#show-databases) the databases on which they have `READ` and/or `WRITE` permissions. + +### User management commands + +#### Admin user management + +When you enable HTTP authentication, InfluxDB requires you to create at least one admin user before you can interact with the system. + +```sql +CREATE USER admin WITH PASSWORD '' WITH ALL PRIVILEGES +``` + +##### Create another admin user + +```sql +CREATE USER WITH PASSWORD '' WITH ALL PRIVILEGES +``` + +{{% note %}} +Repeating the exact `CREATE USER` statement is idempotent. +If any values change the database will return a duplicate user error. + +```sql +> CREATE USER todd WITH PASSWORD '123456' WITH ALL PRIVILEGES +> CREATE USER todd WITH PASSWORD '123456' WITH ALL PRIVILEGES +> CREATE USER todd WITH PASSWORD '123' WITH ALL PRIVILEGES +ERR: user already exists +> CREATE USER todd WITH PASSWORD '123456' +ERR: user already exists +> CREATE USER todd WITH PASSWORD '123456' WITH ALL PRIVILEGES +> +``` +{{% /note %}} + +##### `GRANT` administrative privileges to an existing user +```sql +GRANT ALL PRIVILEGES TO +``` + +##### `REVOKE` administrative privileges from an admin user +```sql +REVOKE ALL PRIVILEGES FROM +``` + +##### `SHOW` all existing users and their admin status +```sql +SHOW USERS +``` + +###### CLI Example +```sql +> SHOW USERS +user admin +todd false +paul true +hermione false +dobby false +``` + +#### Non-admin user management + +##### `CREATE` a new non-admin user +```sql +CREATE USER WITH PASSWORD '' +``` + +###### CLI example +```js +> CREATE USER todd WITH PASSWORD 'influxdb41yf3' +> CREATE USER alice WITH PASSWORD 'wonder\'land' +> CREATE USER "rachel_smith" WITH PASSWORD 'asdf1234!' +> CREATE USER "monitoring-robot" WITH PASSWORD 'XXXXX' +> CREATE USER "$savyadmin" WITH PASSWORD 'm3tr1cL0v3r' +> +``` + +{{% note %}} +##### Important notes about providing user credentials +- The user value must be wrapped in double quotes if it starts with a digit, is an InfluxQL keyword, contains a hyphen and or includes any special characters, for example: `!@#$%^&*()-` +- The password [string](/influxdb/v1.8/query_language/spec/#strings) must be wrapped in single quotes. + Do not include the single quotes when authenticating requests. + We recommend avoiding the single quote (`'`) and backslash (`\`) characters in passwords. + For passwords that include these characters, escape the special character with a backslash (e.g. (`\'`) when creating the password and when submitting authentication requests. +- Repeating the exact `CREATE USER` statement is idempotent. If any values change the database will return a duplicate user error. See GitHub Issue [#6890](https://github.com/influxdata/influxdb/pull/6890) for details. + +###### CLI example +```sql +> CREATE USER "todd" WITH PASSWORD '123456' +> CREATE USER "todd" WITH PASSWORD '123456' +> CREATE USER "todd" WITH PASSWORD '123' +ERR: user already exists +> CREATE USER "todd" WITH PASSWORD '123456' +> CREATE USER "todd" WITH PASSWORD '123456' WITH ALL PRIVILEGES +ERR: user already exists +> CREATE USER "todd" WITH PASSWORD '123456' +> +``` +{{% /note %}} + + +##### `GRANT` `READ`, `WRITE` or `ALL` database privileges to an existing user + +```sql +GRANT [READ,WRITE,ALL] ON TO +``` + +CLI examples: + +`GRANT` `READ` access to `todd` on the `NOAA_water_database` database: + +```sql +> GRANT READ ON "NOAA_water_database" TO "todd" +> +``` + +`GRANT` `ALL` access to `todd` on the `NOAA_water_database` database: + +```sql +> GRANT ALL ON "NOAA_water_database" TO "todd" +> +``` + +##### `REVOKE` `READ`, `WRITE`, or `ALL` database privileges from an existing user + +``` +REVOKE [READ,WRITE,ALL] ON FROM +``` + +CLI examples: + +`REVOKE` `ALL` privileges from `todd` on the `NOAA_water_database` database: + +```sql +> REVOKE ALL ON "NOAA_water_database" FROM "todd" +> +``` + +`REVOKE` `WRITE` privileges from `todd` on the `NOAA_water_database` database: + +```sql +> REVOKE WRITE ON "NOAA_water_database" FROM "todd" +> +``` + +>**Note:** If a user with `ALL` privileges has `WRITE` privileges revoked, they are left with `READ` privileges, and vice versa. + +##### `SHOW` a user's database privileges + +```sql +SHOW GRANTS FOR +``` + +CLI example: + +```sql +> SHOW GRANTS FOR "todd" +database privilege +NOAA_water_database WRITE +another_database_name READ +yet_another_database_name ALL PRIVILEGES +one_more_database_name NO PRIVILEGES +``` + +#### General admin and non-admin user management + +##### Reset a user's password + +```sql +SET PASSWORD FOR = '' +``` + +CLI example: + +```sql +> SET PASSWORD FOR "todd" = 'influxdb4ever' +> +``` + +{{% note %}} +**Note:** The password [string](/influxdb/v1.8/query_language/spec/#strings) must be wrapped in single quotes. +Do not include the single quotes when authenticating requests. + +We recommend avoiding the single quote (`'`) and backslash (`\`) characters in passwords +For passwords that include these characters, escape the special character with a backslash (e.g. (`\'`) when creating the password and when submitting authentication requests. +{{% /note %}} + +##### `DROP` a user + +```sql +DROP USER +``` + +CLI example: + +```sql +> DROP USER "todd" +> +``` + +## Authentication and authorization HTTP errors + +Requests with no authentication credentials or incorrect credentials yield the `HTTP 401 Unauthorized` response. + +Requests by unauthorized users yield the `HTTP 403 Forbidden` response. diff --git a/content/enterprise_influxdb/v1.9/administration/backup-and-restore.md b/content/enterprise_influxdb/v1.9/administration/backup-and-restore.md new file mode 100644 index 000000000..da9a3bc39 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/administration/backup-and-restore.md @@ -0,0 +1,471 @@ +--- +title: Back up and restore InfluxDB Enterprise clusters +description: > + Back up and restore InfluxDB enterprise clusters in case of unexpected data loss. +aliases: + - /enterprise/v1.8/guides/backup-and-restore/ +menu: + enterprise_influxdb_1_9: + name: Back up and restore + weight: 80 + parent: Administration +--- + +## Overview + +When deploying InfluxDB Enterprise in production environments, you should have a strategy and procedures for backing up and restoring your InfluxDB Enterprise clusters to be prepared for unexpected data loss. + +The tools provided by InfluxDB Enterprise can be used to: + +- Provide disaster recovery due to unexpected events +- Migrate data to new environments or servers +- Restore clusters to a consistent state +- Debugging + +Depending on the volume of data to be protected and your application requirements, InfluxDB Enterprise offers two methods, described below, for managing backups and restoring data: + +- [Backup and restore utilities](#backup-and-restore-utilities) — For most applications +- [Exporting and importing data](#exporting-and-importing-data) — For large datasets + +> **Note:** Use the [`backup` and `restore` utilities (InfluxDB OSS 1.5 and later)](/{{< latest "influxdb" "v1" >}}/administration/backup_and_restore/) to: +> +> - Restore InfluxDB Enterprise backup files to InfluxDB OSS instances. +> - Back up InfluxDB OSS data that can be restored in InfluxDB Enterprise clusters. + +## Backup and restore utilities + +InfluxDB Enterprise supports backing up and restoring data in a cluster, +a single database and retention policy, and single shards. +Most InfluxDB Enterprise applications can use the backup and restore utilities. + +Use the `backup` and `restore` utilities to back up and restore between `influxd` +instances with the same versions or with only minor version differences. +For example, you can backup from 1.7.3 and restore on 1.8.2. + +### Backup utility + +A backup creates a copy of the [metastore](/enterprise_influxdb/v1.9/concepts/glossary/#metastore) and [shard](/enterprise_influxdb/v1.9/concepts/glossary/#shard) data at that point in time and stores the copy in the specified directory. + +Or, back up **only the cluster metastore** using the `-strategy only-meta` backup option. For more information, see [perform a metastore only backup](#perform-a-metastore-only-backup). + +All backups include a manifest, a JSON file describing what was collected during the backup. +The filenames reflect the UTC timestamp of when the backup was created, for example: + +- Metastore backup: `20060102T150405Z.meta` (includes usernames and passwords) +- Shard data backup: `20060102T150405Z..tar.gz` +- Manifest: `20060102T150405Z.manifest` + +Backups can be full, metastore only, or incremental, and they are incremental by default: + +- **Full backup**: Creates a copy of the metastore and shard data. +- **Incremental backup**: Creates a copy of the metastore and shard data that have changed since the last incremental backup. If there are no existing incremental backups, the system automatically performs a complete backup. +- **Metastore only backup**: Creates a copy of the metastore data only. + +Restoring different types of backups requires different syntax. +To prevent issues with [restore](#restore-utility), keep full backups, metastore only backups, and incremental backups in separate directories. + +>**Note:** The backup utility copies all data through the meta node that is used to +execute the backup. As a result, performance of a backup and restore is typically limited by the network IO of the meta node. Increasing the resources available to this meta node (such as resizing the EC2 instance) can significantly improve backup and restore performance. + +#### Syntax + +```bash +influxd-ctl [global-options] backup [backup-options] +``` + +> **Note:** The `influxd-ctl backup` command exits with `0` for success and `1` for failure. If the backup fails, output can be directed to a log file to troubleshoot. + +##### Global options + +See the [`influxd-ctl` documentation](/enterprise_influxdb/v1.9/tools/influxd-ctl/#global-options) +for a complete list of the global `influxd-ctl` options. + +##### Backup options + +- `-db `: name of the single database to back up +- `-from `: the data node TCP address to prefer when backing up +- `-strategy`: select the backup strategy to apply during backup + - `incremental`: _**(Default)**_ backup only data added since the previous backup. + - `full` perform a full backup. Same as `-full` + - `only-meta` perform a backup for meta data only: users, roles, + databases, continuous queries, retention policies. Shards are not exported. +- `-full`: perform a full backup. Deprecated in favour of `-strategy=full` +- `-rp `: the name of the single retention policy to back up (must specify `-db` with `-rp`) +- `-shard `: the ID of the single shard to back up + +### Backup examples + +Store the following incremental backups in different directories. +The first backup specifies `-db myfirstdb` and the second backup specifies +different options: `-db myfirstdb` and `-rp autogen`. + +```bash +influxd-ctl backup -db myfirstdb ./myfirstdb-allrp-backup + +influxd-ctl backup -db myfirstdb -rp autogen ./myfirstdb-autogen-backup +``` + +Store the following incremental backups in the same directory. +Both backups specify the same `-db` flag and the same database. + +```bash +influxd-ctl backup -db myfirstdb ./myfirstdb-allrp-backup + +influxd-ctl backup -db myfirstdb ./myfirstdb-allrp-backup +``` + +#### Perform an incremental backup + +Perform an incremental backup into the current directory with the command below. +If there are any existing backups the current directory, the system performs an incremental backup. +If there aren't any existing backups in the current directory, the system performs a backup of all data in InfluxDB. + +```bash +# Syntax +influxd-ctl backup . + +# Example +$ influxd-ctl backup . +Backing up meta data... Done. 421 bytes transferred +Backing up node 7ba671c7644b:8088, db telegraf, rp autogen, shard 4... Done. Backed up in 903.539567ms, 307712 bytes transferred +Backing up node bf5a5f73bad8:8088, db _internal, rp monitor, shard 1... Done. Backed up in 138.694402ms, 53760 bytes transferred +Backing up node 9bf0fa0c302a:8088, db _internal, rp monitor, shard 2... Done. Backed up in 101.791148ms, 40448 bytes transferred +Backing up node 7ba671c7644b:8088, db _internal, rp monitor, shard 3... Done. Backed up in 144.477159ms, 39424 bytes transferred +Backed up to . in 1.293710883s, transferred 441765 bytes +$ ls +20160803T222310Z.manifest 20160803T222310Z.s1.tar.gz 20160803T222310Z.s3.tar.gz +20160803T222310Z.meta 20160803T222310Z.s2.tar.gz 20160803T222310Z.s4.tar.gz +``` + +#### Perform a full backup + +Perform a full backup into a specific directory with the command below. +The directory must already exist. + +```bash +# Sytnax +influxd-ctl backup -full + +# Example +$ influxd-ctl backup -full backup_dir +Backing up meta data... Done. 481 bytes transferred +Backing up node :8088, db _internal, rp monitor, shard 1... Done. Backed up in 33.207375ms, 238080 bytes transferred +Backing up node :8088, db telegraf, rp autogen, shard 2... Done. Backed up in 15.184391ms, 95232 bytes transferred +Backed up to backup_dir in 51.388233ms, transferred 333793 bytes +$ ls backup_dir +20170130T184058Z.manifest +20170130T184058Z.meta +20170130T184058Z.s1.tar.gz +20170130T184058Z.s2.tar.gz +``` + +#### Perform an incremental backup on a single database + +Point at a remote meta server and back up only one database into a given directory (the directory must already exist): + +```bash +# Syntax +influxd-ctl -bind :8091 backup -db + +# Example +$ influxd-ctl -bind 2a1b7a338184:8091 backup -db telegraf ./telegrafbackup +Backing up meta data... Done. 318 bytes transferred +Backing up node 7ba671c7644b:8088, db telegraf, rp autogen, shard 4... Done. Backed up in 997.168449ms, 399872 bytes transferred +Backed up to ./telegrafbackup in 1.002358077s, transferred 400190 bytes +$ ls ./telegrafbackup +20160803T222811Z.manifest 20160803T222811Z.meta 20160803T222811Z.s4.tar.gz +``` + +#### Perform a metastore only backup + +Perform a meta store only backup into a specific directory with the command below. +The directory must already exist. + +```bash +# Syntax +influxd-ctl backup -strategy only-meta + +# Example +$ influxd-ctl backup -strategy only-meta backup_dir +Backing up meta data... Done. 481 bytes transferred +Backed up to backup_dir in 51.388233ms, transferred 481 bytes +~# ls backup_dir +20170130T184058Z.manifest +20170130T184058Z.meta +``` + +### Restore utility + +#### Disable anti-entropy (AE) before restoring a backup + +> Before restoring a backup, stop the anti-entropy (AE) service (if enabled) on **each data node in the cluster, one at a time**. + +> +> 1. Stop the `influxd` service. +> 2. Set `[anti-entropy].enabled` to `false` in the influx configuration file (by default, influx.conf). +> 3. Restart the `influxd` service and wait for the data node to receive read and write requests and for the [hinted handoff queue](/enterprise_influxdb/v1.9/concepts/clustering/#hinted-handoff) to drain. +> 4. Once AE is disabled on all data nodes and each node returns to a healthy state, you're ready to restore the backup. For details on how to restore your backup, see examples below. +> 5. After restoring the backup, restart AE services on each data node. + +##### Restore a backup + +Restore a backup to an existing cluster or a new cluster. +By default, a restore writes to databases using the backed-up data's [replication factor](/enterprise_influxdb/v1.9/concepts/glossary/#replication-factor). +An alternate replication factor can be specified with the `-newrf` flag when restoring a single database. +Restore supports both `-full` backups and incremental backups; the syntax for +a restore differs depending on the backup type. + +##### Restores from an existing cluster to a new cluster + +Restores from an existing cluster to a new cluster restore the existing cluster's +[users](/enterprise_influxdb/v1.9/concepts/glossary/#user), roles, +[databases](/enterprise_influxdb/v1.9/concepts/glossary/#database), and +[continuous queries](/enterprise_influxdb/v1.9/concepts/glossary/#continuous-query-cq) to +the new cluster. + +They do not restore Kapacitor [subscriptions](/enterprise_influxdb/v1.9/concepts/glossary/#subscription). +In addition, restores to a new cluster drop any data in the new cluster's +`_internal` database and begin writing to that database anew. +The restore does not write the existing cluster's `_internal` database to +the new cluster. + +#### Syntax to restore from incremental and metadata backups + +Use the syntax below to restore an incremental or metadata backup to a new cluster or an existing cluster. +**The existing cluster must contain no data in the affected databases.** +Performing a restore from an incremental backup requires the path to the incremental backup's directory. + +```bash +influxd-ctl [global-options] restore [restore-options] +``` + +{{% note %}} +The existing cluster can have data in the `_internal` database (the database InfluxDB creates if +[internal monitoring](/platform/monitoring/influxdata-platform/tools/measurements-internal) is enabled). +The system automatically drops the `_internal` database when it performs a complete restore. +{{% /note %}} + +##### Global options + +See the [`influxd-ctl` documentation](/enterprise_influxdb/v1.9/tools/influxd-ctl/#global-options) +for a complete list of the global `influxd-ctl` options. + +##### Restore options + +- `-db `: the name of the single database to restore +- `-list`: shows the contents of the backup +- `-newdb `: the name of the new database to restore to (must specify with `-db`) +- `-newrf `: the new replication factor to restore to (this is capped to the number of data nodes in the cluster) +- `-newrp `: the name of the new retention policy to restore to (must specify with `-rp`) +- `-rp `: the name of the single retention policy to restore +- `-shard `: the shard ID to restore + +#### Syntax to restore from a full or manifest only backup + +Use the syntax below to restore a full or manifest only backup to a new cluster or an existing cluster. +Note that the existing cluster must contain no data in the affected databases.* +Performing a restore requires the `-full` flag and the path to the backup's manifest file. + +```bash +influxd-ctl [global-options] restore [options] -full +``` + +\* The existing cluster can have data in the `_internal` database, the database +that the system creates by default. +The system automatically drops the `_internal` database when it performs a +complete restore. + +##### Global options + +See the [`influxd-ctl` documentation](/enterprise_influxdb/v1.9/tools/influxd-ctl/#global-options) +for a complete list of the global `influxd-ctl` options. + +##### Restore options + +- `-db `: the name of the single database to restore +- `-list`: shows the contents of the backup +- `-newdb `: the name of the new database to restore to (must specify with `-db`) +- `-newrf `: the new replication factor to restore to (this is capped to the number of data nodes in the cluster) +- `-newrp `: the name of the new retention policy to restore to (must specify with `-rp`) +- `-rp `: the name of the single retention policy to restore +- `-shard `: the shard ID to restore + +#### Examples + +##### Restore from an incremental backup + +```bash +# Syntax +influxd-ctl restore + +# Example +$ influxd-ctl restore my-incremental-backup/ +Using backup directory: my-incremental-backup/ +Using meta backup: 20170130T231333Z.meta +Restoring meta data... Done. Restored in 21.373019ms, 1 shards mapped +Restoring db telegraf, rp autogen, shard 2 to shard 2... +Copying data to :8088... Copying data to :8088... Done. Restored shard 2 into shard 2 in 61.046571ms, 588800 bytes transferred +Restored from my-incremental-backup/ in 83.892591ms, transferred 588800 bytes +``` + +##### Restore from a metadata backup + +In this example, the `restore` command restores an metadata backup stored +in the `metadata-backup/` directory. + +```bash +# Syntax +influxd-ctl restore + +# Example +$ influxd-ctl restore metadata-backup/ +Using backup directory: metadata-backup/ +Using meta backup: 20200101T000000Z.meta +Restoring meta data... Done. Restored in 21.373019ms, 1 shards mapped +Restored from my-incremental-backup/ in 19.2311ms, transferred 588 bytes +``` + +##### Restore from a `-full` backup + +```bash +# Syntax +influxd-ctl restore -full + +# Example +$ influxd-ctl restore -full my-full-backup/20170131T020341Z.manifest +Using manifest: my-full-backup/20170131T020341Z.manifest +Restoring meta data... Done. Restored in 9.585639ms, 1 shards mapped +Restoring db telegraf, rp autogen, shard 2 to shard 2... +Copying data to :8088... Copying data to :8088... Done. Restored shard 2 into shard 2 in 48.095082ms, 569344 bytes transferred +Restored from my-full-backup in 58.58301ms, transferred 569344 bytes +``` + +{{% note %}} +Restoring from a full backup **does not** restore metadata. +To restore metadata, [restore a metadata backup](#restore-from-a-metadata-backup) separately. +{{% /note %}} + +##### Restore from an incremental backup for a single database and give the database a new name + +```bash +# Syntax +influxd-ctl restore -db -newdb + +# Example +$ influxd-ctl restore -db telegraf -newdb restored_telegraf my-incremental-backup/ +Using backup directory: my-incremental-backup/ +Using meta backup: 20170130T231333Z.meta +Restoring meta data... Done. Restored in 8.119655ms, 1 shards mapped +Restoring db telegraf, rp autogen, shard 2 to shard 4... +Copying data to :8088... Copying data to :8088... Done. Restored shard 2 into shard 4 in 57.89687ms, 588800 bytes transferred +Restored from my-incremental-backup/ in 66.715524ms, transferred 588800 bytes +``` + +##### Restore from an incremental backup for a database and merge that database into an existing database + +Your `telegraf` database was mistakenly dropped, but you have a recent backup so you've only lost a small amount of data. + +If Telegraf is still running, it will recreate the `telegraf` database shortly after the database is dropped. +You might try to directly restore your `telegraf` backup just to find that you can't restore: + +```bash +$ influxd-ctl restore -db telegraf my-incremental-backup/ +Using backup directory: my-incremental-backup/ +Using meta backup: 20170130T231333Z.meta +Restoring meta data... Error. +restore: operation exited with error: problem setting snapshot: database already exists +``` + +To work around this, you can restore your telegraf backup into a new database by specifying the `-db` flag for the source and the `-newdb` flag for the new destination: + +```bash +$ influxd-ctl restore -db telegraf -newdb restored_telegraf my-incremental-backup/ +Using backup directory: my-incremental-backup/ +Using meta backup: 20170130T231333Z.meta +Restoring meta data... Done. Restored in 19.915242ms, 1 shards mapped +Restoring db telegraf, rp autogen, shard 2 to shard 7... +Copying data to :8088... Copying data to :8088... Done. Restored shard 2 into shard 7 in 36.417682ms, 588800 bytes transferred +Restored from my-incremental-backup/ in 56.623615ms, transferred 588800 bytes +``` + +Then, in the [`influx` client](/enterprise_influxdb/v1.9/tools/influx-cli/use-influx/), use an [`INTO` query](/enterprise_influxdb/v1.9/query_language/explore-data/#the-into-clause) to copy the data from the new database into the existing `telegraf` database: + +```bash +$ influx +> USE restored_telegraf +Using database restored_telegraf +> SELECT * INTO telegraf..:MEASUREMENT FROM /.*/ GROUP BY * +name: result +------------ +time written +1970-01-01T00:00:00Z 471 +``` + +#### Common issues with restore + +##### Restore writes information not part of the original backup + +If a [restore from an incremental backup](#syntax-to-restore-from-incremental-and-metadata-backups) +does not limit the restore to the same database, retention policy, and shard specified by the backup command, +the restore may appear to restore information that was not part of the original backup. +Backups consist of a shard data backup and a metastore backup. +The **shard data backup** contains the actual time series data: the measurements, tags, fields, and so on. +The **metastore backup** contains user information, database names, retention policy names, shard metadata, continuous queries, and subscriptions. + +When the system creates a backup, the backup includes: + +* the relevant shard data determined by the specified backup options +* all of the metastore information in the cluster regardless of the specified backup options + +Because a backup always includes the complete metastore information, a restore that doesn't include the same options specified by the backup command may appear to restore data that were not targeted by the original backup. +The unintended data, however, include only the metastore information, not the shard data associated with that metastore information. + +##### Restore a backup created prior to version 1.2.0 + +InfluxDB Enterprise introduced incremental backups in version 1.2.0. +To restore a backup created prior to version 1.2.0, be sure to follow the syntax +for [restoring from a full backup](#restore-from-a-full-backup). + +## Exporting and importing data + +For most InfluxDB Enterprise applications, the [backup and restore utilities](#backup-and-restore-utilities) provide the tools you need for your backup and restore strategy. However, in some cases, the standard backup and restore utilities may not adequately handle the volumes of data in your application. + +As an alternative to the standard backup and restore utilities, use the InfluxDB `influx_inspect export` and `influx -import` commands to create backup and restore procedures for your disaster recovery and backup strategy. These commands can be executed manually or included in shell scripts that run the export and import operations at scheduled intervals (example below). + +### Exporting data + +Use the [`influx_inspect export` command](/{{< latest "influxdb" "v1" >}}/tools/influx_inspect#export) to export data in line protocol format from your InfluxDB Enterprise cluster. Options include: + +- Exporting all, or specific, databases +- Filtering with starting and ending timestamps +- Using gzip compression for smaller files and faster exports + +For details on optional settings and usage, see [`influx_inspect export` command](/{{< latest "influxdb" "v1" >}}/tools/influx_inspect#export). + +In the following example, the database is exported filtered to include only one day and compressed for optimal speed and file size. + +```bash +influx_inspect export -database myDB -compress -start 2019-05-19T00:00:00.000Z -end 2019-05-19T23:59:59.999Z +``` + +### Importing data + +After exporting the data in line protocol format, you can import the data using the [`influx -import` CLI command](/{{< latest "influxdb" "v1" >}}/tools/use-influx/#import). + +In the following example, the compressed data file is imported into the specified database. + +```bash +influx -import -database myDB -compress +``` + +For details on using the `influx -import` command, see [Import data from a file with -import](/{{< latest "influxdb" "v1" >}}/tools/use-influx/#import-data-from-a-file-with-import). + +### Example + +For an example of using the exporting and importing data approach for disaster recovery, see the Capital One presentation from Influxdays 2019 on ["Architecting for Disaster Recovery."](https://www.youtube.com/watch?v=LyQDhSdnm4A). In this presentation, Capital One discusses the following: + +- Exporting data every 15 minutes from an active cluster to an AWS S3 bucket. +- Replicating the export file in the S3 bucket using the AWS S3 copy command. +- Importing data every 15 minutes from the AWS S3 bucket to a cluster available for disaster recovery. +- Advantages of the export-import approach over the standard backup and restore utilities for large volumes of data. +- Managing users and scheduled exports and imports with a custom administration tool. diff --git a/content/enterprise_influxdb/v1.9/administration/cluster-commands.md b/content/enterprise_influxdb/v1.9/administration/cluster-commands.md new file mode 100644 index 000000000..5192388c3 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/administration/cluster-commands.md @@ -0,0 +1,33 @@ +--- +title: Manage InfluxDB Enterprise clusters +description: > + Use the `influxd-ctl` and `influx` command line tools to manage InfluxDB Enterprise clusters and data. +aliases: + - /enterprise/v1.8/features/cluster-commands/ + - /enterprise_influxdb/v1.9/features/cluster-commands/ +menu: + enterprise_influxdb_1_9: + name: Manage clusters + weight: 40 + parent: Administration +--- + +Use the following tools to manage and interact with your InfluxDB Enterprise clusters: + +- To manage clusters and nodes, back up and restore data, and rebalance clusters, use the [`influxd-ctl` cluster management utility](#influxd-ctl-cluster-management-utility) +- To write and query data, use the [`influx` command line interface (CLI)](#influx-command-line-interface-cli) + +## `influxd-ctl` cluster management utility + +The [`influxd-ctl`](/enterprise_influxdb/v1.9/tools/influxd-ctl/) utility provides commands for managing your InfluxDB Enterprise clusters. +Use the `influxd-ctl` cluster management utility to manage your cluster nodes, back up and restore data, and rebalance clusters. +The `influxd-ctl` utility is available on all [meta nodes](/enterprise_influxdb/v1.9/concepts/glossary/#meta-node). + +For more information, see [`influxd-ctl`](/enterprise_influxdb/v1.9/tools/influxd-ctl/). + +## `influx` command line interface (CLI) + +Use the `influx` command line interface (CLI) to write data to your cluster, query data interactively, and view query output in different formats. +The `influx` CLI is available on all [data nodes](/enterprise_influxdb/v1.9/concepts/glossary/#data-node). + +See [InfluxDB command line interface (CLI/shell)](/enterprise_influxdb/v1.9/tools/use-influx/) for details on using the `influx` command line interface. diff --git a/content/enterprise_influxdb/v1.9/administration/config-data-nodes.md b/content/enterprise_influxdb/v1.9/administration/config-data-nodes.md new file mode 100644 index 000000000..6ffc7cb65 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/administration/config-data-nodes.md @@ -0,0 +1,1366 @@ +--- +title: Configure InfluxDB Enterprise data nodes +description: > + Configure InfluxDB Enterprise data node settings and environmental variables. +menu: + enterprise_influxdb_1_9: + name: Configure data nodes + weight: 20 + parent: Administration +--- + +* [Data node configuration settings](#data-node-configuration-settings) + * [Global](#global-settings) + * [Enterprise license [enterprise]](#enterprise-license-settings) + * [Meta node `[meta]`](#meta-node-settings) + * [Data `[data]`](#data-settings) + * [Cluster `[cluster]`](#cluster-settings) + * [Retention `[retention]`](#retention-policy-settings) + * [Hinted Handoff `[hinted-handoff]`](#hinted-handoff-settings) + * [Anti-Entropy `[anti-entropy]`](#anti-entropy-ae-settings) + * [Shard precreation `[shard-precreation]`](#shard-precreation-settings) + * [Monitor `[monitor]`](#monitor-settings) + * [HTTP endpoints `[http]`](#http-endpoint-settings) + * [Logging `[logging]`](#logging-settings) + * [Subscriber `[subscriber]`](#subscriber-settings) + * [Graphite `[graphite]`](#graphite-settings) + * [Collectd `[collectd]`](#collectd-settings) + * [OpenTSDB `[opentsdb]`](#opentsdb-settings) + * [UDP `[udp]`](#udp-settings) + * [Continuous queries `[continuous-queries]`](#continuous-queries-settings) + * [TLS `[tls]`](#tls-settings) + * [InfluxQL Query controls `[coordinator]`](#coordinator) + * [Flux Query controls `[flux-controller]`](#flux-controller) + +## Data node configuration settings + +The InfluxDB Enterprise data node configuration settings overlap significantly +with the settings in InfluxDB OSS. + +> **Note:** +The system has internal defaults for every configuration file setting. +View the default settings with the `influxd config` command. +The local configuration file (`/etc/influxdb/influxdb.conf`) overrides any +internal defaults but the configuration file does not need to include +every configuration setting. +Starting with version 1.0.1, most of the settings in the local configuration +file are commented out. +All commented-out settings will be determined by the internal defaults. + +----- + +## Global settings + +#### `reporting-disabled = false` + +Once every 24 hours InfluxDB Enterprise will report usage data to usage.influxdata.com. +The data includes a random ID, os, arch, version, the number of series and other usage data. No data from user databases is ever transmitted. +Change this option to true to disable reporting. + +#### `bind-address = ":8088"` + +The TCP bind address used by the RPC service for inter-node communication and [backup and restore](/enterprise_influxdb/v1.9/administration/backup-and-restore/). + +Environment variable: `INFLUXDB_BIND_ADDRESS` + +#### `hostname = "localhost"` + +The hostname of the [data node](/enterprise_influxdb/v1.9/concepts/glossary/#data-node). This must be resolvable by all other nodes in the cluster. + +Environment variable: `INFLUXDB_HOSTNAME` + +#### `gossip-frequency = "3s"` + +How often to update the cluster with this node's internal status. + +Environment variable: `INFLUXDB_GOSSIP_FREQUENCY` + +----- + +## Enterprise license settings + +### `[enterprise]` + +The `[enterprise]` section contains the parameters for the meta node's registration with the [InfluxDB Enterprise License Portal](https://portal.influxdata.com/). + +#### `license-key = ""` + +The license key created for you on [InfluxPortal](https://portal.influxdata.com). The meta node transmits the license key to [portal.influxdata.com](https://portal.influxdata.com) over port 80 or port 443 and receives a temporary JSON license file in return. +The server caches the license file locally. +The data process will only function for a limited time without a valid license file. +You must use the [`license-path` setting](#license-path) if your server cannot communicate with [https://portal.influxdata.com](https://portal.influxdata.com). + +{{% warn %}} +Use the same key for all nodes in the same cluster. +The `license-key` and `license-path` settings are +mutually exclusive and one must remain set to the empty string. +{{% /warn %}} + +> **Note:** You must trigger data nodes to reload your configuration. For more information, see how to [renew or update your license key](/enterprise_influxdb/v1.9/administration/renew-license/). + +Environment variable: `INFLUXDB_ENTERPRISE_LICENSE_KEY` + +#### `license-path = ""` + +The local path to the permanent JSON license file that you received from InfluxData for instances that do not have access to the internet. +The data process will only function for a limited time without a valid license file. +Contact [sales@influxdb.com](mailto:sales@influxdb.com) if a license file is required. + +The license file should be saved on every server in the cluster, including Meta, Data, and Enterprise nodes. +The file contains the JSON-formatted license, and must be readable by the `influxdb` user. Each server in the cluster independently verifies its license. + +> **Note:** You must trigger data nodes to reload your configuration. For more information, see how to [renew or update your license key](/enterprise_influxdb/v1.9/administration/renew-license/). + +{{% warn %}} +Use the same license file for all nodes in the same cluster. +The `license-key` and `license-path` settings are mutually exclusive and one must remain set to the empty string. +{{% /warn %}} + +Environment variable: `INFLUXDB_ENTERPRISE_LICENSE_PATH` + +----- + +## Meta node settings + +### `[meta]` + +Settings related to how the data nodes interact with the meta nodes. + +#### `dir = "/var/lib/influxdb/meta"` + +The directory where the cluster metadata is stored. + +> **Note:** Data nodes do require a local meta directory. + +Environment variable: `INFLUXDB_META_DIR` + +#### `meta-tls-enabled = false` + +Whether to use TLS when connecting to meta nodes. +Set to `true` to if [`https-enabled`](#https-enabled-false) is set to `true`. + +Environment variable: `INFLUXDB_META_META_TLS_ENABLED` + +#### `meta-insecure-tls = false` + +Allows insecure TLS connections to meta nodes. +This is useful when testing with self-signed certificates. + +Set to `true` to allow the data node to accept self-signed certificates if [`https-enabled`](#https-enabled-false) is set to `true`. + +Environment variable: `INFLUXDB_META_META_INSECURE_TLS` + +#### `meta-auth-enabled = false` + +This setting must have the same value as the meta nodes' `[meta] auth-enabled` configuration. + +Set to `true` if [`auth-enabled`](#auth-enabled-false) is set to `true` in the meta node configuration files. +For JWT authentication, also see the [`meta-internal-shared-secret`](#meta-internal-shared-secret) configuration option. + +Environment variable: `INFLUXDB_META_META_AUTH_ENABLED` + +#### `meta-internal-shared-secret = ""` + +The shared secret used by the internal API for JWT authentication between InfluxDB nodes. +This value must be the same as the [`internal-shared-secret`](/enterprise_influxdb/v1.9/administration/config-meta-nodes/#internal-shared-secret) specified in the meta node configuration file. + +Environment variable: `INFLUXDB_META_META_INTERNAL_SHARED_SECRET` + +#### `retention-autocreate = true` + +Automatically creates a default [retention policy](/enterprise_influxdb/v1.9/concepts/glossary/#retention-policy-rp) (RP) when the system creates a database. +The default RP (`autogen`) has an infinite duration, a shard group duration of seven days, and a replication factor set to the number of data nodes in the cluster. +The system targets the `autogen` RP when a write or query does not specify an RP. +Set this option to `false` to prevent the system from creating the `autogen` RP when the system creates a database. + +Environment variable: `INFLUXDB_META_RETENTION_AUTOCREATE` + +#### `logging-enabled = true` + +Whether log messages are printed for the meta service. + +Environment variable: `INFLUXDB_META_LOGGING_ENABLED` + +----- + +## Data settings + +### `[data]` + +Controls where the actual shard data for InfluxDB lives and how it is compacted from the WAL. +"dir" may need to be changed to a suitable place for your system. +The defaults should work for most systems. + +For InfluxDB OSS, see the [OSS documentation](/enterprise_influxdb/v1.9/administration/config/#data-settings). + +#### `dir = "/var/lib/influxdb/data"` + +The directory where the TSM storage engine stores TSM (read-optimized) files. + +Environment variable: `INFLUXDB_DATA_DIR` + +#### `wal-dir = "/var/lib/influxdb/wal"` + +The directory where the TSM storage engine stores WAL (write-optimized) files. + +Environment variable: `INFLUXDB_DATA_WAL_DIR` + +#### `trace-logging-enabled = false` + +Trace logging provides more verbose output around the TSM engine. +Turning this on can provide more useful output for debugging TSM engine issues. + +Environmental variable: `INFLUXDB_DATA_TRACE_LOGGING_ENABLED` + +#### `query-log-enabled = true` + +Whether queries should be logged before execution. +Very useful for troubleshooting, but will log any sensitive data contained within a query. + +Environment variable: `INFLUXDB_DATA_QUERY_LOG_ENABLED` + +#### `wal-fsync-delay = "0s"` + +The amount of time that a write waits before fsyncing. +Use a duration greater than 0 to batch up multiple fsync calls. +This is useful for slower disks or when experiencing WAL write contention. +A value of `0s` fsyncs every write to the WAL. +InfluxData recommends values ranging from `0ms` to `100ms` for non-SSD disks. + +Environment variable: `INFLUXDB_DATA_WAL_FSYNC_DELAY` + +#### `ingress-metric-by-measurement-enabled = false` + +When `true`, collect statistics of points, values and new series written per-measurement. Metrics are gathered per data node. +These can be accessed via the `/debug/vars` endpoint and in the `_internal` database if enabled. + +Environment variable: `INFLUXDB_DATA_INGRESS_METRIC_BY_MEASUREMENT_ENABLED` + +#### `ingress-metric-by-login-enabled = false` + +When `true`, collect statistics of points, values and new series written per-login. Metrics are gathered per data node. +These can be accessed via the `/debug/vars` endpoint and in the `_internal` database if enabled. + +Environment variable: `INFLUXDB_DATA_INGRESS_METRIC_BY_LOGIN_ENABLED` + +### Data settings for the TSM engine + +#### `cache-max-memory-size = "1g"` + +The maximum size a shard cache can reach before it starts rejecting writes. + +Environment variable: `INFLUXDB_DATA_CACHE_MAX_MEMORY_SIZE` + +#### `cache-snapshot-memory-size = "25m"` + +The size at which the TSM engine will snapshot the cache and write it to a TSM file, freeing up memory. + +Environment variable: `INFLUXDB_DATA_CACHE_SNAPSHOT_MEMORY_SIZE` + +#### `cache-snapshot-write-cold-duration = "10m"` + +The length of time at which the TSM engine will snapshot the cache and write it to a new TSM file if the shard hasn't received writes or deletes. + +Environment variable: `INFLUXDB_DATA_CACHE_SNAPSHOT_WRITE_COLD_DURATION` + +#### `max-concurrent-compactions = 0` + +The maximum number of concurrent full and level compactions that can run at one time. +A value of `0` (unlimited compactions) results in 50% of `runtime.GOMAXPROCS(0)` used at runtime, +so when 50% of the CPUs aren't available, compactions are limited. +Any number greater than `0` limits compactions to that value. +This setting does not apply to cache snapshotting. + +Environmental variable: `INFLUXDB_DATA_CACHE_MAX_CONCURRENT_COMPACTIONS` + +#### `compact-throughput = "48m"` + +The maximum number of bytes per seconds TSM compactions write to disk. Default is `"48m"` (48 million). +Note that short bursts are allowed to happen at a possibly larger value, set by `compact-throughput-burst`. + +Environment variable: `INFLUXDB_DATA_COMPACT_THROUGHPUT` + +#### `compact-throughput-burst = "48m"` + +The maximum number of bytes per seconds TSM compactions write to disk during brief bursts. Default is `"48m"` (48 million). + +Environment variable: `INFLUXDB_DATA_COMPACT_THROUGHPUT_BURST` + +#### `compact-full-write-cold-duration = "4h"` + +The duration at which the TSM engine will compact all TSM files in a shard if it hasn't received a write or delete. + +Environment variable: `INFLUXDB_DATA_COMPACT_FULL_WRITE_COLD_DURATION` + +#### `index-version = "inmem"` + +The type of shard index to use for new shards. +The default (`inmem`) is to use an in-memory index that is recreated at startup. +A value of `tsi1` will use a disk-based index that supports higher cardinality datasets. +Value should be enclosed in double quotes. + +Environment variable: `INFLUXDB_DATA_INDEX_VERSION` + +### In-memory (`inmem`) index settings + +#### `max-series-per-database = 1000000` + +The maximum number of [series](/enterprise_influxdb/v1.9/concepts/glossary/#series) allowed per database before writes are dropped. +The default setting is `1000000` (one million). +Change the setting to `0` to allow an unlimited number of series per database. + +If a point causes the number of series in a database to exceed +`max-series-per-database`, InfluxDB will not write the point, and it returns a +`500` with the following error: + +```bash +{"error":"max series per database exceeded: "} +``` + +> **Note:** Any existing databases with a series count that exceeds `max-series-per-database` +> will continue to accept writes to existing series, but writes that create a +> new series will fail. + +Environment variable: `INFLUXDB_DATA_MAX_SERIES_PER_DATABASE` + +#### `max-values-per-tag = 100000` + +The maximum number of [tag values](/enterprise_influxdb/v1.9/concepts/glossary/#tag-value) allowed per [tag key](/enterprise_influxdb/v1.9/concepts/glossary/#tag-key). +The default value is `100000` (one hundred thousand). +Change the setting to `0` to allow an unlimited number of tag values per tag +key. +If a tag value causes the number of tag values of a tag key to exceed +`max-values-per-tag`, then InfluxDB will not write the point, and it returns +a `partial write` error. + +Any existing tag keys with tag values that exceed `max-values-per-tag` +will continue to accept writes, but writes that create a new tag value +will fail. + +Environment variable: `INFLUXDB_DATA_MAX_VALUES_PER_TAG` + +### TSI (`tsi1`) index settings + +#### `max-index-log-file-size = "1m"` + +The threshold, in bytes, when an index write-ahead log (WAL) file will compact +into an index file. Lower sizes will cause log files to be compacted more +quickly and result in lower heap usage at the expense of write throughput. +Higher sizes will be compacted less frequently, store more series in-memory, +and provide higher write throughput. +Valid size suffixes are `k`, `m`, or `g` (case-insensitive, 1024 = 1k). +Values without a size suffix are in bytes. + +Environment variable: `INFLUXDB_DATA_MAX_INDEX_LOG_FILE_SIZE` + +#### `series-id-set-cache-size = 100` + +The size of the internal cache used in the TSI index to store previously +calculated series results. Cached results will be returned quickly from the cache rather +than needing to be recalculated when a subsequent query with a matching tag key-value +predicate is executed. +Setting this value to `0` will disable the cache, which may lead to query performance issues. +This value should only be increased if it is known that the set of regularly used +tag key-value predicates across all measurements for a database is larger than 100. An +increase in cache size may lead to an increase in heap usage. + +Environment variable: `INFLUXDB_DATA_SERIES_ID_SET_CACHE_SIZE` + +----- + +## Cluster settings + +### `[cluster]` + +Settings related to how the data nodes interact with other data nodes. +Controls how data is shared across shards and the options for query management. + +An InfluxDB Enterprise cluster uses remote procedure calls (RPCs) for inter-node communication. +An RPC connection pool manages the stream connections and efficiently uses system resources. +InfluxDB data nodes multiplex RPC streams over a single TCP connection to avoid the overhead of +frequently establishing and destroying TCP connections and exhausting ephemeral ports. +Typically, a data node establishes a single, persistent TCP connection to each of the other data nodes +to perform most RPC requests. In special circumstances, for example, when copying shards, +a single-use TCP connection may be used. + +For information on InfluxDB `_internal` measurement statistics related to clusters, RPCs, and shards, +see [Measurements for monitoring InfluxDB OSS and InfluxDB Enterprise (`_inernal`)](/platform/monitoring/influxdata-platform/tools/measurements-internal/#cluster-enterprise-only). + +#### `dial-timeout = "1s"` + +The duration for which the meta node waits for a connection to a remote data node before the meta node attempts to connect to a different remote data node. +This setting applies to queries only. + +Environment variable: `INFLUXDB_CLUSTER_DIAL_TIMEOUT` + +#### `pool-max-idle-time = "60s"` + +The maximum time that a TCP connection to another data node remains idle in the connection pool. +When the connection is idle longer than the specified duration, the inactive connection is reaped — +retired or recycled — so that the connection pool is not filled with inactive connections. Reaping +idle connections minimizes inactive connections, decreases system load, and prevents system failure. + +Environment variable: `INFLUXDB_CLUSTER_POOL_MAX_IDLE_TIME` + +#### `pool-max-idle-streams = 100` + +The maximum number of idle RPC stream connections to retain in an idle pool between two nodes. +When a new RPC request is issued, a connection is temporarily pulled from the idle pool, used, and then returned. +If an idle pool is full and a stream connection is no longer required, the system closes the stream connection and resources become available. +The number of active streams can exceed the maximum number of idle pool connections, +but are not returned to the idle pool when released. +Creating streams are relatively inexpensive operations to perform, +so it is unlikely that changing this value will measurably improve performance between two nodes. + +Environment variable: `INFLUXDB_CLUSTER_POOL_MAX_IDLE_STREAMS` + +#### allow-out-of-order = "false" + +By default, this option is set to false and writes are processed in the order that they are received. This means if any points are in the hinted handoff (HH) queue for a shard, all incoming points must go into the HH queue. + +If true, writes may process in a different order than they were received. This can reduce the time required to drain the HH queue and increase throughput during recovery. + +**Do not enable if your use case involves updating points, which may cause points to be overwritten.** To overwrite an existing point, the measurement name, tag keys and values (if the point includes tags), field keys, and timestamp all have to be the same as a previous write. + +For example, if you have two points with the same measurement (`cpu`), field key (`v`), and timestamp (`1234`), the following could happen: + +Point 1 (`cpu v=1.0 1234`) arrives at `node1`, attempts to replicate on `node2`, and finds `node2` is down, so point 1 goes to the local HH queue. Now, `node2` comes back online and point 2 `cpu v=20. 1234` arrives at `node1`, overwrites point 1, and is written to `node2` (bypassing the HH queue). Because the point 2 arrives at `node2` before point 1, point 2 is stored before point 1. + +Environment variable: `INFLUXDB_CLUSTER_ALLOW_OUT_OF_ORDER` + +#### `shard-reader-timeout = "0"` + +The default timeout set on shard readers. +The time in which a query connection must return its response after which the system returns an error. + +Environment variable: `INFLUXDB_CLUSTER_SHARD_READER_TIMEOUT` + +#### `https-enabled = false` + +Determines whether data nodes use HTTPS to communicate with each other. + +#### `https-certificate = ""` + +The SSL certificate to use when HTTPS is enabled. +The certificate should be a PEM-encoded bundle of the certificate and key. +If it is just the certificate, a key must be specified in `https-private-key`. + +#### `https-private-key = ""` + +Use a separate private key location. + +#### `https-insecure-tls = false` + +Whether data nodes will skip certificate validation communicating with each other over HTTPS. +This is useful when testing with self-signed certificates. + +#### `cluster-tracing = false` + +Enables cluster trace logging. +Set to `true` to enable logging of cluster communications. +Enable this setting to verify connectivity issues between data nodes. + +Environment variable: `INFLUXDB_CLUSTER_CLUSTER_TRACING` + +#### `write-timeout = "10s"` + +The duration a write request waits until a "timeout" error is returned to the caller. The default value is 10 seconds. + +Environment variable: `INFLUXDB_CLUSTER_WRITE_TIMEOUT` + +#### `max-concurrent-queries = 0` + +The maximum number of concurrent queries allowed to be executing at one time. +If a query is executed and exceeds this limit, an error is returned to the caller. +This limit can be disabled by setting it to `0`. + +Environment variable: `INFLUXDB_CLUSTER_MAX_CONCURRENT_QUERIES` + +#### `query-timeout = "0s"` + +The maximum time a query is allowed to execute before being killed by the system. +This limit can help prevent run away queries. Setting the value to `0` disables the limit. + +Environment variable: `INFLUXDB_CLUSTER_QUERY_TIMEOUT` + +#### `log-queries-after = "0s"` + +The time threshold when a query will be logged as a slow query. +This limit can be set to help discover slow or resource intensive queries. +Setting the value to `0` disables the slow query logging. + +Environment variable: `INFLUXDB_CLUSTER_LOG_QUERIES_AFTER` + +#### `max-select-point = 0` + +The maximum number of points a SELECT statement can process. +A value of `0` will make the maximum point count unlimited. + +Environment variable: `INFLUXDB_CLUSTER_MAX_SELECT_POINT` + +#### `max-select-series = 0` + +The maximum number of series a SELECT can run. +A value of `0` will make the maximum series count unlimited. + +Environment variable: `INFLUXDB_CLUSTER_MAX_SELECT_SERIES` + +#### `max-select-buckets = 0` + +The maximum number of group by time buckets a SELECT can create. +A value of `0` will make the maximum number of buckets unlimited. + +Environment variable: `INFLUXDB_CLUSTER_MAX_SELECT_BUCKETS` + +----- + +## Hinted Handoff settings + +### `[hinted-handoff]` + +Controls the hinted handoff (HH) queue, which allows data nodes to temporarily cache writes destined for another data node when that data node is unreachable. + +#### `batch-size = 512000` + +The maximum number of bytes to write to a shard in a single request. + +Environment variable: `INFLUXDB_HINTED_HANDOFF_BATCH_SIZE` + +#### `max-pending-writes = 1024` + +The maximum number of incoming pending writes allowed in the hinted handoff queue. + +Environment variable: `INFLUXDB_HINTED_HANDOFF_MAX_PENDING_WRITES` + +#### `dir = "/var/lib/influxdb/hh"` + +The hinted handoff directory where the durable queue will be stored on disk. + +Environment variable: `INFLUXDB_HINTED_HANDOFF_DIR` + +#### `enabled = true` + +Set to `false` to disable hinted handoff. +Disabling hinted handoff is not recommended and can lead to data loss if another data node is unreachable for any length of time. + +Environment variable: `INFLUXDB_HINTED_HANDOFF_ENABLED` + +#### `max-size = 10737418240` + +The maximum size of the hinted handoff queue in bytes. +Each queue is for one and only one other data node in the cluster. +If there are N data nodes in the cluster, each data node may have up to N-1 hinted handoff queues. + +Environment variable: `INFLUXDB_HINTED_HANDOFF_MAX_SIZE` + +#### `max-age = "168h0m0s"` + +The time interval that writes sit in the queue before they are purged. +The time is determined by how long the batch has been in the queue, not by the timestamps in the data. +If another data node is unreachable for more than the `max-age` it can lead to data loss. + +Environment variable: `INFLUXDB_HINTED_HANDOFF_MAX_AGE` + +#### `retry-concurrency = 20` + +The maximum number of hinted handoff blocks that the source data node attempts to write to each destination data node. +Hinted handoff blocks are sets of data that belong to the same shard and have the same destination data node. + +If `retry-concurrency` is 20 and the source data node's hinted handoff has 25 blocks for destination data node A, then the source data node attempts to concurrently write 20 blocks to node A. +If `retry-concurrency` is 20 and the source data node's hinted handoff has 25 blocks for destination data node A and 30 blocks for destination data node B, then the source data node attempts to concurrently write 20 blocks to node A and 20 blocks to node B. +If the source data node successfully writes 20 blocks to a destination data node, it continues to write the remaining hinted handoff data to that destination node in sets of 20 blocks. + +If the source data node successfully writes data to destination data nodes, a higher `retry-concurrency` setting can accelerate the rate at which the source data node empties its hinted handoff queue. + +Note that increasing `retry-concurrency` also increases network traffic. + +Environment variable: `INFLUXDB_HINTED_HANDOFF_RETRY_CONCURRENCY` + +#### `retry-rate-limit = 0` + +The rate limit (in bytes per second) that hinted handoff retries hints. A value of `0` disables the rate limit. + +Environment variable: `INFLUXDB_HINTED_HANDOFF_RETRY_RATE_LIMIT` + +#### `retry-interval = "1s"` + +The time period after which the hinted handoff retries a write after the write fails. + +Environment variable: `INFLUXDB_HINTED_HANDOFF_RETRY_INTERVAL` + +#### `retry-max-interval = "10s"` + +The maximum interval after which the hinted handoff retries a write after the write fails. +The `retry-max-interval` option is no longer in use and will be removed from the configuration file in a future release. +Changing the `retry-max-interval` setting has no effect on your cluster. + +Environment variable: `INFLUXDB_HINTED_HANDOFF_RETRY_MAX_INTERVAL` + +#### `purge-interval = "1m0s"` + +The interval at which InfluxDB checks to purge data that are above `max-age`. + +Environment variable: `INFLUXDB_HINTED_HANDOFF_PURGE_INTERVAL` + +----- + +## Anti-Entropy (AE) settings + +For information about the Anti-Entropy service, see [Anti-entropy service in InfluxDB Enterprise](/enterprise_influxdb/v1.9/administration/anti-entropy). + +### `[anti-entropy]` + +Controls the copying and repairing of shards to ensure that data nodes contain the shard data they are supposed to. + +#### `enabled = false` + +Enables the anti-entropy service. +Default value is `false`. + +Environment variable: `INFLUXDB_ANTI_ENTROPY_ENABLED` + +#### `check-interval = "5m"` + +The interval of time when anti-entropy checks run on each data node. + +Environment variable: `INFLUXDB_ANTI_ENTROPY_CHECK_INTERVAL` + +#### `max-fetch = 10` + +The maximum number of shards that a single data node will copy or repair in parallel. + +Environment variable: `INFLUXDB_ANTI_ENTROPY_MAX_FETCH` + +{{% note %}} +Having `max-fetch=10` with higher numbers of shards (100+) can add significant overhead to running nodes. +The more shards you have, the lower this should be set. +If AE is enabled while lowering your `max-fetch`, initially, you'll see +higher CPU load as new shard digest files are created. +The added load drops off after shard digests are completed for existing shards. +{{% /note %}} + +#### `max-sync = 1` + +The maximum number of concurrent sync operations that should be performed. +Modify this setting only when requested by InfluxData support. + +Environment variable: `INFLUXDB_ANTI_ENTROPY_MAX_SYNC` + +#### `auto-repair-missing = true` + +Enables missing shards to automatically be repaired. + +Environment variable: `INFLUXDB_ANTI_ENTROPY_AUTO_REPAIR_MISSING` + +----- + +## Retention policy settings + +### `[retention]` + +Controls the enforcement of retention policies for evicting old data. + +#### `enabled = true` + +Enables retention policy enforcement. +Default value is `true`. + +Environment variable: `INFLUXDB_RETENTION_ENABLED` + +#### `check-interval = "30m0s"` + +The interval of time when retention policy enforcement checks run. + +Environment variable: `INFLUXDB_RETENTION_CHECK_INTERVAL` + +----- + +## Shard precreation settings + +### `[shard-precreation]` + +Controls the precreation of shards, so they are available before data arrives. +Only shards that, after creation, will have both a start- and end-time in the future, will ever be created. Shards are never precreated that would be wholly or partially in the past. + +#### `enabled = true` + +Enables the shard precreation service. + +Environment variable: `INFLUXDB_SHARD_PRECREATION_ENABLED` + +#### `check-interval = "10m"` + +The interval of time when the check to precreate new shards runs. + +Environment variable: `INFLUXDB_SHARD_PRECREATION_CHECK_INTERVAL` + +#### `advance-period = "30m"` + +The default period ahead of the end time of a shard group that its successor group is created. + +Environment variable: `INFLUXDB_SHARD_PRECREATION_ADVANCE_PERIOD` + +----- + +## Monitor settings + +### `[monitor]` + +By default, InfluxDB writes system monitoring data to the `_internal` database. +If that database does not exist, InfluxDB creates it automatically. +The `DEFAULT` retention policy on the `internal` database is seven days. +To change the default seven-day retention policy, you must [create](/enterprise_influxdb/v1.9/query_language/manage-database/#retention-policy-management) it. + +For InfluxDB Enterprise production systems, InfluxData recommends including a dedicated InfluxDB (OSS) monitoring instance for monitoring InfluxDB Enterprise cluster nodes. + +* On the dedicated InfluxDB monitoring instance, set `store-enabled = false` to avoid potential performance and storage issues. +* On each InfluxDB cluster node, install a Telegraf input plugin and Telegraf output plugin configured to report data to the dedicated InfluxDB monitoring instance. + +#### `store-enabled = true` + +Enables the internal storage of statistics. + +Environment variable: `INFLUXDB_MONITOR_STORE_ENABLED` + +#### `store-database = "_internal"` + +The destination database for recorded statistics. + +Environment variable: `INFLUXDB_MONITOR_STORE_DATABASE` + +#### `store-interval = "10s"` + +The interval at which to record statistics. + +Environment variable: `INFLUXDB_MONITOR_STORE_INTERVAL` + +#### `remote-collect-interval = "10s"` + +The time interval to poll other data nodes' stats when aggregating cluster stats. + +Environment variable: `INFLUXDB_MONITOR_REMOTE_COLLECT_INTERVAL` + +----- + +## HTTP endpoint settings + +### `[http]` + +Controls how the HTTP endpoints are configured. These are the primary mechanism for getting data into and out of InfluxDB. + +For InfluxDB OSS, see the [OSS documentation](/enterprise_influxdb/v1.9/administration/config/#http-endpoint-settings-http). + +#### `enabled = true` + +Enables HTTP endpoints. + +Environment variable: `INFLUXDB_HTTP_ENABLED` + +#### `flux-enabled = false` + +Determines whether the Flux query endpoint is enabled. To enable the use of Flux queries, set the value to `true`. + +Environment variable: `INFLUXDB_HTTP_FLUX_ENABLED` + +#### `bind-address = ":8086"` + +The bind address used by the HTTP service. + +Environment variable: `INFLUXDB_HTTP_BIND_ADDRESS` + +#### `auth-enabled = false` + +Enables HTTP authentication. + +Environment variable: `INFLUXDB_HTTP_AUTH_ENABLED` + +#### `realm = "InfluxDB"` + +The default realm sent back when issuing a basic authorization challenge. + +Environment variable: `INFLUXDB_HTTP_REALM` + +#### `log-enabled = true` + +Enables HTTP request logging. + +Environment variable: `INFLUXDB_HTTP_LOG_ENABLED` + +#### `suppress-write-log = false` + +Determines whether the HTTP write request logs should be suppressed when the log is enabled. + +#### `access-log-path = ""` + +The path to the access log, which determines whether detailed write logging is enabled using `log-enabled = true`. +Specifies whether HTTP request logging is written to the specified path when enabled. +If `influxd` is unable to access the specified path, it will log an error and fall back to `stderr`. +When HTTP request logging is enabled, this option specifies the path where log entries should be written. +If unspecified, the default is to write to stderr, which intermingles HTTP logs with internal InfluxDB logging. +If `influxd` is unable to access the specified path, it will log an error and fall back to writing the request log to `stderr`. + +Environment variable: `INFLUXDB_HTTP_ACCESS_LOG_PATH` + +#### `access-log-status-filters = []` + +Filters which requests should be logged. Each filter is of the pattern `nnn`, `nnx`, or `nxx` where `n` is +a number and `x` is the wildcard for any number. +To filter all `5xx` responses, use the string `5xx`. +If multiple filters are used, then only one has to match. +The default value is no filters, with every request being printed. + +Environment variable: `INFLUXDB_HTTP_ACCESS_LOG_STATUS_FILTERS_x` + +##### Examples + +###### Setting access log status filters using configuration settings + +`access-log-status-filter = ["4xx", "5xx"]` + +`"4xx"` is in array position `0` +`"5xx"` is in array position `1` + +###### Setting access log status filters using environment variables + +The input values for the `access-log-status-filters` is an array. +When using environment variables, the values can be supplied as follows. + +`INFLUXDB_HTTP_ACCESS_LOG_STATUS_FILTERS_0=4xx` + +`INFLUXDB_HTTP_ACCESS_LOG_STATUS_FILTERS_1=5xx` + +The `_n` at the end of the environment variable represents the array position of the entry. + +#### `write-tracing = false` + +Enables detailed write logging. + +Environment variable: `INFLUXDB_HTTP_WRITE_TRACING` + +#### `pprof-enabled = true` + +Determines whether the `/pprof` endpoint is enabled. +This endpoint is used for troubleshooting and monitoring. + +Environment variable: `INFLUXDB_HTTP_PPROF_ENABLED` + +#### `https-enabled = false` + +Enables HTTPS. + +Environment variable: `INFLUXDB_HTTP_HTTPS_ENABLED` + +#### `https-certificate = "/etc/ssl/influxdb.pem"` + +The SSL certificate to use when HTTPS is enabled. +The certificate should be a PEM-encoded bundle of the certificate and key. +If it is just the certificate, a key must be specified in `https-private-key`. + +Environment variable: `INFLUXDB_HTTP_HTTPS_CERTIFICATE` + +#### `https-private-key = ""` + +The location of the separate private key. + +Environment variable: `INFLUXDB_HTTP_HTTPS_PRIVATE_KEY` + +#### `shared-secret = ""` + +The JWT authorization shared secret used to validate requests using JSON web tokens (JWTs). + +Environment variable: `INFLUXDB_HTTP_SHARED_SECRET` + +#### `max-row-limit = 0` + +The default chunk size for result sets that should be chunked. +The maximum number of rows that can be returned in a non-chunked query. +The default setting of `0` allows for an unlimited number of rows. +InfluxDB includes a `"partial":true` tag in the response body if query results exceed the `max-row-limit` setting. + +Environment variable: `INFLUXDB_HTTP_MAX_ROW_LIMIT` + +#### `max-connection-limit = 0` + +The maximum number of HTTP connections that may be open at once. +New connections that would exceed this limit are dropped. +The default value of `0` disables the limit. + +Environment variable: `INFLUXDB_HTTP_MAX_CONNECTION_LIMIT` + +#### `unix-socket-enabled = false` + +Enables the HTTP service over the UNIX domain socket. + +Environment variable: `INFLUXDB_HTTP_UNIX_SOCKET_ENABLED` + +#### `bind-socket = "/var/run/influxdb.sock"` + +The path of the UNIX domain socket. + +Environment variable: `INFLUXDB_HTTP_BIND_SOCKET` + +#### `max-concurrent-write-limit = 0` + +The maximum number of writes processed concurrently. +The default value of `0` disables the limit. + +Environment variable: `INFLUXDB_HTTP_MAX_CONCURRENT_WRITE_LIMIT` + +#### `max-enqueued-write-limit = 0` + +The maximum number of writes queued for processing. +The default value of `0` disables the limit. + +Environment variable: `INFLUXDB_HTTP_MAX_ENQUEUED_WRITE_LIMIT` + +#### `enqueued-write-timeout = 0` + +The maximum duration for a write to wait in the queue to be processed. +Setting this to `0` or setting `max-concurrent-write-limit` to `0` disables the limit. + +----- + +## Logging settings + +### `[logging]` + +#### `format = "logfmt"` + +Determines which log encoder to use for logs. +Valid options are `auto`, `logfmt`, and `json`. +A setting of `auto` will use a more a more user-friendly output format if the output terminal is a TTY, but the format is not as easily machine-readable. +When the output is a non-TTY, `auto` will use `logfmt`. + +Environment variable: `INFLUXDB_LOGGING_FORMAT` + +#### `level = "info"` + +Determines which level of logs will be emitted. + +Environment variable: `INFLUXDB_LOGGING_LEVEL` + +#### `suppress-logo = false` + +Suppresses the logo output that is printed when the program is started. + +Environment variable: `INFLUXDB_LOGGING_SUPPRESS_LOGO` + +----- + +## Subscriber settings + +### `[subscriber]` + +Controls the subscriptions, which can be used to fork a copy of all data received by the InfluxDB host. + +#### `enabled = true` + +Determines whether the subscriber service is enabled. + +Environment variable: `INFLUXDB_SUBSCRIBER_ENABLED` + +#### `http-timeout = "30s"` + +The default timeout for HTTP writes to subscribers. + +Environment variable: `INFLUXDB_SUBSCRIBER_HTTP_TIMEOUT` + +#### `insecure-skip-verify = false` + +Allows insecure HTTPS connections to subscribers. +This option is useful when testing with self-signed certificates. + +Environment variable: `INFLUXDB_SUBSCRIBER_INSECURE_SKIP_VERIFY` + +#### `ca-certs = ""` + +The path to the PEM-encoded CA certs file. +If the set to the empty string (`""`), the default system certs will used. + +Environment variable: `INFLUXDB_SUBSCRIBER_CA_CERTS` + +#### `write-concurrency = 40` + +The number of writer Goroutines processing the write channel. + +Environment variable: `INFLUXDB_SUBSCRIBER_WRITE_CONCURRENCY` + +#### `write-buffer-size = 1000` + +The number of in-flight writes buffered in the write channel. + +Environment variable: `INFLUXDB_SUBSCRIBER_WRITE_BUFFER_SIZE` + +----- + +## Graphite settings + +### `[[graphite]]` + +This section controls one or many listeners for Graphite data. +For more information, see [Graphite protocol support in InfluxDB](/enterprise_influxdb/v1.9/supported_protocols/graphite/). + +#### `enabled = false` + +Determines whether the graphite endpoint is enabled. + +These next lines control how batching works. +You should have this enabled otherwise you could get dropped metrics or poor performance. +Batching will buffer points in memory if you have many coming in. + +```toml +# database = "graphite" +# retention-policy = "" +# bind-address = ":2003" +# protocol = "tcp" +# consistency-level = "one" +``` + +#### `batch-size = 5000` + +Flush if this many points get buffered. + +#### `batch-pending = 10` + +The number of batches that may be pending in memory. + +#### `batch-timeout = "1s"` + +Flush at least this often even if we haven't hit buffer limit. + +#### `udp-read-buffer = 0` + +UDP Read buffer size, `0` means OS default. UDP listener will fail if set above OS max. + +#### `separator = "."` + +This string joins multiple matching 'measurement' values providing more control over the final measurement name. + +#### `tags = ["region=us-east", "zone=1c"]` + +Default tags that will be added to all metrics. +These can be overridden at the template level or by tags extracted from metric. + +#### Templates pattern + +```toml +# templates = [ +# "*.app env.service.resource.measurement", +# # Default template +# "server.*", +# ] +``` + +Each template line requires a template pattern. +It can have an optional filter before the template and separated by spaces. +It can also have optional extra tags following the template. +Multiple tags should be separated by commas and no spaces similar to the line protocol format. +There can be only one default template. + +----- + +## CollectD settings + +The `[[collectd]]` settings control the listener for `collectd` data. +For more information, see [CollectD protocol support in InfluxDB](/enterprise_influxdb/v1.9/supported_protocols/collectd/). + +### `[[collectd]]` + +```toml +# enabled = false +# bind-address = ":25826" +# database = "collectd" +# retention-policy = "" +# typesdb = "/usr/share/collectd/types.db" +``` + +#### `security-level = ""` + +The collectd security level can be "" (or "none"), "sign", or "encrypt". + +#### `auth-file = ""` + +The path to the `collectd` authorization file. +Must be set if security level is sign or encrypt. + + +These next lines control how batching works. +You should have this enabled otherwise you could get dropped metrics or poor performance. +Batching will buffer points in memory if you have many coming in. + +#### `batch-size = 5000` + +Flush if this many points get buffered. + +#### `batch-pending = 10` + +The number of batches that may be pending in memory. + +#### `batch-timeout = "10s"` + +Flush at least this often even if we haven't hit buffer limit. + +#### `read-buffer = 0` + +UDP Read buffer size, 0 means OS default. UDP listener will fail if set above OS max. + +----- + +## OpenTSDB settings + +Controls the listener for OpenTSDB data. +For more information, see [OpenTSDB protocol support in InfluxDB](/enterprise_influxdb/v1.9/supported_protocols/opentsdb/). + +### `[[opentsdb]]` + +```toml +# enabled = false +# bind-address = ":4242" +# database = "opentsdb" +# retention-policy = "" +# consistency-level = "one" +# tls-enabled = false +# certificate= "/etc/ssl/influxdb.pem" +``` + +#### `log-point-errors = true` + +Log an error for every malformed point. + +#### Settings for batching + +These next lines control how batching works. +You should have this enabled otherwise you could get dropped metrics or poor performance. +Only points metrics received over the telnet protocol undergo batching. + +#### `batch-size = 1000` + +Flush if this many points get buffered. + +#### `batch-pending = 5` + +The number of batches that may be pending in memory. + +#### `batch-timeout = "1s"` + +Flush at least this often even if we haven't hit buffer limit. + +----- + +## UDP settings + +The `[[udp]]` settings control the listeners for InfluxDB line protocol data using UDP. +For more information, see [UDP protocol support in InfluxDB](/enterprise_influxdb/v1.9/supported_protocols/udp/). + +### `[[udp]]` + +```toml +# enabled = false +# bind-address = ":8089" +# database = "udp" +# retention-policy = "" +``` + +#### `precision = ""` + +InfluxDB precision for timestamps on received points ("" or "n", "u", "ms", "s", "m", "h") + +These next lines control how batching works. You should have this enabled otherwise you could get dropped metrics or poor performance. +Batching will buffer points in memory if you have many coming in. + +#### `batch-size = 5000` + +Flush if this many points get buffered. + +#### `batch-pending = 10` + +The number of batches that may be pending in memory. + +#### `batch-timeout = "1s"` + +Will flush at least this often even if we haven't hit buffer limit. + +#### `read-buffer = 0` + +UDP Read buffer size, 0 means OS default. UDP listener will fail if set above OS max. + +----- + +## Continuous queries settings + +### `[continuous_queries]` + +Controls how continuous queries are run within InfluxDB. + +#### `enabled = true` + +Determines whether the continuous query service is enabled. + +Environment variable: `INFLUXDB_CONTINUOUS_QUERIES_ENABLED` + +#### `log-enabled = true` + +Controls whether queries are logged when executed by the CQ service. + +Environment variable: `INFLUXDB_CONTINUOUS_QUERIES_LOG_ENABLED` + +#### `query-stats-enabled = false` + +Write continuous query execution statistics to the default monitor store. + +Environment variable: `INFLUXDB_CONTINUOUS_QUERIES_QUERY_STATS_ENABLED` + +#### `run-interval = "1s"` + +The interval for how often continuous queries will be checked whether they need to run. + +Environment variable: `INFLUXDB_CONTINUOUS_QUERIES_RUN_INTERVAL` + +----- + +## TLS settings + + +### `[tls]` + +Global configuration settings for Transport Layer Security (TLS) in InfluxDB. + +If the TLS configuration settings is not specified, InfluxDB supports all of the cipher suite IDs listed and all TLS versions implemented in the [Constants section of the Go `crypto/tls` package documentation](https://golang.org/pkg/crypto/tls/#pkg-constants), depending on the version of Go used to build InfluxDB. +Use the `SHOW DIAGNOSTICS` command to see the version of Go used to build InfluxDB. + +### Recommended server configuration for "modern compatibility" + +InfluxData recommends configuring your InfluxDB server's TLS settings for "modern compatibility" that provides a higher level of security and assumes that backward compatibility is not required. +Our recommended TLS configuration settings for `ciphers`, `min-version`, and `max-version` are based on Mozilla's "modern compatibility" TLS server configuration described in [Security/Server Side TLS](https://wiki.mozilla.org/Security/Server_Side_TLS#Modern_compatibility). + +InfluxData's recommended TLS settings for "modern compatibility" are specified in the following configuration settings example. + +#### Recommended "modern compatibility" cipher settings + +```toml +ciphers = [ "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305", + "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305", + "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", + "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", + "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", + "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384" +] + +min-version = "tls1.3" + +max-version = "tls1.3" + +``` + +#### `min-version = "tls1.3"` + +Minimum version of the TLS protocol that will be negotiated. +Valid values include: `tls1.0`, `tls1.1`, and `tls1.3`. +If not specified, `min-version` is the minimum TLS version specified in the [Go `crypto/tls` package](https://golang.org/pkg/crypto/tls/#pkg-constants). +In this example, `tls1.3` specifies the minimum version as TLS 1.3. + +Environment variable: `INFLUXDB_TLS_MIN_VERSION` + +#### `max-version = "tls1.3"` + +The maximum version of the TLS protocol that will be negotiated. +Valid values include: `tls1.0`, `tls1.1`, and `tls1.3`. +If not specified, `max-version` is the maximum TLS version specified in the [Go `crypto/tls` package](https://golang.org/pkg/crypto/tls/#pkg-constants). +In this example, `tls1.3` specifies the maximum version as TLS 1.3. + +Environment variable: `INFLUXDB_TLS_MAX_VERSION` + +## InfluxQL query management settings + +### `[coordinator]` + +This section contains configuration settings for query management. +For more on managing queries, see [Query Management](/enterprise_influxdb/v1.9/troubleshooting/query_management/). + +#### `write-timeout = "10s"` + +Duration a write request waits until a "timeout" error is returned to the caller. Default is `10s` (10 seconds). + +Environment variable: `INFLUXDB_COORDINATOR_WRITE_TIMEOUT` + +#### `max-concurrent-queries = 0` + +Maximum number of concurrently running queries allowed. `0` means unlimited. Default is `0`. + +Environment variable: `INFLUXDB_COORDINATOR_MAX_CONCURRENT_QUERIES` + +#### `query-timeout = "0s"` + +Maximum duration a query is allowed to execute before InfluxDB kills the query. +`0s` allows queries to run with no time restrictions. +Default is `0s`. + +Environment variable: `INFLUXDB_COORDINATOR_QUERY_TIMEOUT` + +#### `log-queries-after = "0s"` + +Maximum duration a query can run until InfluxDB logs the query with a +`Detected slow query` message. +`0s` disables slow query logging. +Default is `0s`. + +Environment variable: `INFLUXDB_COORDINATOR_LOG_QUERIES_AFTER` + +#### `max-select-point = 0` + +Maximum number of [points](/enterprise_influxdb/v1.9/concepts/glossary/#point) a +`SELECT` statement can process. +`0` lets the `SELECT` statement to process an unlimited number of points. +Default is `0`. + +Environment variable: `INFLUXDB_COORDINATOR_MAX_SELECT_POINT` + +#### `max-select-series = 0` + +Maximum number of [series](/enterprise_influxdb/v1.9/concepts/glossary/#series) a +`SELECT` statement can process. +`0` lets the `SELECT` statement to process an unlimited number of series. +Default is `0`. + +Environment variable: `INFLUXDB_COORDINATOR_MAX_SELECT_SERIES` + +#### `max-select-buckets = 0` + +Maximum number of `GROUP BY time()` buckets a query can process. +`0` allows a query to process an unlimited number of buckets. +Default is `0`. + +Environment variable: `INFLUXDB_COORDINATOR_MAX_SELECT_BUCKETS` + +## Flux query management settings + +### `[flux-controller]` + +This section contains configuration settings for Flux query management. +For more on managing queries, see [Query Management](/enterprise_influxdb/v1.9/troubleshooting/query_management/). + +#### query-concurrency +Number of queries allowed to execute concurrently. +`0` means unlimited. +Default is `0`. + +#### query-initial-memory-bytes +Initial bytes of memory allocated for a query. +`0` means unlimited. +Default is `0`. + +#### query-max-memory-bytes +Maximum total bytes of memory allowed for an individual query. +`0` means unlimited. +Default is `0`. + +#### total-max-memory-bytes +Maximum total bytes of memory allowed for all running Flux queries. +`0` means unlimited. +Default is `0`. + +#### query-queue-size +Maximum number of queries allowed in execution queue. +When queue limit is reached, new queries are rejected. +`0` means unlimited. +Default is `0`. diff --git a/content/enterprise_influxdb/v1.9/administration/config-meta-nodes.md b/content/enterprise_influxdb/v1.9/administration/config-meta-nodes.md new file mode 100644 index 000000000..501b530a8 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/administration/config-meta-nodes.md @@ -0,0 +1,286 @@ +--- +title: Configure InfluxDB Enterprise meta modes +description: > + Configure InfluxDB Enterprise data node settings and environmental variables. +menu: + enterprise_influxdb_1_9: + name: Configure meta nodes + weight: 30 + parent: Administration +--- + +* [Meta node configuration settings](#meta-node-configuration-settings) + * [Global options](#global-options) + * [Enterprise license `[enterprise]`](#enterprise) + * [Meta node `[meta]`](#meta) + * [TLS `[tls]`](#tls-settings) + +## Meta node configuration settings + +### Global options + +#### `reporting-disabled = false` + +InfluxData, the company, relies on reported data from running nodes primarily to +track the adoption rates of different InfluxDB versions. +These data help InfluxData support the continuing development of InfluxDB. + +The `reporting-disabled` option toggles the reporting of data every 24 hours to +`usage.influxdata.com`. +Each report includes a randomly-generated identifier, OS, architecture, +InfluxDB version, and the number of databases, measurements, and unique series. +To disable reporting, set this option to `true`. + +> **Note:** No data from user databases are ever transmitted. + +#### `bind-address = ""` + +This setting is not intended for use. +It will be removed in future versions. + +#### `hostname = ""` + +The hostname of the [meta node](/enterprise_influxdb/v1.9/concepts/glossary/#meta-node). +This must be resolvable and reachable by all other members of the cluster. + +Environment variable: `INFLUXDB_HOSTNAME` + +----- + +### Enterprise license settings +#### `[enterprise]` + +The `[enterprise]` section contains the parameters for the meta node's +registration with the [InfluxData portal](https://portal.influxdata.com/). + +#### `license-key = ""` + +The license key created for you on [InfluxData portal](https://portal.influxdata.com). +The meta node transmits the license key to +[portal.influxdata.com](https://portal.influxdata.com) over port 80 or port 443 +and receives a temporary JSON license file in return. +The server caches the license file locally. +If your server cannot communicate with [https://portal.influxdata.com](https://portal.influxdata.com), you must use the [`license-path` setting](#license-path). + +Use the same key for all nodes in the same cluster. +{{% warn %}}The `license-key` and `license-path` settings are mutually exclusive and one must remain set to the empty string. +{{% /warn %}} + +> **Note:** You must restart meta nodes to update your configuration. For more information, see how to [renew or update your license key](/enterprise_influxdb/v1.9/administration/renew-license/). + +Environment variable: `INFLUXDB_ENTERPRISE_LICENSE_KEY` + +#### `license-path = ""` + +The local path to the permanent JSON license file that you received from InfluxData +for instances that do not have access to the internet. +To obtain a license file, contact [sales@influxdb.com](mailto:sales@influxdb.com). + +The license file must be saved on every server in the cluster, including meta nodes +and data nodes. +The file contains the JSON-formatted license, and must be readable by the `influxdb` user. +Each server in the cluster independently verifies its license. + +{{% warn %}} +The `license-key` and `license-path` settings are mutually exclusive and one must remain set to the empty string. +{{% /warn %}} + +> **Note:** You must restart meta nodes to update your configuration. For more information, see how to [renew or update your license key](/enterprise_influxdb/v1.9/administration/renew-license/). + +Environment variable: `INFLUXDB_ENTERPRISE_LICENSE_PATH` + +----- +### Meta node settings + +#### `[meta]` + +#### `dir = "/var/lib/influxdb/meta"` + +The directory where cluster meta data is stored. + +Environment variable: `INFLUXDB_META_DIR` + +#### `bind-address = ":8089"` + +The bind address(port) for meta node communication. +For simplicity, InfluxData recommends using the same port on all meta nodes, +but this is not necessary. + +Environment variable: `INFLUXDB_META_BIND_ADDRESS` + +#### `http-bind-address = ":8091"` + +The default address to bind the API to. + +Environment variable: `INFLUXDB_META_HTTP_BIND_ADDRESS` + +#### `https-enabled = false` + +Determines whether meta nodes use HTTPS to communicate with each other. By default, HTTPS is disabled. We strongly recommend enabling HTTPS. + +To enable HTTPS, set https-enabled to `true`, specify the path to the SSL certificate `https-certificate = " "`, and specify the path to the SSL private key `https-private-key = ""`. + +Environment variable: `INFLUXDB_META_HTTPS_ENABLED` + +#### `https-certificate = ""` + +If HTTPS is enabled, specify the path to the SSL certificate. +Use either: + +* PEM-encoded bundle with both the certificate and key (`[bundled-crt-and-key].pem`) +* Certificate only (`[certificate].crt`) + +Environment variable: `INFLUXDB_META_HTTPS_CERTIFICATE` + +#### `https-private-key = ""` + +If HTTPS is enabled, specify the path to the SSL private key. +Use either: + +* PEM-encoded bundle with both the certificate and key (`[bundled-crt-and-key].pem`) +* Private key only (`[private-key].key`) + +Environment variable: `INFLUXDB_META_HTTPS_PRIVATE_KEY` + +#### `https-insecure-tls = false` + +Whether meta nodes will skip certificate validation communicating with each other over HTTPS. +This is useful when testing with self-signed certificates. + +Environment variable: `INFLUXDB_META_HTTPS_INSECURE_TLS` + +#### `data-use-tls = false` + +Whether to use TLS to communicate with data nodes. + +#### `data-insecure-tls = false` + +Whether meta nodes will skip certificate validation communicating with data nodes over TLS. +This is useful when testing with self-signed certificates. + +#### `gossip-frequency = "5s"` + +The default frequency with which the node will gossip its known announcements. + +#### `announcement-expiration = "30s"` + +The default length of time an announcement is kept before it is considered too old. + +#### `retention-autocreate = true` + +Automatically create a default retention policy when creating a database. + +#### `election-timeout = "1s"` + +The amount of time in candidate state without a leader before we attempt an election. + +#### `heartbeat-timeout = "1s"` + +The amount of time in follower state without a leader before we attempt an election. + +#### `leader-lease-timeout = "500ms"` + +The leader lease timeout is the amount of time a Raft leader will remain leader + if it does not hear from a majority of nodes. +After the timeout the leader steps down to the follower state. +Clusters with high latency between nodes may want to increase this parameter to + avoid unnecessary Raft elections. + +Environment variable: `INFLUXDB_META_LEADER_LEASE_TIMEOUT` + +#### `commit-timeout = "50ms"` + +The commit timeout is the amount of time a Raft node will tolerate between +commands before issuing a heartbeat to tell the leader it is alive. +The default setting should work for most systems. + +Environment variable: `INFLUXDB_META_COMMIT_TIMEOUT` + +#### `consensus-timeout = "30s"` + +Timeout waiting for consensus before getting the latest Raft snapshot. + +Environment variable: `INFLUXDB_META_CONSENSUS_TIMEOUT` + +#### `cluster-tracing = false` + +Cluster tracing toggles the logging of Raft logs on Raft nodes. +Enable this setting when debugging Raft consensus issues. + +Environment variable: `INFLUXDB_META_CLUSTER_TRACING` + +#### `logging-enabled = true` + +Meta logging toggles the logging of messages from the meta service. + +Environment variable: `INFLUXDB_META_LOGGING_ENABLED` + +#### `pprof-enabled = true` + +Enables the `/debug/pprof` endpoint for troubleshooting. +To disable, set the value to `false`. + +Environment variable: `INFLUXDB_META_PPROF_ENABLED` + +#### `lease-duration = "1m0s"` + +The default duration of the leases that data nodes acquire from the meta nodes. +Leases automatically expire after the `lease-duration` is met. + +Leases ensure that only one data node is running something at a given time. +For example, [continuous queries](/enterprise_influxdb/v1.9/concepts/glossary/#continuous-query-cq) +(CQs) use a lease so that all data nodes aren't running the same CQs at once. + +For more details about `lease-duration` and its impact on continuous queries, see +[Configuration and operational considerations on a cluster](/enterprise_influxdb/v1.9/features/clustering-features/#configuration-and-operational-considerations-on-a-cluster). + +Environment variable: `INFLUXDB_META_LEASE_DURATION` + +#### `auth-enabled = false` + +If true, HTTP endpoints require authentication. +This setting must have the same value as the data nodes' meta.meta-auth-enabled configuration. + +#### `ldap-allowed = false` + +Whether LDAP is allowed to be set. +If true, you will need to use `influxd ldap set-config` and set enabled=true to use LDAP authentication. + +#### `shared-secret = ""` + +The shared secret to be used by the public API for creating custom JWT authentication. +If you use this setting, set [`auth-enabled`](#auth-enabled-false) to `true`. + +Environment variable: `INFLUXDB_META_SHARED_SECRET` + +#### `internal-shared-secret = ""` + +The shared secret used by the internal API for JWT authentication for +inter-node communication within the cluster. +Set this to a long pass phrase. +This value must be the same value as the +[`[meta] meta-internal-shared-secret`](/enterprise_influxdb/v1.9/administration/config-data-nodes#meta-internal-shared-secret) in the data node configuration file. +To use this option, set [`auth-enabled`](#auth-enabled-false) to `true`. + +Environment variable: `INFLUXDB_META_INTERNAL_SHARED_SECRET` + +### TLS settings + +For more information, see [TLS settings for data nodes](/enterprise_influxdb/v1.9/administration/config-data-nodes#tls-settings). + +#### Recommended "modern compatibility" cipher settings + +```toml +ciphers = [ "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305", + "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305", + "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", + "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", + "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", + "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384" +] + +min-version = "tls1.2" + +max-version = "tls1.2" + +``` diff --git a/content/enterprise_influxdb/v1.9/administration/configuration.md b/content/enterprise_influxdb/v1.9/administration/configuration.md new file mode 100644 index 000000000..48b6030cc --- /dev/null +++ b/content/enterprise_influxdb/v1.9/administration/configuration.md @@ -0,0 +1,182 @@ +--- +title: Configure InfluxDB Enterprise clusters +description: > + Learn about global options, meta node options, data node options and other InfluxDB Enterprise configuration settings, including +aliases: + - /enterprise/v1.8/administration/configuration/ +menu: + enterprise_influxdb_1_9: + name: Configure clusters + weight: 10 + parent: Administration +--- + +This page contains general information about configuring InfluxDB Enterprise clusters. +For complete listings and descriptions of the configuration settings, see: + +* [Configure data nodes](/enterprise_influxdb/v1.9/administration/config-data-nodes) +* [Configure meta nodes](/enterprise_influxdb/v1.9/administration/config-meta-nodes) + +## Use configuration files + +### Display the default configurations + +The following commands print out a TOML-formatted configuration with all +available options set to their default values. + +#### Meta node configuration + +```bash +influxd-meta config +``` + +#### Data node configuration + +```bash +influxd config +``` + +#### Create a configuration file + +On POSIX systems, generate a new configuration file by redirecting the output +of the command to a file. + +New meta node configuration file: +``` +influxd-meta config > /etc/influxdb/influxdb-meta-generated.conf +``` + +New data node configuration file: +``` +influxd config > /etc/influxdb/influxdb-generated.conf +``` + +Preserve custom settings from older configuration files when generating a new +configuration file with the `-config` option. +For example, this overwrites any default configuration settings in the output +file (`/etc/influxdb/influxdb.conf.new`) with the configuration settings from +the file (`/etc/influxdb/influxdb.conf.old`) passed to `-config`: + +``` +influxd config -config /etc/influxdb/influxdb.conf.old > /etc/influxdb/influxdb.conf.new +``` + +#### Launch the process with a configuration file + +There are two ways to launch the meta or data processes using your customized +configuration file. + +* Point the process to the desired configuration file with the `-config` option. + + To start the meta node process with `/etc/influxdb/influxdb-meta-generate.conf`: + + influxd-meta -config /etc/influxdb/influxdb-meta-generate.conf + + To start the data node process with `/etc/influxdb/influxdb-generated.conf`: + + influxd -config /etc/influxdb/influxdb-generated.conf + + +* Set the environment variable `INFLUXDB_CONFIG_PATH` to the path of your +configuration file and start the process. + + To set the `INFLUXDB_CONFIG_PATH` environment variable and launch the data + process using `INFLUXDB_CONFIG_PATH` for the configuration file path: + + export INFLUXDB_CONFIG_PATH=/root/influxdb.generated.conf + echo $INFLUXDB_CONFIG_PATH + /root/influxdb.generated.conf + influxd + +If set, the command line `-config` path overrides any environment variable path. +If you do not supply a configuration file, InfluxDB uses an internal default +configuration (equivalent to the output of `influxd config` and `influxd-meta +config`). + +{{% warn %}} Note for 1.3, the influxd-meta binary, if no configuration is specified, will check the INFLUXDB_META_CONFIG_PATH. +If that environment variable is set, the path will be used as the configuration file. +If unset, the binary will check the ~/.influxdb and /etc/influxdb folder for an influxdb-meta.conf file. +If it finds that file at either of the two locations, the first will be loaded as the configuration file automatically. +
+This matches a similar behavior that the open source and data node versions of InfluxDB already follow. +{{% /warn %}} + +Configure InfluxDB using the configuration file (`influxdb.conf`) and environment variables. +The default value for each configuration setting is shown in the documentation. +Commented configuration options use the default value. + +Configuration settings with a duration value support the following duration units: + +- `ns` _(nanoseconds)_ +- `us` or `µs` _(microseconds)_ +- `ms` _(milliseconds)_ +- `s` _(seconds)_ +- `m` _(minutes)_ +- `h` _(hours)_ +- `d` _(days)_ +- `w` _(weeks)_ + +### Environment variables + +All configuration options can be specified in the configuration file or in +environment variables. +Environment variables override the equivalent options in the configuration +file. +If a configuration option is not specified in either the configuration file +or in an environment variable, InfluxDB uses its internal default +configuration. + +In the sections below we name the relevant environment variable in the +description for the configuration setting. +Environment variables can be set in `/etc/default/influxdb-meta` and +`/etc/default/influxdb`. + +> **Note:** +To set or override settings in a config section that allows multiple +configurations (any section with double_brackets (`[[...]]`) in the header supports +multiple configurations), the desired configuration must be specified by ordinal +number. +For example, for the first set of `[[graphite]]` environment variables, +prefix the configuration setting name in the environment variable with the +relevant position number (in this case: `0`): +> + INFLUXDB_GRAPHITE_0_BATCH_PENDING + INFLUXDB_GRAPHITE_0_BATCH_SIZE + INFLUXDB_GRAPHITE_0_BATCH_TIMEOUT + INFLUXDB_GRAPHITE_0_BIND_ADDRESS + INFLUXDB_GRAPHITE_0_CONSISTENCY_LEVEL + INFLUXDB_GRAPHITE_0_DATABASE + INFLUXDB_GRAPHITE_0_ENABLED + INFLUXDB_GRAPHITE_0_PROTOCOL + INFLUXDB_GRAPHITE_0_RETENTION_POLICY + INFLUXDB_GRAPHITE_0_SEPARATOR + INFLUXDB_GRAPHITE_0_TAGS + INFLUXDB_GRAPHITE_0_TEMPLATES + INFLUXDB_GRAPHITE_0_UDP_READ_BUFFER +> +For the Nth Graphite configuration in the configuration file, the relevant +environment variables would be of the form `INFLUXDB_GRAPHITE_(N-1)_BATCH_PENDING`. +For each section of the configuration file the numbering restarts at zero. + +### `GOMAXPROCS` environment variable + +{{% note %}} +_**Note:**_ `GOMAXPROCS` cannot be set using the InfluxDB configuration file. +It can only be set as an environment variable. +{{% /note %}} + +The `GOMAXPROCS` [Go language environment variable](https://golang.org/pkg/runtime/#hdr-Environment_Variables) +can be used to set the maximum number of CPUs that can execute simultaneously. + +The default value of `GOMAXPROCS` is the number of CPUs +that are visible to the program *on startup* +(based on what the operating system considers to be a CPU). +For a 32-core machine, the `GOMAXPROCS` value would be `32`. +You can override this value to be less than the maximum value, +which can be useful in cases where you are running the InfluxDB +along with other processes on the same machine +and want to ensure that the database doesn't negatively affect those processes. + +{{% note %}} +_**Note:**_ Setting `GOMAXPROCS=1` eliminates all parallelization. +{{% /note %}} diff --git a/content/enterprise_influxdb/v1.9/administration/ldap.md b/content/enterprise_influxdb/v1.9/administration/ldap.md new file mode 100644 index 000000000..9ae9fb21b --- /dev/null +++ b/content/enterprise_influxdb/v1.9/administration/ldap.md @@ -0,0 +1,211 @@ +--- +title: Configure LDAP authentication in InfluxDB Enterprise +description: > + Configure LDAP authentication in InfluxDB Enterprise and test LDAP connectivity. +menu: + enterprise_influxdb_1_9: + name: Configure LDAP authentication + weight: 40 + parent: Administration +--- + +Configure InfluxDB Enterprise to use LDAP (Lightweight Directory Access Protocol) to: + +- Validate user permissions +- Synchronize InfluxDB and LDAP so each LDAP request doesn't need to be queried + +{{% note %}} +To configure InfluxDB Enterprise to support LDAP, all users must be managed in the remote LDAP service. +If LDAP is configured and enabled, users **must** authenticate through LDAP, including users who may have existed before enabling LDAP. +{{% /note %}} + +## Configure LDAP for an InfluxDB Enterprise cluster + +To use LDAP with an InfluxDB Enterprise cluster, do the following: + +1. [Configure data nodes](#configure-data-nodes) +2. [Configure meta nodes](#configure-meta-nodes) +3. [Create, verify, and upload the LDAP configuration file](#create-verify-and-upload-the-ldap-configuration-file) +4. [Restart meta and data nodes](#restart-meta-and-data-nodes) + +### Configure data nodes + +Update the following settings in each data node configuration file (`/etc/influxdb/influxdb.conf`): + +1. Under `[http]`, enable HTTP authentication by setting `auth-enabled` to `true`. + (Or set the corresponding environment variable `INFLUXDB_HTTP_AUTH_ENABLED` to `true`.) +2. Configure the HTTP shared secret to validate requests using JSON web tokens (JWT) and sign each HTTP payload with the secret and username. + Set the `[http]` configuration setting for `shared-secret`, or the corresponding environment variable `INFLUXDB_HTTP_SHARED_SECRET`. +3. If you're enabling authentication on meta nodes, you must also include the following configurations: + - `INFLUXDB_META_META_AUTH_ENABLED` environment variable, or `[http]` configuration setting `meta-auth-enabled`, is set to `true`. + This value must be the same value as the meta node's `meta.auth-enabled` configuration. + - `INFLUXDB_META_META_INTERNAL_SHARED_SECRET`, or the corresponding `[meta]` configuration setting `meta-internal-shared-secret`, is set to `true`. + This value must be the same value as the meta node's `meta.internal-shared-secret`. + +### Configure meta nodes + +Update the following settings in each meta node configuration file (`/etc/influxdb/influxdb-meta.conf`): + +1. Configure the meta node META shared secret to validate requests using JSON web tokens (JWT) and sign each HTTP payload with the username and shared secret. +2. Set the `[meta]` configuration setting `internal-shared-secret` to `""`. + (Or set the `INFLUXDB_META_INTERNAL_SHARED_SECRET` environment variable.) +3. Set the `[meta]` configuration setting `meta.ldap-allowed` to `true` on all meta nodes in your cluster. + (Or set the `INFLUXDB_META_LDAP_ALLOWED`environment variable.) + +### Authenticate your connection to InfluxDB + +To authenticate your InfluxDB connection, run the following command, replacing `username:password` with your credentials: + +{{< keep-url >}} +```bash +curl -u username:password -XPOST "http://localhost:8086/..." +``` + +For more detail on authentication, see [Authentication and authorization in InfluxDB](/enterprise_influxdb/v1.9/administration/authentication_and_authorization/). + +### Create, verify, and upload the LDAP configuration file + +1. To create a sample LDAP configuration file, run the following command: + + ```bash + influxd-ctl ldap sample-config + ``` + +2. Save the sample file and edit as needed for your LDAP server. + For detail, see the [sample LDAP configuration file](#sample-ldap-configuration) below. + + > To use fine-grained authorization (FGA) with LDAP, you must map InfluxDB Enterprise roles to key-value pairs in the LDAP database. + For more information, see [Fine-grained authorization in InfluxDB Enterprise](/enterprise_influxdb/v1.9/guides/fine-grained-authorization/). + The InfluxDB admin user doesn't include permissions for InfluxDB Enterprise roles. + +3. Restart all meta and data nodes in your InfluxDB Enterprise cluster to load your updated configuration. + + On each **meta** node, run: + + {{< code-tabs-wrapper >}} + {{% code-tabs %}} + [sysvinit](#) + [systemd](#) + {{% /code-tabs %}} + {{% code-tab-content %}} + ```sh + service influxdb-meta restart + ``` + {{% /code-tab-content %}} + {{% code-tab-content %}} + ```sh + sudo systemctl restart influxdb-meta + ``` + {{% /code-tab-content %}} + {{< /code-tabs-wrapper >}} + + On each **data** node, run: + + {{< code-tabs-wrapper >}} + {{% code-tabs %}} + [sysvinit](#) + [systemd](#) + {{% /code-tabs %}} + {{% code-tab-content %}} + ```sh + service influxdb restart + ``` + {{% /code-tab-content %}} + {{% code-tab-content %}} + ```sh + sudo systemctl restart influxdb + ``` + {{% /code-tab-content %}} + {{< /code-tabs-wrapper >}} + + +4. To verify your LDAP configuration, run: + + ```bash + influxd-ctl ldap verify -ldap-config /path/to/ldap.toml + ``` + +5. To load your LDAP configuration file, run the following command: + + ```bash + influxd-ctl ldap set-config /path/to/ldap.toml + ``` + + +## Sample LDAP configuration + +The following is a sample configuration file that connects to a publicly available LDAP server. + +A `DN` ("distinguished name") uniquely identifies an entry and describes its position in the directory information tree (DIT) hierarchy. +The DN of an LDAP entry is similar to a file path on a file system. +`DNs` refers to multiple DN entries. + +{{% truncate %}} +```toml +enabled = true + +[[servers]] + enabled = true + +[[servers]] + host = "" + port = 389 + + # Security mode for LDAP connection to this server. + # The recommended security is set "starttls" by default. This uses an initial unencrypted connection + # and upgrades to TLS as the first action against the server, + # per the LDAPv3 standard. + # Other options are "starttls+insecure" to behave the same as starttls + # but skip server certificate verification, or "none" to use an unencrypted connection. + security = "starttls" + + # Credentials to use when searching for a user or group. + bind-dn = "cn=read-only-admin,dc=example,dc=com" + bind-password = "password" + + # Base DNs to use when applying the search-filter to discover an LDAP user. + search-base-dns = [ + "dc=example,dc=com", + ] + + # LDAP filter to discover a user's DN. + # %s will be replaced with the provided username. + search-filter = "(uid=%s)" + # On Active Directory you might use "(sAMAccountName=%s)". + + # Base DNs to use when searching for groups. + group-search-base-dns = ["dc=example,dc=com"] + + # LDAP filter to identify groups that a user belongs to. + # %s will be replaced with the user's DN. + group-membership-search-filter = "(&(objectClass=groupOfUniqueNames)(uniqueMember=%s))" + # On Active Directory you might use "(&(objectClass=group)(member=%s))". + + # Attribute to use to determine the "group" in the group-mappings section. + group-attribute = "ou" + # On Active Directory you might use "cn". + + # LDAP filter to search for a group with a particular name. + # This is used when warming the cache to load group membership. + group-search-filter = "(&(objectClass=groupOfUniqueNames)(cn=%s))" + # On Active Directory you might use "(&(objectClass=group)(cn=%s))". + + # Attribute of a group that contains the DNs of the group's members. + group-member-attribute = "uniqueMember" + # On Active Directory you might use "member". + + # Create an administrator role in InfluxDB and then log in as a member of the admin LDAP group. Only members of a group with the administrator role can complete admin tasks. + # For example, if tesla is the only member of the `italians` group, you must log in as tesla/password. + admin-groups = ["italians"] + + # These two roles would have to be created by hand if you want these LDAP group memberships to do anything. + [[servers.group-mappings]] + group = "mathematicians" + role = "arithmetic" + + [[servers.group-mappings]] + group = "scientists" + role = "laboratory" + +``` +{{% /truncate %}} diff --git a/content/enterprise_influxdb/v1.9/administration/logs.md b/content/enterprise_influxdb/v1.9/administration/logs.md new file mode 100644 index 000000000..1611e3353 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/administration/logs.md @@ -0,0 +1,133 @@ +--- +title: Log and trace InfluxDB Enterprise operations +description: > + Learn about logging locations, redirecting HTTP request logging, structured logging, and tracing. +menu: + enterprise_influxdb_1_9: + name: Log and trace + weight: 90 + parent: Administration +--- + + +* [Logging locations](#logging-locations) +* [Redirect HTTP request logging](#redirect-http-access-logging) +* [Structured logging](#structured-logging) +* [Tracing](#tracing) + + +InfluxDB writes log output, by default, to `stderr`. +Depending on your use case, this log information can be written to another location. +Some service managers may override this default. + +## Logging locations + +### Run InfluxDB directly + +If you run InfluxDB directly, using `influxd`, all logs will be written to `stderr`. +You may redirect this log output as you would any output to `stderr` like so: + +```bash +influxdb-meta 2>$HOME/my_log_file # Meta nodes +influxd 2>$HOME/my_log_file # Data nodes +influx-enterprise 2>$HOME/my_log_file # Enterprise Web +``` + +### Launched as a service + +#### sysvinit + +If InfluxDB was installed using a pre-built package, and then launched +as a service, `stderr` is redirected to +`/var/log/influxdb/.log`, and all log data will be written to +that file. You can override this location by setting the variable +`STDERR` in the file `/etc/default/`. + +For example, if on a data node `/etc/default/influxdb` contains: + +```bash +STDERR=/dev/null +``` + +all log data will be discarded. You can similarly direct output to +`stdout` by setting `STDOUT` in the same file. Output to `stdout` is +sent to `/dev/null` by default when InfluxDB is launched as a service. + +InfluxDB must be restarted to pick up any changes to `/etc/default/`. + + +##### Meta nodes + +For meta nodes, the is `influxdb-meta`. +The default log file is `/var/log/influxdb/influxdb-meta.log` +The service configuration file is `/etc/default/influxdb-meta`. + +##### Data nodes + +For data nodes, the is `influxdb`. +The default log file is `/var/log/influxdb/influxdb.log` +The service configuration file is `/etc/default/influxdb`. + +##### Enterprise Web + +For Enterprise Web nodes, the is `influx-enterprise`. +The default log file is `/var/log/influxdb/influx-enterprise.log` +The service configuration file is `/etc/default/influx-enterprise`. + +#### systemd + +Starting with version 1.0, InfluxDB on systemd systems no longer +writes files to `/var/log/.log` by default, and now uses the +system configured default for logging (usually `journald`). On most +systems, the logs will be directed to the systemd journal and can be +accessed with the command: + +``` +sudo journalctl -u .service +``` + +Please consult the systemd journald documentation for configuring +journald. + +##### Meta nodes + +For data nodes the is `influxdb-meta`. +The default log command is `sudo journalctl -u influxdb-meta.service` +The service configuration file is `/etc/default/influxdb-meta`. + +##### Data nodes + +For data nodes the is `influxdb`. +The default log command is `sudo journalctl -u influxdb.service` +The service configuration file is `/etc/default/influxdb`. + +##### Enterprise Web + +For data nodes the is `influx-enterprise`. +The default log command is `sudo journalctl -u influx-enterprise.service` +The service configuration file is `/etc/default/influx-enterprise`. + +### Use logrotate + +You can use [logrotate](http://manpages.ubuntu.com/manpages/cosmic/en/man8/logrotate.8.html) +to rotate the log files generated by InfluxDB on systems where logs are written to flat files. +If using the package install on a sysvinit system, the config file for logrotate is installed in `/etc/logrotate.d`. +You can view the file [here](https://github.com/influxdb/influxdb/blob/master/scripts/logrotate). + +## Redirect HTTP access logging + +InfluxDB 1.5 introduces the option to log HTTP request traffic separately from the other InfluxDB log output. When HTTP request logging is enabled, the HTTP logs are intermingled by default with internal InfluxDB logging. By redirecting the HTTP request log entries to a separate file, both log files are easier to read, monitor, and debug. + +See [Redirecting HTTP request logging](/enterprise_influxdb/v1.9/administration/logs/#redirecting-http-access-logging) in the InfluxDB OSS documentation. + +## Structured logging + +With InfluxDB 1.5, structured logging is supported and enable machine-readable and more developer-friendly log output formats. The two new structured log formats, `logfmt` and `json`, provide easier filtering and searching with external tools and simplifies integration of InfluxDB logs with Splunk, Papertrail, Elasticsearch, and other third party tools. + +See [Structured logging](/enterprise_influxdb/v1.9/administration/logs/#structured-logging) in the InfluxDB OSS documentation. + +## Tracing + +Logging has been enhanced, starting in InfluxDB 1.5, to provide tracing of important InfluxDB operations. Tracing is useful for error reporting and discovering performance bottlenecks. + +See [Tracing](/enterprise_influxdb/v1.9/administration/logs/#tracing) in the InfluxDB OSS documentation. diff --git a/content/enterprise_influxdb/v1.9/administration/ports.md b/content/enterprise_influxdb/v1.9/administration/ports.md new file mode 100644 index 000000000..304d2a792 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/administration/ports.md @@ -0,0 +1,91 @@ +--- +title: TCP and UDP ports used in InfluxDB Enterprise +description: Configure TCP and UDP ports in InfluxDB Enterprise. +menu: + enterprise_influxdb_1_9: + name: Configure TCP and UDP Ports + weight: 120 + parent: Administration +--- + +![InfluxDB Enterprise network diagram](/img/enterprise/1-8-network-diagram.png) + +## Enabled ports + +### 8086 + +The default port that runs the InfluxDB HTTP service. +It is used for the primary public write and query API. +Clients include the CLI, Chronograf, InfluxDB client libraries, Grafana, curl, or anything that wants to write and read time series data to and from InfluxDB. +[Configure this port](/enterprise_influxdb/v1.9/administration/config-data-nodes/#bind-address-8088) +in the data node configuration file. + +_See also: [API Reference](/enterprise_influxdb/v1.9/tools/api/)._ + +### 8088 + +Data nodes listen on this port. +Primarily used by other data nodes to handle distributed reads and writes at runtime. +Used to control a data node (e.g., tell it to write to a specific shard or execute a query). +It's also used by meta nodes for cluster-type operations (e.g., tell a data node to join or leave the cluster). + +This is the default port used for RPC calls used for inter-node communication and by the CLI for backup and restore operations +(`influxdb backup` and `influxd restore`). +[Configure this port](/enterprise_influxdb/v1.9/administration/config#bind-address-127-0-0-1-8088) +in the configuration file. + +This port should not be exposed outside the cluster. + +_See also: [Backup and Restore](/enterprise_influxdb/v1.9/administration/backup_and_restore/)._ + +### 8089 + +Used for communcation between meta nodes. +It is used by the Raft consensus protocol. +The only clients using `8089` should be the other meta nodes in the cluster. + +This port should not be exposed outside the cluster. + +### 8091 + +Meta nodes listen on this port. +It is used for the meta service API. +Primarily used by data nodes to stay in sync about databases, retention policies, shards, users, privileges, etc. +Used by meta nodes to receive incoming connections by data nodes and Chronograf. +Clients also include the `influxd-ctl` command line tool and Chronograph, + +This port should not be exposed outside the cluster. + +## Disabled ports + +### 2003 + +The default port that runs the Graphite service. +[Enable and configure this port](/enterprise_influxdb/v1.9/administration/config#bind-address-2003) +in the configuration file. + +**Resources** [Graphite README](https://github.com/influxdata/influxdb/tree/1.8/services/graphite/README.md) + +### 4242 + +The default port that runs the OpenTSDB service. +[Enable and configure this port](/enterprise_influxdb/v1.9/administration/config#bind-address-4242) +in the configuration file. + +**Resources** [OpenTSDB README](https://github.com/influxdata/influxdb/tree/1.8/services/opentsdb/README.md) + +### 8089 + +The default port that runs the UDP service. +[Enable and configure this port](/enterprise_influxdb/v1.9/administration/config#bind-address-8089) +in the configuration file. + +**Resources** [UDP README](https://github.com/influxdata/influxdb/tree/1.8/services/udp/README.md) + +### 25826 + +The default port that runs the Collectd service. +[Enable and configure this port](/enterprise_influxdb/v1.9/administration/config#bind-address-25826) +in the configuration file. + +**Resources** [Collectd README](https://github.com/influxdata/influxdb/tree/1.8/services/collectd/README.md) diff --git a/content/enterprise_influxdb/v1.9/administration/renaming.md b/content/enterprise_influxdb/v1.9/administration/renaming.md new file mode 100644 index 000000000..b1e48fcd4 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/administration/renaming.md @@ -0,0 +1,56 @@ +--- +title: Rename hosts in InfluxDB Enterprise +description: Rename a host within your InfluxDB Enterprise instance. +aliases: + - /enterprise/v1.8/administration/renaming/ +menu: + enterprise_influxdb_1_9: + name: Rename hosts + weight: 100 + parent: Administration +--- + +## Host renaming + +The following instructions allow you to rename a host within your InfluxDB Enterprise instance. + +First, suspend write and query activity to the cluster. + +### Rename meta nodes + +- Find the meta node leader with `curl localhost:8091/status`. The `leader` field in the JSON output reports the leader meta node. We will start with the two meta nodes that are not leaders. +- On a non-leader meta node, run `influxd-ctl remove-meta`. Once removed, confirm by running `influxd-ctl show` on the meta leader. +- Stop the meta service on the removed node, edit its configuration file to set the new "hostname" under "/etc/influxdb/influxdb-meta.conf". +- Update the actual OS host's name if needed, apply DNS changes. +- Start the meta service. +- On the meta leader, add the meta node with the new hostname using `influxd-ctl add-meta newmetanode:8091`. Confirm with `influxd-ctl show` +- Repeat for the second meta node. +- Once the two non-leaders are updated, stop the leader and wait for another meta node to become the leader - check with `curl localhost:8091/status`. +- Repeat the process for the last meta node (former leader). + +### Intermediate verification + +- Verify the state of the cluster with `influxd-ctl show`. The version must be reported on all nodes for them to be healthy. +- Verify there is a meta leader with `curl localhost:8091/status` and that all meta nodes list the rest in the output. +- Restart all data nodes one by one. Verify that `/var/lib/influxdb/meta/client.json` on all data nodes references the new meta names. +- Verify the `show shards` output lists all shards and node ownership as expected. +- Verify that the cluster is in good shape functional-wise, responds to writes and queries. + +### Rename data nodes + +- Find the meta node leader with `curl localhost:8091/status`. The `leader` field in the JSON output reports the leader meta node. +- Stop the service on the data node you want to rename. Edit its configuration file to set the new `hostname` under `/etc/influxdb/influxdb.conf`. +- Update the actual OS host's name if needed, apply DNS changes. +- Start the data service. Errors will be logged until it is added to the cluster again. +- On the meta node leader, run `influxd-ctl update-data oldname:8088 newname:8088`. Upon success you will get a message updated data node ID to `newname:8088`. +- Verify with `influxd-ctl show` on the meta node leader. Verify there are no errors in the logs of the updated data node and other data nodes. Restart the service on the updated data node. Verify writes, replication and queries work as expected. +- Repeat on the remaining data nodes. Remember to only execute the `update-data` command from the meta leader. + +### Final verification + +- Verify the state of the cluster with `influxd-ctl show`. The version must be reported on all nodes for them to be healthy. +- Verify the `show shards` output lists all shards and node ownership as expected. +- Verify meta queries work (show measurements under a database). +- Verify data are being queried successfully. + +Once you've performed the verification steps, resume write and query activity. diff --git a/content/enterprise_influxdb/v1.9/administration/renew-license.md b/content/enterprise_influxdb/v1.9/administration/renew-license.md new file mode 100644 index 000000000..5f1d69bf9 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/administration/renew-license.md @@ -0,0 +1,22 @@ +--- +title: Renew or update a license key or file +description: > + Renew or update a license key or file for your InfluxDB enterprise cluster. +menu: + enterprise_influxdb_1_9: + name: Renew a license + weight: 50 + parent: Administration +--- + +Use this procedure to renew or update an existing license key or file, switch from a license key to a license file, or switch from a license file to a license key. + +> **Note:** To request a new license to renew or expand your InfluxDB Enterprise cluster, contact [sales@influxdb.com](mailto:sales@influxdb.com). + +To update a license key or file, do the following: + +1. If you are switching from a license key to a license file (or vice versa), delete your existing license key or file. +2. **Add the license key or file** to your [meta nodes](/enterprise_influxdb/v1.9/administration/config-meta-nodes/#enterprise-license-settings) and [data nodes](/enterprise_influxdb/v1.9/administration/config-data-nodes/#enterprise-license-settings) configuration settings. For more information, see [how to configure InfluxDB Enterprise clusters](/enterprise_influxdb/v1.9/administration/configuration/). +3. **On each meta node**, run `service influxdb-meta restart`, and wait for the meta node service to come back up successfully before restarting the next meta node. +The cluster should remain unaffected as long as only one node is restarting at a time. +4. **On each data node**, run `killall -s HUP influxd` to signal the `influxd` process to reload its configuration file. diff --git a/content/enterprise_influxdb/v1.9/administration/security.md b/content/enterprise_influxdb/v1.9/administration/security.md new file mode 100644 index 000000000..3d648f0bc --- /dev/null +++ b/content/enterprise_influxdb/v1.9/administration/security.md @@ -0,0 +1,53 @@ +--- +title: Manage security in InfluxDB Enterprise +description: Protect the data in your InfluxDB Enterprise instance. +menu: + enterprise_influxdb_1_9: + name: Manage security + weight: 110 + parent: Administration +--- + +Some customers may choose to install InfluxDB Enterprise with public internet access, however doing so can inadvertently expose your data and invite unwelcome attacks on your database. +Check out the sections below for how protect the data in your InfluxDB Enterprise instance. + +## Enable authentication + +Password protect your InfluxDB Enterprise instance to keep any unauthorized individuals +from accessing your data. + +Resources: +[Set up Authentication](/enterprise_influxdb/v1.9/administration/authentication_and_authorization/#set-up-authentication) + +## Manage users and permissions + +Restrict access by creating individual users and assigning them relevant +read and/or write permissions. + +Resources: +[User types and privileges](/enterprise_influxdb/v1.9/administration/authentication_and_authorization/#user-types-and-privileges), +[User management commands](/enterprise_influxdb/v1.9/administration/authentication_and_authorization/#user-management-commands), +[Fine-grained authorization](/enterprise_influxdb/v1.9/guides/fine-grained-authorization/) + +## Enable HTTPS + +Using HTTPS secures the communication between clients and the InfluxDB server, and, in +some cases, HTTPS verifies the authenticity of the InfluxDB server to clients (bi-directional authentication). +The communicatio between the meta nodes and the data nodes are also secured via HTTPS. + +Resources: +[Enabling HTTPS](/enterprise_influxdb/v1.9/guides/https_setup/) + +## Secure your host + +### Ports + +For InfluxDB Enterprise data nodes, close all ports on each host except for port `8086`. +You can also use a proxy to port `8086`. By default, data nodes and meta nodes communicate with each other over '8088','8089',and'8091' + +For InfluxDB Enterprise, [backing up and restoring](/enterprise_influxdb/v1.9/administration/backup-and-restore/) is performed from the meta nodes. + + +### AWS Recommendations + +InfluxData recommends implementing on-disk encryption; InfluxDB does not offer built-in support to encrypt the data. diff --git a/content/enterprise_influxdb/v1.9/administration/server_monitoring.md b/content/enterprise_influxdb/v1.9/administration/server_monitoring.md new file mode 100644 index 000000000..14bcec62d --- /dev/null +++ b/content/enterprise_influxdb/v1.9/administration/server_monitoring.md @@ -0,0 +1,167 @@ +--- +title: Monitor InfluxDB servers +description: Troubleshoot and monitor InfluxDB OSS. +aliases: + - /enterprise_influxdb/v1.9/administration/statistics/ + - /enterprise_influxdb/v1.9/troubleshooting/statistics/ +menu: + enterprise_influxdb_1_9: + name: Monitor InfluxDB + weight: 80 + parent: Administration +--- + +**On this page** + +* [SHOW STATS](#show-stats) +* [SHOW DIAGNOSTICS](#show-diagnostics) +* [Internal monitoring](#internal-monitoring) +* [Useful performance metrics commands](#useful-performance-metrics-commands) +* [InfluxDB `/metrics` HTTP endpoint](#influxdb-metrics-http-endpoint) + + +InfluxDB can display statistical and diagnostic information about each node. +This information can be very useful for troubleshooting and performance monitoring. + +## SHOW STATS + +To see node statistics, execute the command `SHOW STATS`. +For details on this command, see [`SHOW STATS`](/enterprise_influxdb/v1.9/query_language/spec#show-stats) in the InfluxQL specification. + +The statistics returned by `SHOW STATS` are stored in memory only, and are reset to zero when the node is restarted. + +## SHOW DIAGNOSTICS + +To see node diagnostic information, execute the command `SHOW DIAGNOSTICS`. +This returns information such as build information, uptime, hostname, server configuration, memory usage, and Go runtime diagnostics. +For details on this command, see [`SHOW DIAGNOSTICS`](/enterprise_influxdb/v1.9/query_language/spec#show-diagnostics) in the InfluxQL specification. + +## Internal monitoring +InfluxDB also writes statistical and diagnostic information to database named `_internal`, which records metrics on the internal runtime and service performance. +The `_internal` database can be queried and manipulated like any other InfluxDB database. +Check out the [monitor service README](https://github.com/influxdata/influxdb/blob/1.8/monitor/README.md) and the [internal monitoring blog post](https://www.influxdata.com/blog/how-to-use-the-show-stats-command-and-the-_internal-database-to-monitor-influxdb/) for more detail. + +## Useful performance metrics commands + +Below are a collection of commands to find useful performance metrics about your InfluxDB instance. + +To find the number of points per second being written to the instance. Must have the `monitor` service enabled: +```bash +$ influx -execute 'select derivative(pointReq, 1s) from "write" where time > now() - 5m' -database '_internal' -precision 'rfc3339' +``` + +To find the number of writes separated by database since the beginnning of the log file: + +```bash +grep 'POST' /var/log/influxdb/influxd.log | awk '{ print $10 }' | sort | uniq -c +``` + +Or, for systemd systems logging to journald: + +```bash +journalctl -u influxdb.service | awk '/POST/ { print $10 }' | sort | uniq -c +``` + +### InfluxDB `/metrics` HTTP endpoint + +> ***Note:*** There are no outstanding PRs for improvements to the `/metrics` endpoint, but we’ll add them to the CHANGELOG as they occur. + +The InfluxDB `/metrics` endpoint is configured to produce the default Go metrics in Prometheus metrics format. + + +#### Example using InfluxDB `/metrics' endpoint + +Below is an example of the output generated using the `/metrics` endpoint. Note that HELP is available to explain the Go statistics. + +``` +# HELP go_gc_duration_seconds A summary of the GC invocation durations. +# TYPE go_gc_duration_seconds summary +go_gc_duration_seconds{quantile="0"} 6.4134e-05 +go_gc_duration_seconds{quantile="0.25"} 8.8391e-05 +go_gc_duration_seconds{quantile="0.5"} 0.000131335 +go_gc_duration_seconds{quantile="0.75"} 0.000169204 +go_gc_duration_seconds{quantile="1"} 0.000544705 +go_gc_duration_seconds_sum 0.004619405 +go_gc_duration_seconds_count 27 +# HELP go_goroutines Number of goroutines that currently exist. +# TYPE go_goroutines gauge +go_goroutines 29 +# HELP go_info Information about the Go environment. +# TYPE go_info gauge +go_info{version="go1.10"} 1 +# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use. +# TYPE go_memstats_alloc_bytes gauge +go_memstats_alloc_bytes 1.581062048e+09 +# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed. +# TYPE go_memstats_alloc_bytes_total counter +go_memstats_alloc_bytes_total 2.808293616e+09 +# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table. +# TYPE go_memstats_buck_hash_sys_bytes gauge +go_memstats_buck_hash_sys_bytes 1.494326e+06 +# HELP go_memstats_frees_total Total number of frees. +# TYPE go_memstats_frees_total counter +go_memstats_frees_total 1.1279913e+07 +# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started. +# TYPE go_memstats_gc_cpu_fraction gauge +go_memstats_gc_cpu_fraction -0.00014404354379774563 +# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata. +# TYPE go_memstats_gc_sys_bytes gauge +go_memstats_gc_sys_bytes 6.0936192e+07 +# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use. +# TYPE go_memstats_heap_alloc_bytes gauge +go_memstats_heap_alloc_bytes 1.581062048e+09 +# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used. +# TYPE go_memstats_heap_idle_bytes gauge +go_memstats_heap_idle_bytes 3.8551552e+07 +# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use. +# TYPE go_memstats_heap_inuse_bytes gauge +go_memstats_heap_inuse_bytes 1.590673408e+09 +# HELP go_memstats_heap_objects Number of allocated objects. +# TYPE go_memstats_heap_objects gauge +go_memstats_heap_objects 1.6924595e+07 +# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS. +# TYPE go_memstats_heap_released_bytes gauge +go_memstats_heap_released_bytes 0 +# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system. +# TYPE go_memstats_heap_sys_bytes gauge +go_memstats_heap_sys_bytes 1.62922496e+09 +# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection. +# TYPE go_memstats_last_gc_time_seconds gauge +go_memstats_last_gc_time_seconds 1.520291233297057e+09 +# HELP go_memstats_lookups_total Total number of pointer lookups. +# TYPE go_memstats_lookups_total counter +go_memstats_lookups_total 397 +# HELP go_memstats_mallocs_total Total number of mallocs. +# TYPE go_memstats_mallocs_total counter +go_memstats_mallocs_total 2.8204508e+07 +# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures. +# TYPE go_memstats_mcache_inuse_bytes gauge +go_memstats_mcache_inuse_bytes 13888 +# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system. +# TYPE go_memstats_mcache_sys_bytes gauge +go_memstats_mcache_sys_bytes 16384 +# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures. +# TYPE go_memstats_mspan_inuse_bytes gauge +go_memstats_mspan_inuse_bytes 1.4781696e+07 +# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system. +# TYPE go_memstats_mspan_sys_bytes gauge +go_memstats_mspan_sys_bytes 1.4893056e+07 +# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place. +# TYPE go_memstats_next_gc_bytes gauge +go_memstats_next_gc_bytes 2.38107752e+09 +# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations. +# TYPE go_memstats_other_sys_bytes gauge +go_memstats_other_sys_bytes 4.366786e+06 +# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator. +# TYPE go_memstats_stack_inuse_bytes gauge +go_memstats_stack_inuse_bytes 983040 +# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator. +# TYPE go_memstats_stack_sys_bytes gauge +go_memstats_stack_sys_bytes 983040 +# HELP go_memstats_sys_bytes Number of bytes obtained from system. +# TYPE go_memstats_sys_bytes gauge +go_memstats_sys_bytes 1.711914744e+09 +# HELP go_threads Number of OS threads created. +# TYPE go_threads gauge +go_threads 16 +``` diff --git a/content/enterprise_influxdb/v1.9/administration/stability_and_compatibility.md b/content/enterprise_influxdb/v1.9/administration/stability_and_compatibility.md new file mode 100644 index 000000000..e217f2491 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/administration/stability_and_compatibility.md @@ -0,0 +1,29 @@ +--- +title: Stability and compatibility +description: > + API and storage engine compatibility and stability in InfluxDB OSS. +menu: + enterprise_influxdb_1_9: + weight: 90 + parent: Administration +--- + +## 1.x API compatibility and stability + +One of the more important aspects of the 1.0 release is that this marks the stabilization of our API and storage format. Over the course of the last three years we’ve iterated aggressively, often breaking the API in the process. With the release of 1.0 and for the entire 1.x line of releases we’re committing to the following: + +### No breaking InfluxDB API changes + +When it comes to the InfluxDB API, if a command works in 1.0 it will work unchanged in all 1.x releases...with one caveat. We will be adding [keywords](/enterprise_influxdb/v1.9/query_language/spec/#keywords) to the query language. New keywords won't break your queries if you wrap all [identifiers](/enterprise_influxdb/v1.9/concepts/glossary/#identifier) in double quotes and all string literals in single quotes. This is generally considered best practice so it should be followed anyway. For users following that guideline, the query and ingestion APIs will have no breaking changes for all 1.x releases. Note that this does not include the Go code in the project. The underlying Go API in InfluxDB can and will change over the course of 1.x development. Users should be accessing InfluxDB through the [InfluxDB API](/enterprise_influxdb/v1.9/tools/api/). + +### Storage engine stability + +The [TSM](/enterprise_influxdb/v1.9/concepts/glossary/#tsm-time-structured-merge-tree) storage engine file format is now at version 1. While we may introduce new versions of the format in the 1.x releases, these new versions will run side-by-side with previous versions. What this means for users is there will be no lengthy migrations when upgrading from one 1.x release to another. + +### Additive changes + +The query engine will have additive changes over the course of the new releases. We’ll introduce new query functions and new functionality into the language without breaking backwards compatibility. We may introduce new protocol endpoints (like a binary format) and versions of the line protocol and query API to improve performance and/or functionality, but they will have to run in parallel with the existing versions. Existing versions will be supported for the entirety of the 1.x release line. + +### Ongoing support + +We’ll continue to fix bugs on the 1.x versions of the [line protocol](/enterprise_influxdb/v1.9/concepts/glossary/#influxdb-line-protocol), query API, and TSM storage format. Users should expect to upgrade to the latest 1.x.x release for bug fixes, but those releases will all be compatible with the 1.0 API and won’t require data migrations. For instance, if a user is running 1.2 and there are bug fixes released in 1.3, they should upgrade to the 1.3 release. Until 1.4 is released, patch fixes will go into 1.3.x. Because all future 1.x releases are drop in replacements for previous 1.x releases, users should upgrade to the latest in the 1.x line to get all bug fixes. diff --git a/content/enterprise_influxdb/v1.9/administration/subscription-management.md b/content/enterprise_influxdb/v1.9/administration/subscription-management.md new file mode 100644 index 000000000..8f20e2df4 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/administration/subscription-management.md @@ -0,0 +1,208 @@ +--- +title: Manage subscriptions in InfluxDB +description: > + Manage subscriptions, which copy all written data to a local or remote endpoint, in InfluxDB OSS. +menu: + enterprise_influxdb_1_9: + parent: Administration + name: Manage subscriptions + weight: 100 +--- + +InfluxDB subscriptions are local or remote endpoints to which all data written to InfluxDB is copied. +Subscriptions are primarily used with [Kapacitor](/kapacitor/), but any endpoint +able to accept UDP, HTTP, or HTTPS connections can subscribe to InfluxDB and receive +a copy of all data as it is written. + +## How subscriptions work + +As data is written to InfluxDB, writes are duplicated to subscriber endpoints via +HTTP, HTTPS, or UDP in [line protocol](/enterprise_influxdb/v1.9/write_protocols/line_protocol_tutorial/). +the InfluxDB subscriber service creates multiple "writers" ([goroutines](https://golangbot.com/goroutines/)) +which send writes to the subscription endpoints. + +_The number of writer goroutines is defined by the [`write-concurrency`](/enterprise_influxdb/v1.9/administration/config#write-concurrency-40) configuration._ + +As writes occur in InfluxDB, each subscription writer sends the written data to the +specified subscription endpoints. +However, with a high `write-concurrency` (multiple writers) and a high ingest rate, +nanosecond differences in writer processes and the transport layer can result +in writes being received out of order. + +> #### Important information about high write loads +> While setting the subscriber `write-concurrency` to greater than 1 does increase your +> subscriber write throughput, it can result in out-of-order writes under high ingest rates. +> Setting `write-concurrency` to 1 ensures writes are passed to subscriber endpoints sequentially, +> but can create a bottleneck under high ingest rates. +> +> What `write-concurrency` should be set to depends on your specific workload +> and need for in-order writes to your subscription endpoint. + +## InfluxQL subscription statements + +Use the following InfluxQL statements to manage subscriptions: + +[`CREATE SUBSCRIPTION`](#create-subscriptions) +[`SHOW SUBSCRIPTIONS`](#show-subscriptions) +[`DROP SUBSCRIPTION`](#remove-subscriptions) + +## Create subscriptions + +Create subscriptions using the `CREATE SUBSCRIPTION` InfluxQL statement. +Specify the subscription name, the database name and retention policy to subscribe to, +and the URL of the host to which data written to InfluxDB should be copied. + +```sql +-- Pattern: +CREATE SUBSCRIPTION "" ON ""."" DESTINATIONS "" + +-- Examples: +-- Create a SUBSCRIPTION on database 'mydb' and retention policy 'autogen' that sends data to 'example.com:9090' via HTTP. +CREATE SUBSCRIPTION "sub0" ON "mydb"."autogen" DESTINATIONS ALL 'http://example.com:9090' + +-- Create a SUBSCRIPTION on database 'mydb' and retention policy 'autogen' that round-robins the data to 'h1.example.com:9090' and 'h2.example.com:9090' via UDP. +CREATE SUBSCRIPTION "sub0" ON "mydb"."autogen" DESTINATIONS ANY 'udp://h1.example.com:9090', 'udp://h2.example.com:9090' +``` +In case authentication is enabled on the subscriber host, adapt the URL to contain the credentials. + +``` +-- Create a SUBSCRIPTION on database 'mydb' and retention policy 'autogen' that sends data to another InfluxDB on 'example.com:8086' via HTTP. Authentication is enabled on the subscription host (user: subscriber, pass: secret). +CREATE SUBSCRIPTION "sub0" ON "mydb"."autogen" DESTINATIONS ALL 'http://subscriber:secret@example.com:8086' +``` + +{{% warn %}} +`SHOW SUBSCRIPTIONS` outputs all subscriber URL in plain text, including those with authentication credentials. +Any user with the privileges to run `SHOW SUBSCRIPTIONS` is able to see these credentials. +{{% /warn %}} + +### Sending subscription data to multiple hosts + +The `CREATE SUBSCRIPTION` statement allows you to specify multiple hosts as endpoints for the subscription. +In your `DESTINATIONS` clause, you can pass multiple host strings separated by commas. +Using `ALL` or `ANY` in the `DESTINATIONS` clause determines how InfluxDB writes data to each endpoint: + +`ALL`: Writes data to all specified hosts. + +`ANY`: Round-robins writes between specified hosts. + +_**Subscriptions with multiple hosts**_ + +```sql +-- Write all data to multiple hosts +CREATE SUBSCRIPTION "mysub" ON "mydb"."autogen" DESTINATIONS ALL 'http://host1.example.com:9090', 'http://host2.example.com:9090' + +-- Round-robin writes between multiple hosts +CREATE SUBSCRIPTION "mysub" ON "mydb"."autogen" DESTINATIONS ANY 'http://host1.example.com:9090', 'http://host2.example.com:9090' +``` + +### Subscription protocols + +Subscriptions can use HTTP, HTTPS, or UDP transport protocols. +Which to use is determined by the protocol expected by the subscription endpoint. +If creating a Kapacitor subscription, this is defined by the `subscription-protocol` +option in the `[[influxdb]]` section of your [`kapacitor.conf`](/{{< latest "kapacitor" >}}/administration/subscription-management/#subscription-protocol). + +_**kapacitor.conf**_ + +```toml +[[influxdb]] + + # ... + + subscription-protocol = "http" + + # ... + +``` + +_For information regarding HTTPS connections and secure communication between InfluxDB and Kapacitor, +view the [Kapacitor security](/kapacitor/v1.5/administration/security/#secure-influxdb-and-kapacitor) documentation._ + +## Show subscriptions + +The `SHOW SUBSCRIPTIONS` InfluxQL statement returns a list of all subscriptions registered in InfluxDB. + +```sql +SHOW SUBSCRIPTIONS +``` + +_**Example output:**_ + +```bash +name: _internal +retention_policy name mode destinations +---------------- ---- ---- ------------ +monitor kapacitor-39545771-7b64-4692-ab8f-1796c07f3314 ANY [http://localhost:9092] +``` + +## Remove subscriptions + +Remove or drop subscriptions using the `DROP SUBSCRIPTION` InfluxQL statement. + +```sql +-- Pattern: +DROP SUBSCRIPTION "" ON ""."" + +-- Example: +DROP SUBSCRIPTION "sub0" ON "mydb"."autogen" +``` + +### Drop all subscriptions + +In some cases, it may be necessary to remove all subscriptions. +Run the following bash script that utilizes the `influx` CLI, loops through all subscriptions, and removes them. +This script depends on the `$INFLUXUSER` and `$INFLUXPASS` environment variables. +If these are not set, export them as part of the script. + +```bash +# Environment variable exports: +# Uncomment these if INFLUXUSER and INFLUXPASS are not already globally set. +# export INFLUXUSER=influxdb-username +# export INFLUXPASS=influxdb-password + +IFS=$'\n'; for i in $(influx -format csv -username $INFLUXUSER -password $INFLUXPASS -database _internal -execute 'show subscriptions' | tail -n +2 | grep -v name); do influx -format csv -username $INFLUXUSER -password $INFLUXPASS -database _internal -execute "drop subscription \"$(echo "$i" | cut -f 3 -d ',')\" ON \"$(echo "$i" | cut -f 1 -d ',')\".\"$(echo "$i" | cut -f 2 -d ',')\""; done +``` + +## Configure InfluxDB subscriptions + +InfluxDB subscription configuration options are available in the `[subscriber]` +section of the `influxdb.conf`. +In order to use subcriptions, the `enabled` option in the `[subscriber]` section must be set to `true`. +Below is an example `influxdb.conf` subscriber configuration: + +```toml +[subscriber] + enabled = true + http-timeout = "30s" + insecure-skip-verify = false + ca-certs = "" + write-concurrency = 40 + write-buffer-size = 1000 +``` + +_**Descriptions of `[subscriber]` configuration options are available in the [Configuring InfluxDB](/enterprise_influxdb/v1.9/administration/config#subscription-settings) documentation.**_ + +## Troubleshooting + +### Inaccessible or decommissioned subscription endpoints + +Unless a subscription is [dropped](#remove-subscriptions), InfluxDB assumes the endpoint +should always receive data and will continue to attempt to send data. +If an endpoint host is inaccessible or has been decommissioned, you will see errors +similar to the following: + +```bash +# Some message content omitted (...) for the sake of brevity +"Post http://x.y.z.a:9092/write?consistency=...: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" ... service=subscriber +"Post http://x.y.z.a:9092/write?consistency=...: dial tcp x.y.z.a:9092: getsockopt: connection refused" ... service=subscriber +"Post http://x.y.z.a:9092/write?consistency=...: dial tcp 172.31.36.5:9092: getsockopt: no route to host" ... service=subscriber +``` + +In some cases, this may be caused by a networking error or something similar +preventing a successful connection to the subscription endpoint. +In other cases, it's because the subscription endpoint no longer exists and +the subscription hasn't been dropped from InfluxDB. + +> Because InfluxDB does not know if a subscription endpoint will or will not become accessible again, +> subscriptions are not automatically dropped when an endpoint becomes inaccessible. +> If a subscription endpoint is removed, you must manually [drop the subscription](#remove-subscriptions) from InfluxDB. diff --git a/content/enterprise_influxdb/v1.9/administration/upgrading.md b/content/enterprise_influxdb/v1.9/administration/upgrading.md new file mode 100644 index 000000000..7b0c25d7e --- /dev/null +++ b/content/enterprise_influxdb/v1.9/administration/upgrading.md @@ -0,0 +1,245 @@ +--- +title: Upgrade InfluxDB Enterprise clusters +description: Upgrade to the latest version of InfluxDB Enterprise. +aliases: + - /enterprise/v1.8/administration/upgrading/ +menu: + enterprise_influxdb_1_9: + name: Upgrade + weight: 50 + parent: Administration +--- + +To successfully perform a rolling upgrade of InfluxDB Enterprise clusters to {{< latest-patch >}}, complete the following steps: + +1. [Back up your cluster](#back-up-your-cluster). +2. [Upgrade meta nodes](#upgrade-meta-nodes). +3. [Upgrade data nodes](#upgrade-data-nodes). + +> ***Note:*** A rolling upgrade lets you update your cluster with zero downtime. To downgrade to an earlier version, complete the following procedures, replacing the version numbers with the version that you want to downgrade to. + +## Back up your cluster + +Before performing an upgrade, create a full backup of your InfluxDB Enterprise cluster. Also, if you create incremental backups, trigger a final incremental backup. + +> ***Note:*** For information on performing a final incremental backup or a full backup, +> see [Back up and restore InfluxDB Enterprise clusters](/enterprise_influxdb/v1.9/administration/backup-and-restore/). + +## Upgrade meta nodes + +Complete the following steps to upgrade meta nodes: + +1. [Download the meta node package](#download-the-meta-node-package). +2. [Install the meta node package](#install-the-meta-node-package). +3. [Update the meta node configuration file](#update-the-meta-node-configuration-file). +4. [Restart the `influxdb-meta` service](#restart-the-influxdb-meta-service). +5. Repeat steps 1-4 for each meta node in your cluster. +6. [Confirm the meta nodes upgrade](#confirm-the-meta-nodes-upgrade). + +### Download the meta node package + +##### Ubuntu and Debian (64-bit) + +```bash +wget https://dl.influxdata.com/enterprise/releases/influxdb-meta_{{< latest-patch >}}-c{{< latest-patch >}}_amd64.deb +``` + +##### RedHat and CentOS (64-bit) + +```bash +wget https://dl.influxdata.com/enterprise/releases/influxdb-meta-{{< latest-patch >}}_c{{< latest-patch >}}.x86_64.rpm +``` + +### Install the meta node package + +##### Ubuntu and Debian (64-bit) + +```bash +sudo dpkg -i influxdb-meta_{{< latest-patch >}}-c{{< latest-patch >}}_amd64.deb +``` + +##### RedHat and CentOS (64-bit) + +```bash +sudo yum localinstall influxdb-meta-{{< latest-patch >}}-c{{< latest-patch >}}.x86_64.rpm +``` + +### Update the meta node configuration file + +Migrate any custom settings from your previous meta node configuration file. + +To enable HTTPS, you must update the meta node configuration file (`influxdb-meta.conf`). For information, see [Enable HTTPS within the configuration file for each Meta Node](/enterprise_influxdb/v1.9/guides/https_setup/#set-up-https-in-an-influxdb-enterprise-cluster). + +### Restart the `influxdb-meta` service + +##### sysvinit systems + +```bash +service influxdb-meta restart +``` + +##### systemd systems + +```bash +sudo systemctl restart influxdb-meta +``` + +### Confirm the meta nodes upgrade + +After upgrading _**all**_ meta nodes, check your node version numbers using the +`influxd-ctl show` command. +The [`influxd-ctl` utility](/enterprise_influxdb/v1.9/tools/influxd-ctl/) is available on all meta nodes. + +```bash +~# influxd-ctl show + +Data Nodes +========== +ID TCP Address Version +4 rk-upgrading-01:8088 1.8.x_c1.8.y +5 rk-upgrading-02:8088 1.8.x_c1.8.y +6 rk-upgrading-03:8088 1.8.x_c1.8.y + +Meta Nodes +========== +TCP Address Version +rk-upgrading-01:8091 {{< latest-patch >}}-c{{< latest-patch >}} # {{< latest-patch >}}-c{{< latest-patch >}} = 👍 +rk-upgrading-02:8091 {{< latest-patch >}}-c{{< latest-patch >}} +rk-upgrading-03:8091 {{< latest-patch >}}-c{{< latest-patch >}} +``` + +Ensure that the meta cluster is healthy before upgrading the data nodes. + +## Upgrade data nodes + +Complete the following steps to upgrade data nodes: + +1. [Download the data node package](#download-the-data-node-package). +2. [Stop traffic to data nodes](#stop-traffic-to-the-data-node). +3. [Install the data node package](#install-the-data-node-package). +4. [Update the data node configuration file](#update-the-data-node-configuration-file). +5. For Time Series Index (TSI) only. [Rebuild TSI indexes](#rebuild-tsi-indexes). +6. [Restart the `influxdb` service](#restart-the-influxdb-service). +7. [Restart traffic to data nodes](#restart-traffic-to-data-nodes). +8. Repeat steps 1-7 for each data node in your cluster. +9. [Confirm the data nodes upgrade](#confirm-the-data-nodes-upgrade). + +### Download the data node package + +##### Ubuntu and Debian (64-bit) + +```bash +wget https://dl.influxdata.com/enterprise/releases/influxdb-data_{{< latest-patch >}}-c{{< latest-patch >}}_amd64.deb +``` + +##### RedHat and CentOS (64-bit) + +```bash +wget https://dl.influxdata.com/enterprise/releases/influxdb-data-{{< latest-patch >}}-c{{< latest-patch >}}.x86_64.rpm +``` + +### Stop traffic to the data node + +- If you have access to the load balancer configuration, use your load balancer to stop routing read and write requests to the data node server (port 8086). + +- If you cannot access the load balancer configuration, work with your networking team to prevent traffic to the data node server before continuing to upgrade. + +### Install the data node package + +When you run the install command, you're prompted to keep or overwrite your current configuration file with the file for version {{< latest-patch >}}. Enter `N` or `O` to keep your current configuration file. You'll make the configuration changes for version {{< latest-patch >}}. in the next procedure, [Update the data node configuration file](#update-the-data-node-configuration-file). + + +##### Ubuntu and Debian (64-bit) + +```bash +sudo dpkg -i influxdb-data_{{< latest-patch >}}-c{{< latest-patch >}}_amd64.deb +``` + +##### RedHat & CentOS (64-bit) + +```bash +sudo yum localinstall influxdb-data-{{< latest-patch >}}-c{{< latest-patch >}}.x86_64.rpm +``` + +### Update the data node configuration file + +Migrate any custom settings from your previous data node configuration file. + +- To enable HTTPS, see [Enable HTTPS within the configuration file for each Data Node](/enterprise_influxdb/v1.9/guides/https_setup/#set-up-https-in-an-influxdb-enterprise-cluster). + +- To enable TSI, open `/etc/influxdb/influxdb.conf`, and then adjust and save the settings shown in the following table. + + | Section | Setting | + | --------| ----------------------------------------------------------| + | `[data]` |
  • To use Time Series Index (TSI) disk-based indexing, add [`index-version = "tsi1"`](/enterprise_influxdb/v1.9/administration/config-data-nodes#index-version-inmem)
  • To use TSM in-memory index, add [`index-version = "inmem"`](/enterprise_influxdb/v1.9/administration/config-data-nodes#index-version-inmem)
  • Add [`wal-fsync-delay = "0s"`](/enterprise_influxdb/v1.9/administration/config-data-nodes#wal-fsync-delay-0s)
  • Add [`max-concurrent-compactions = 0`](/enterprise_influxdb/v1.9/administration/config-data-nodes#max-concurrent-compactions-0)
  • Set[`cache-max-memory-size`](/enterprise_influxdb/v1.9/administration/config-data-nodes#cache-max-memory-size-1g) to `1073741824` | + | `[cluster]`|
    • Add [`pool-max-idle-streams = 100`](/enterprise_influxdb/v1.9/administration/config-data-nodes#pool-max-idle-streams-100)
    • Add[`pool-max-idle-time = "1m0s"`](/enterprise_influxdb/v1.9/administration/config-data-nodes#pool-max-idle-time-60s)
    • Remove `max-remote-write-connections` + |[`[anti-entropy]`](/enterprise_influxdb/v1.9/administration/config-data-nodes#anti-entropy)|
      • Add `enabled = true`
      • Add `check-interval = "30s"`
      • Add `max-fetch = 10`| + |`[admin]`| Remove entire section.| + + For more information about TSI, see [TSI overview](/enterprise_influxdb/v1.9/concepts/time-series-index/) and [TSI details](/enterprise_influxdb/v1.9/concepts/tsi-details/). + +### Rebuild TSI indexes + +Complete the following steps for Time Series Index (TSI) only. + +1. Delete all `_series` directories in the `/data` directory (by default, stored at `/data//_series`). + +2. Delete all TSM-based shard `index` directories (by default, located at `/data///index`). + +3. Use the [`influx_inspect buildtsi`](/enterprise_influxdb/v1.9/tools/influx_inspect#buildtsi) utility to rebuild the TSI index. For example, run the following command: + + ```js + influx_inspect buildtsi -datadir /yourDataDirectory -waldir /wal + ``` + + Replacing `yourDataDirectory` with the name of your directory. Running this command converts TSM-based shards to TSI shards or rebuilds existing TSI shards. + + > **Note:** Run the `buildtsi` command using the same system user that runs the `influxd` service, or a user with the same permissions. + +### Restart the `influxdb` service + +Restart the `influxdb` service to restart the data nodes. + +##### sysvinit systems + +```bash +service influxdb restart +``` + +##### systemd systems + +```bash +sudo systemctl restart influxdb +``` + +### Restart traffic to data nodes + +Restart routing read and write requests to the data node server (port 8086) through your load balancer. + +> **Note:** Allow the hinted handoff queue (HHQ) to write all missed data to the updated node before upgrading the next data node. Once all data has been written, the disk space used in the hinted handoff queue should be 0. Check the disk space on your hh directory by running the [`du`] command, for example, `du /var/lib/influxdb/hh`. + +### Confirm the data nodes upgrade + +After upgrading _**all**_ data nodes, check your node version numbers using the +`influxd-ctl show` command. +The [`influxd-ctl` utility](/enterprise_influxdb/v1.9/tools/influxd-ctl/) is available on all meta nodes. + +```bash +~# influxd-ctl show + +Data Nodes +========== +ID TCP Address Version +4 rk-upgrading-01:8088 {{< latest-patch >}}-c{{< latest-patch >}} # {{< latest-patch >}}-c{{< latest-patch >}} = 👍 +5 rk-upgrading-02:8088 {{< latest-patch >}}-c{{< latest-patch >}} +6 rk-upgrading-03:8088 {{< latest-patch >}}-c{{< latest-patch >}} + +Meta Nodes +========== +TCP Address Version +rk-upgrading-01:8091 {{< latest-patch >}}-c{{< latest-patch >}} +rk-upgrading-02:8091 {{< latest-patch >}}-c{{< latest-patch >}} +rk-upgrading-03:8091 {{< latest-patch >}}-c{{< latest-patch >}} +``` + +If you have any issues upgrading your cluster, contact InfluxData support. diff --git a/content/enterprise_influxdb/v1.9/concepts/_index.md b/content/enterprise_influxdb/v1.9/concepts/_index.md new file mode 100644 index 000000000..8e048e741 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/concepts/_index.md @@ -0,0 +1,12 @@ +--- +title: InfluxDB Enterprise concepts +description: Clustering and other key concepts in InfluxDB Enterprise. +aliases: + - /enterprise/v1.9/concepts/ +menu: + enterprise_influxdb_1_9: + name: Concepts + weight: 50 +--- + +{{< children hlevel="h2" type="list" >}} diff --git a/content/enterprise_influxdb/v1.9/concepts/clustering.md b/content/enterprise_influxdb/v1.9/concepts/clustering.md new file mode 100644 index 000000000..ca2379b7c --- /dev/null +++ b/content/enterprise_influxdb/v1.9/concepts/clustering.md @@ -0,0 +1,138 @@ +--- +title: Clustering in InfluxDB Enterprise +description: > + Learn how meta nodes, data nodes, and the Enterprise web server interact in InfluxDB Enterprise. +aliases: + - /enterprise/v1.9/concepts/clustering/ +menu: + enterprise_influxdb_1_9: + name: Clustering + weight: 10 + parent: Concepts +--- + +This document describes in detail how clustering works in InfluxDB Enterprise. It starts with a high level description of the different components of a cluster and then delves into the implementation details. + +## Architectural overview + +An InfluxDB Enterprise installation consists of three separate software processes: data nodes, meta nodes, and the Enterprise web server. To run an InfluxDB cluster, only the meta and data nodes are required. Communication within a cluster looks like this: + +{{< diagram >}} +flowchart TB + subgraph meta[Meta Nodes] + Meta1 <-- TCP :8089 --> Meta2 <-- TCP :8089 --> Meta3 + end + meta <-- HTTP :8091 --> data + subgraph data[Data Nodes] + Data1 <-- TCP :8088 --> Data2 + end +{{< /diagram >}} + + +The meta nodes communicate with each other via a TCP protocol and the Raft consensus protocol that all use port `8089` by default. This port must be reachable between the meta nodes. The meta nodes also expose an HTTP API bound to port `8091` by default that the `influxd-ctl` command uses. + +Data nodes communicate with each other through a TCP protocol that is bound to port `8088`. Data nodes communicate with the meta nodes through their HTTP API bound to `8091`. These ports must be reachable between the meta and data nodes. + +Within a cluster, all meta nodes must communicate with all other meta nodes. All data nodes must communicate with all other data nodes and all meta nodes. + +The meta nodes keep a consistent view of the metadata that describes the cluster. The meta cluster uses the [HashiCorp implementation of Raft](https://github.com/hashicorp/raft) as the underlying consensus protocol. This is the same implementation that they use in Consul. + +The data nodes replicate data and query each other using the Protobuf protocol over TCP. Details on replication and querying are covered later in this document. + +## Where data lives + +The meta and data nodes are each responsible for different parts of the database. + +### Meta nodes + +Meta nodes hold all of the following meta data: + +* all nodes in the cluster and their role +* all databases and retention policies that exist in the cluster +* all shards and shard groups, and on what nodes they exist +* cluster users and their permissions +* all continuous queries + +The meta nodes keep this data in the Raft database on disk, backed by BoltDB. By default the Raft database is `/var/lib/influxdb/meta/raft.db`. + +> **Note:** Meta nodes require the `/meta` directory. + +### Data nodes + +Data nodes hold all of the raw time series data and metadata, including: + +* measurements +* tag keys and values +* field keys and values + +On disk, the data is always organized by `//`. By default the parent directory is `/var/lib/influxdb/data`. + +> **Note:** Data nodes require all four subdirectories of `/var/lib/influxdb/`, including `/meta` (specifically, the clients.json file), `/data`, `/wal`, and `/hh`. + +## Optimal server counts + +When creating a cluster, you need to decide how many meta and data nodes to configure and connect. You can think of InfluxDB Enterprise as two separate clusters that communicate with each other: a cluster of meta nodes and one of data nodes. The number of meta nodes is driven by the number of meta node failures they need to be able to handle, while the number of data nodes scales based on your storage and query needs. + +The Raft consensus protocol requires a quorum to perform any operation, so there should always be an odd number of meta nodes. For almost all applications, 3 meta nodes is what you want. It gives you an odd number of meta nodes so that a quorum can be reached. And, if one meta node is lost, the cluster can still operate with the remaining 2 meta nodes until the third one is replaced. Additional meta nodes exponentially increases the communication overhead and is not recommended unless you expect the cluster to frequently lose meta nodes. + +Data nodes hold the actual time series data. The minimum number of data nodes to run is 1 and can scale up from there. **Generally, you'll want to run a number of data nodes that is evenly divisible by your replication factor.** For instance, if you have a replication factor of 2, you'll want to run 2, 4, 6, 8, 10, etc. data nodes. + +## Chronograf + +[Chronograf](/{{< latest "chronograf" >}}/introduction/getting-started/) is the user interface component of InfluxData’s TICK stack. +It makes owning the monitoring and alerting for your infrastructure easy to setup and maintain. +It talks directly to the data and meta nodes over their HTTP protocols, which are bound by default to ports `8086` for data nodes and port `8091` for meta nodes. + +## Writes in a cluster + +This section describes how writes in a cluster work. We'll work through some examples using a cluster of four data nodes: `A`, `B`, `C`, and `D`. Assume that we have a retention policy with a replication factor of 2 with shard durations of 1 day. + +### Shard groups + +The cluster creates shards within a shard group to maximize the number of data nodes utilized. If there are N data nodes in the cluster and the replication factor is X, then N/X shards are created in each shard group, discarding any fractions. + +This means that a new shard group gets created for each day of data that gets written in. Within each shard group 2 shards are created. Because of the replication factor of 2, each of those two shards are copied on 2 servers. For example we have a shard group for `2016-09-19` that has two shards `1` and `2`. Shard `1` is replicated to servers `A` and `B` while shard `2` is copied to servers `C` and `D`. + +When a write comes in with values that have a timestamp in `2016-09-19` the cluster must first determine which shard within the shard group should receive the write. This is done by taking a hash of the `measurement` + sorted `tagset` (the metaseries) and bucketing into the correct shard. In Go this looks like: + +```go +// key is measurement + tagset +// shardGroup is the group for the values based on timestamp +// hash with fnv and then bucket +shard := shardGroup.shards[fnv.New64a(key) % len(shardGroup.Shards)] +``` + +There are multiple implications to this scheme for determining where data lives in a cluster. First, for any given metaseries all data on any given day exists in a single shard, and thus only on those servers hosting a copy of that shard. Second, once a shard group is created, adding new servers to the cluster won't scale out write capacity for that shard group. The replication is fixed when the shard group is created. + +However, there is a method for expanding writes in the current shard group (i.e. today) when growing a cluster. The current shard group can be truncated to stop at the current time using `influxd-ctl truncate-shards`. This immediately closes the current shard group, forcing a new shard group to be created. That new shard group inherits the latest retention policy and data node changes and then copies itself appropriately to the newly available data nodes. Run `influxd-ctl truncate-shards help` for more information on the command. + +### Write consistency + +Each request to the HTTP API can specify the consistency level via the `consistency` query parameter. For this example let's assume that an HTTP write is being sent to server `D` and the data belongs in shard `1`. The write needs to be replicated to the owners of shard `1`: data nodes `A` and `B`. When the write comes into `D`, that node determines from its local cache of the metastore that the write needs to be replicated to the `A` and `B`, and it immediately tries to write to both. The subsequent behavior depends on the consistency level chosen: + +* `any` - return success to the client as soon as any node has responded with a write success, or the receiving node has written the data to its hinted handoff queue. In our example, if `A` or `B` return a successful write response to `D`, or if `D` has cached the write in its local hinted handoff, `D` returns a write success to the client. +* `one` - return success to the client as soon as any node has responded with a write success, but not if the write is only in hinted handoff. In our example, if `A` or `B` return a successful write response to `D`, `D` returns a write success to the client. If `D` could not send the data to either `A` or `B` but instead put the data in hinted handoff, `D` returns a write failure to the client. Note that this means writes may return a failure and yet the data may eventually persist successfully when hinted handoff drains. +* `quorum` - return success when a majority of nodes return success. This option is only useful if the replication factor is greater than 2, otherwise it is equivalent to `all`. In our example, if both `A` and `B` return a successful write response to `D`, `D` returns a write success to the client. If either `A` or `B` does not return success, then a majority of nodes have not successfully persisted the write and `D` returns a write failure to the client. If we assume for a moment the data were bound for three nodes, `A`, `B`, and `C`, then if any two of those nodes respond with a write success, `D` returns a write success to the client. If one or fewer nodes respond with a success, `D` returns a write failure to the client. Note that this means writes may return a failure and yet the data may eventually persist successfully when hinted handoff drains. +* `all` - return success only when all nodes return success. In our example, if both `A` and `B` return a successful write response to `D`, `D` returns a write success to the client. If either `A` or `B` does not return success, then `D` returns a write failure to the client. If we again assume three destination nodes `A`, `B`, and `C`, then all if three nodes respond with a write success, `D` returns a write success to the client. Otherwise, `D` returns a write failure to the client. Note that this means writes may return a failure and yet the data may eventually persist successfully when hinted handoff drains. + +The important thing to note is how failures are handled. In the case of failures, the database uses the hinted handoff system. + +### Hinted handoff + +Hinted handoff is how InfluxDB Enterprise deals with data node outages while writes are happening. Hinted handoff is essentially a durable disk based queue. When writing at `any`, `one` or `quorum` consistency, hinted handoff is used when one or more replicas return an error after a success has already been returned to the client. When writing at `all` consistency, writes cannot return success unless all nodes return success. Temporarily stalled or failed writes may still go to the hinted handoff queues but the cluster would have already returned a failure response to the write. The receiving node creates a separate queue on disk for each data node (and shard) it cannot reach. + +Let's again use the example of a write coming to `D` that should go to shard `1` on `A` and `B`. If we specified a consistency level of `one` and node `A` returns success, `D` immediately returns success to the client even though the write to `B` is still in progress. + +Now let's assume that `B` returns an error. Node `D` then puts the write into its hinted handoff queue for shard `1` on node `B`. In the background, node `D` continues to attempt to empty the hinted handoff queue by writing the data to node `B`. The configuration file has settings for the maximum size and age of data in hinted handoff queues. + +If a data node is restarted it checks for pending writes in the hinted handoff queues and resume attempts to replicate the writes. The important thing to note is that the hinted handoff queue is durable and does survive a process restart. + +When restarting nodes within an active cluster, during upgrades or maintenance, for example, other nodes in the cluster store hinted handoff writes to the offline node and replicates them when the node is again available. Thus, a healthy cluster should have enough resource headroom on each data node to handle the burst of hinted handoff writes following a node outage. The returning node needs to handle both the steady state traffic and the queued hinted handoff writes from other nodes, meaning its write traffic will have a significant spike following any outage of more than a few seconds, until the hinted handoff queue drains. + +If a node with pending hinted handoff writes for another data node receives a write destined for that node, it adds the write to the end of the hinted handoff queue rather than attempt a direct write. This ensures that data nodes receive data in mostly chronological order, as well as preventing unnecessary connection attempts while the other node is offline. + +## Queries in a cluster + +Queries in a cluster are distributed based on the time range being queried and the replication factor of the data. For example if the retention policy has a replication factor of 4, the coordinating data node receiving the query randomly picks any of the 4 data nodes that store a replica of the shard(s) to receive the query. If we assume that the system has shard durations of one day, then for each day of time covered by a query the coordinating node selects one data node to receive the query for that day. + +The coordinating node executes and fulfill the query locally whenever possible. If a query must scan multiple shard groups (multiple days in the example above), the coordinating node forwards queries to other nodes for shards it does not have locally. The queries are forwarded in parallel to scanning its own local data. The queries are distributed to as many nodes as required to query each shard group once. As the results come back from each data node, the coordinating data node combines them into the final result that gets returned to the user. diff --git a/content/enterprise_influxdb/v1.9/concepts/crosswalk.md b/content/enterprise_influxdb/v1.9/concepts/crosswalk.md new file mode 100644 index 000000000..ea05ea453 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/concepts/crosswalk.md @@ -0,0 +1,219 @@ +--- +title: Compare InfluxDB to SQL databases +description: Differences between InfluxDB and SQL databases. +menu: + enterprise_influxdb_1_9: + name: Compare InfluxDB to SQL databases + weight: 30 + parent: Concepts +--- + +InfluxDB is similar to a SQL database, but different in many ways. +InfluxDB is purpose-built for time series data. +Relational databases _can_ handle time series data, but are not optimized for common time series workloads. +InfluxDB is designed to store large volumes of time series data and quickly perform real-time analysis on that data. + +### Timing is everything + +In InfluxDB, a timestamp identifies a single point in any given data series. +This is like an SQL database table where the primary key is pre-set by the system and is always time. + +InfluxDB also recognizes that your [schema](/enterprise_influxdb/v1.9/concepts/glossary/#schema) preferences may change over time. +In InfluxDB you don't have to define schemas up front. +Data points can have one of the fields on a measurement, all of the fields on a measurement, or any number in-between. +You can add new fields to a measurement simply by writing a point for that new field. +If you need an explanation of the terms measurements, tags, and fields check out the next section for an SQL database to InfluxDB terminology crosswalk. + +## Terminology + +The table below is a (very) simple example of a table called `foodships` in an SQL database +with the unindexed column `#_foodships` and the indexed columns `park_id`, `planet`, and `time`. + +``` sql ++---------+---------+---------------------+--------------+ +| park_id | planet | time | #_foodships | ++---------+---------+---------------------+--------------+ +| 1 | Earth | 1429185600000000000 | 0 | +| 1 | Earth | 1429185601000000000 | 3 | +| 1 | Earth | 1429185602000000000 | 15 | +| 1 | Earth | 1429185603000000000 | 15 | +| 2 | Saturn | 1429185600000000000 | 5 | +| 2 | Saturn | 1429185601000000000 | 9 | +| 2 | Saturn | 1429185602000000000 | 10 | +| 2 | Saturn | 1429185603000000000 | 14 | +| 3 | Jupiter | 1429185600000000000 | 20 | +| 3 | Jupiter | 1429185601000000000 | 21 | +| 3 | Jupiter | 1429185602000000000 | 21 | +| 3 | Jupiter | 1429185603000000000 | 20 | +| 4 | Saturn | 1429185600000000000 | 5 | +| 4 | Saturn | 1429185601000000000 | 5 | +| 4 | Saturn | 1429185602000000000 | 6 | +| 4 | Saturn | 1429185603000000000 | 5 | ++---------+---------+---------------------+--------------+ +``` + +Those same data look like this in InfluxDB: + +```sql +name: foodships +tags: park_id=1, planet=Earth +time #_foodships +---- ------------ +2015-04-16T12:00:00Z 0 +2015-04-16T12:00:01Z 3 +2015-04-16T12:00:02Z 15 +2015-04-16T12:00:03Z 15 + +name: foodships +tags: park_id=2, planet=Saturn +time #_foodships +---- ------------ +2015-04-16T12:00:00Z 5 +2015-04-16T12:00:01Z 9 +2015-04-16T12:00:02Z 10 +2015-04-16T12:00:03Z 14 + +name: foodships +tags: park_id=3, planet=Jupiter +time #_foodships +---- ------------ +2015-04-16T12:00:00Z 20 +2015-04-16T12:00:01Z 21 +2015-04-16T12:00:02Z 21 +2015-04-16T12:00:03Z 20 + +name: foodships +tags: park_id=4, planet=Saturn +time #_foodships +---- ------------ +2015-04-16T12:00:00Z 5 +2015-04-16T12:00:01Z 5 +2015-04-16T12:00:02Z 6 +2015-04-16T12:00:03Z 5 +``` + +Referencing the example above, in general: + +* An InfluxDB measurement (`foodships`) is similar to an SQL database table. +* InfluxDB tags ( `park_id` and `planet`) are like indexed columns in an SQL database. +* InfluxDB fields (`#_foodships`) are like unindexed columns in an SQL database. +* InfluxDB points (for example, `2015-04-16T12:00:00Z 5`) are similar to SQL rows. + +Building on this comparison of database terminology, +InfluxDB [continuous queries](/enterprise_influxdb/v1.9/concepts/glossary/#continuous-query-cq) +and [retention policies](/enterprise_influxdb/v1.9/concepts/glossary/#retention-policy-rp) are +similar to stored procedures in an SQL database. +They're specified once and then performed regularly and automatically. + +Of course, there are some major disparities between SQL databases and InfluxDB. +SQL `JOIN`s aren't available for InfluxDB measurements; your schema design should reflect that difference. +And, as we mentioned above, a measurement is like an SQL table where the primary index is always pre-set to time. +InfluxDB timestamps must be in UNIX epoch (GMT) or formatted as a date-time string valid under RFC3339. + +For more detailed descriptions of the InfluxDB terms mentioned in this section see our [Glossary of Terms](/enterprise_influxdb/v1.9/concepts/glossary/). + +## Query languages +InfluxDB supports multiple query languages: + +- [Flux](#flux) +- [InfluxQL](#influxql) + +### Flux + +[Flux](/enterprise_influxdb/v1.9/flux/) is a data scripting language designed for querying, analyzing, and acting on time series data. +Beginning with **InfluxDB 1.8.0**, Flux is available for production use along side InfluxQL. + +For those familiar with [InfluxQL](#influxql), Flux is intended to address +many of the outstanding feature requests that we've received since introducing InfluxDB 1.0. +For a comparison between Flux and InfluxQL, see [Flux vs InfluxQL](/enterprise_influxdb/v1.9/flux/flux-vs-influxql/). + +Flux is the primary language for working with data in [InfluxDB OSS 2.0](/influxdb/v2.0/get-started) +and [InfluxDB Cloud](/influxdb/cloud/get-started/), +a generally available Platform as a Service (PaaS) available across multiple Cloud Service Providers. +Using Flux with InfluxDB 1.8+ lets you get familiar with Flux concepts and syntax +and ease the transition to InfluxDB 2.0. + +### InfluxQL + +InfluxQL is an SQL-like query language for interacting with InfluxDB. +It has been crafted to feel familiar to those coming from other +SQL or SQL-like environments while also providing features specific +to storing and analyzing time series data. +However **InfluxQL is not SQL** and lacks support for more advanced operations +like `UNION`, `JOIN` and `HAVING` that SQL power-users are accustomed to. +This functionality is available with [Flux](/flux/latest/introduction). + +InfluxQL's `SELECT` statement follows the form of an SQL `SELECT` statement: + +```sql +SELECT FROM WHERE +``` + +where `WHERE` is optional. + +To get the InfluxDB output in the section above, you'd enter: + +```sql +SELECT * FROM "foodships" +``` + +If you only wanted to see data for the planet `Saturn`, you'd enter: + +```sql +SELECT * FROM "foodships" WHERE "planet" = 'Saturn' +``` + +If you wanted to see data for the planet `Saturn` after 12:00:01 UTC on April 16, 2015, you'd enter: + +```sql +SELECT * FROM "foodships" WHERE "planet" = 'Saturn' AND time > '2015-04-16 12:00:01' +``` + +As shown in the example above, InfluxQL allows you to specify the time range of your query in the `WHERE` clause. +You can use date-time strings wrapped in single quotes that have the +format `YYYY-MM-DD HH:MM:SS.mmm` +(`mmm` is milliseconds and is optional, and you can also specify microseconds or nanoseconds). +You can also use relative time with `now()` which refers to the server's current timestamp: + +```sql +SELECT * FROM "foodships" WHERE time > now() - 1h +``` + +That query outputs the data in the `foodships` measure where the timestamp is newer than the server's current time minus one hour. +The options for specifying time durations with `now()` are: + +|Letter|Meaning| +|:---:|:---:| +| ns | nanoseconds | +|u or µ|microseconds| +| ms | milliseconds | +|s | seconds | +| m | minutes | +| h | hours | +| d | days | +| w | weeks | + +InfluxQL also supports regular expressions, arithmetic in expressions, `SHOW` statements, and `GROUP BY` statements. +See our [data exploration](/enterprise_influxdb/v1.9/query_language/explore-data/) page for an in-depth discussion of those topics. +InfluxQL functions include `COUNT`, `MIN`, `MAX`, `MEDIAN`, `DERIVATIVE` and more. +For a full list check out the [functions](/enterprise_influxdb/v1.9/query_language/functions/) page. + +Now that you have the general idea, check out our [Getting Started Guide](/enterprise_influxdb/v1.9/introduction/getting-started/). + +## InfluxDB is not CRUD + +InfluxDB is a database that has been optimized for time series data. +This data commonly comes from sources like distributed sensor groups, click data from large websites, or lists of financial transactions. + +One thing this data has in common is that it is more useful in the aggregate. +One reading saying that your computer’s CPU is at 12% utilization at 12:38:35 UTC on a Tuesday is hard to draw conclusions from. +It becomes more useful when combined with the rest of the series and visualized. +This is where trends over time begin to show, and actionable insight can be drawn from the data. +In addition, time series data is generally written once and rarely updated. + +The result is that InfluxDB is not a full CRUD database but more like a CR-ud, prioritizing the performance of creating and reading data over update and destroy, and [preventing some update and destroy behaviors](/enterprise_influxdb/v1.9/concepts/insights_tradeoffs/) to make create and read more performant: + +* To update a point, insert one with [the same measurement, tag set, and timestamp](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#how-does-influxdb-handle-duplicate-points). +* You can [drop or delete a series](/enterprise_influxdb/v1.9/query_language/manage-database/#drop-series-from-the-index-with-drop-series), but not individual points based on field values. As a workaround, you can search for the field value, retrieve the time, then [DELETE based on the `time` field](/enterprise_influxdb/v1.9/query_language/manage-database/#delete-series-with-delete). +* You can't update or rename tags yet - see GitHub issue [#4157](https://github.com/influxdata/influxdb/issues/4157) for more information. To modify the tag of a series of points, find the points with the offending tag value, change the value to the desired one, write the points back, then drop the series with the old tag value. +* You can't delete tags by tag key (as opposed to value) - see GitHub issue [#8604](https://github.com/influxdata/influxdb/issues/8604). diff --git a/content/enterprise_influxdb/v1.9/concepts/glossary.md b/content/enterprise_influxdb/v1.9/concepts/glossary.md new file mode 100644 index 000000000..48ee1de0a --- /dev/null +++ b/content/enterprise_influxdb/v1.9/concepts/glossary.md @@ -0,0 +1,457 @@ +--- +title: Glossary +description: Terms related to InfluxDB Enterprise. +aliases: + - /enterprise/v1.8/concepts/glossary/ +menu: + enterprise_influxdb_1_9: + weight: 20 + parent: Concepts +--- + +## data node + +A node that runs the data service. + +For high availability, installations must have at least two data nodes. +The number of data nodes in your cluster must be the same as your highest +replication factor. +Any replication factor greater than two gives you additional fault tolerance and +query capacity within the cluster. + +Data node sizes will depend on your needs. +The Amazon EC2 m4.large or m4.xlarge are good starting points. + +Related entries: [data service](#data-service), [replication factor](#replication-factor) + +## data service + +Stores all time series data and handles all writes and queries. + +Related entries: [data node](#data-node) + +## meta node + +A node that runs the meta service. + +For high availability, installations must have three meta nodes. +Meta nodes can be very modestly sized instances like an EC2 t2.micro or even a +nano. +For additional fault tolerance installations may use five meta nodes; the +number of meta nodes must be an odd number. + +Related entries: [meta service](#meta-service) + +## meta service + +The consistent data store that keeps state about the cluster, including which +servers, databases, users, continuous queries, retention policies, subscriptions, +and blocks of time exist. + +Related entries: [meta node](#meta-node) + +## replication factor + +The attribute of the retention policy that determines how many copies of the +data are stored in the cluster. +InfluxDB replicates data across `N` data nodes, where `N` is the replication +factor. + +To maintain data availability for queries, the replication factor should be less +than or equal to the number of data nodes in the cluster: + +* Data is fully available when the replication factor is greater than the +number of unavailable data nodes. +* Data may be unavailable when the replication factor is less than the number of +unavailable data nodes. + +Any replication factor greater than two gives you additional fault tolerance and +query capacity within the cluster. + +## web console + +Legacy user interface for the InfluxDB Enterprise. + +This has been deprecated and the suggestion is to use [Chronograf](/{{< latest "chronograf" >}}/introduction/). + +If you are transitioning from the Enterprise Web Console to Chronograf, see how to [transition from the InfluxDB Web Admin Interface](/chronograf/v1.7/guides/transition-web-admin-interface/). + + + +## aggregation + +An InfluxQL function that returns an aggregated value across a set of points. +For a complete list of the available and upcoming aggregations, see [InfluxQL functions](/enterprise_influxdb/v1.9/query_language/functions/#aggregations). + +Related entries: [function](#function), [selector](#selector), [transformation](#transformation) + +## batch + +A collection of data points in InfluxDB line protocol format, separated by newlines (`0x0A`). +A batch of points may be submitted to the database using a single HTTP request to the write endpoint. +This makes writes using the InfluxDB API much more performant by drastically reducing the HTTP overhead. +InfluxData recommends batch sizes of 5,000-10,000 points, although different use cases may be better served by significantly smaller or larger batches. + +Related entries: [InfluxDB line protocol](#influxdb-line-protocol), [point](#point) + +## bucket + +A bucket is a named location where time series data is stored in **InfluxDB 2.0**. In InfluxDB 1.8+, each combination of a database and a retention policy (database/retention-policy) represents a bucket. Use the [InfluxDB 2.0 API compatibility endpoints](/enterprise_influxdb/v1.9/tools/api#influxdb-2-0-api-compatibility-endpoints) included with InfluxDB 1.8+ to interact with buckets. + +## continuous query (CQ) + +An InfluxQL query that runs automatically and periodically within a database. +Continuous queries require a function in the `SELECT` clause and must include a `GROUP BY time()` clause. +See [Continuous Queries](/enterprise_influxdb/v1.9/query_language/continuous_queries/). + + +Related entries: [function](#function) + +## database + +A logical container for users, retention policies, continuous queries, and time series data. + +Related entries: [continuous query](#continuous-query-cq), [retention policy](#retention-policy-rp), [user](#user) + +## duration + +The attribute of the retention policy that determines how long InfluxDB stores data. +Data older than the duration are automatically dropped from the database. +See [Database Management](/enterprise_influxdb/v1.9/query_language/manage-database/#create-retention-policies-with-create-retention-policy) for how to set duration. + +Related entries: [retention policy](#retention-policy-rp) + +## field + +The key-value pair in an InfluxDB data structure that records metadata and the actual data value. +Fields are required in InfluxDB data structures and they are not indexed - queries on field values scan all points that match the specified time range and, as a result, are not performant relative to tags. + +*Query tip:* Compare fields to tags; tags are indexed. + +Related entries: [field key](#field-key), [field set](#field-set), [field value](#field-value), [tag](#tag) + +## field key + +The key part of the key-value pair that makes up a field. +Field keys are strings and they store metadata. + +Related entries: [field](#field), [field set](#field-set), [field value](#field-value), [tag key](#tag-key) + +## field set + +The collection of field keys and field values on a point. + +Related entries: [field](#field), [field key](#field-key), [field value](#field-value), [point](#point) + +## field value + +The value part of the key-value pair that makes up a field. +Field values are the actual data; they can be strings, floats, integers, or booleans. +A field value is always associated with a timestamp. + +Field values are not indexed - queries on field values scan all points that match the specified time range and, as a result, are not performant. + +*Query tip:* Compare field values to tag values; tag values are indexed. + +Related entries: [field](#field), [field key](#field-key), [field set](#field-set), [tag value](#tag-value), [timestamp](#timestamp) + +## function + +InfluxQL aggregations, selectors, and transformations. +See [InfluxQL Functions](/enterprise_influxdb/v1.9/query_language/functions/) for a complete list of InfluxQL functions. + +Related entries: [aggregation](#aggregation), [selector](#selector), [transformation](#transformation) + +## identifier + +Tokens that refer to continuous query names, database names, field keys, +measurement names, retention policy names, subscription names, tag keys, and +user names. +See [Query Language Specification](/enterprise_influxdb/v1.9/query_language/spec/#identifiers). + +Related entries: +[database](#database), +[field key](#field-key), +[measurement](#measurement), +[retention policy](#retention-policy-rp), +[tag key](#tag-key), +[user](#user) + +## InfluxDB line protocol + +The text based format for writing points to InfluxDB. See [InfluxDB line protocol](/enterprise_influxdb/v1.9/write_protocols/). + +## measurement + +The part of the InfluxDB data structure that describes the data stored in the associated fields. +Measurements are strings. + +Related entries: [field](#field), [series](#series) + +## metastore + +Contains internal information about the status of the system. +The metastore contains the user information, databases, retention policies, shard metadata, continuous queries, and subscriptions. + +Related entries: [database](#database), [retention policy](#retention-policy-rp), [user](#user) + +## node + +An independent `influxd` process. + +Related entries: [server](#server) + +## now() + +The local server's nanosecond timestamp. + +## point + +In InfluxDB, a point represents a single data record, similar to a row in a SQL database table. Each point: + +- has a measurement, a tag set, a field key, a field value, and a timestamp; +- is uniquely identified by its series and timestamp. + +You cannot store more than one point with the same timestamp in a series. +If you write a point to a series with a timestamp that matches an existing point, the field set becomes a union of the old and new field set, and any ties go to the new field set. +For more information about duplicate points, see [How does InfluxDB handle duplicate points?](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#how-does-influxdb-handle-duplicate-points) + +Related entries: [field set](#field-set), [series](#series), [timestamp](#timestamp) + +## points per second + +A deprecated measurement of the rate at which data are persisted to InfluxDB. +The schema allows and even encourages the recording of multiple metric values per point, rendering points per second ambiguous. + +Write speeds are generally quoted in values per second, a more precise metric. + +Related entries: [point](#point), [schema](#schema), [values per second](#values-per-second) + +## query + +An operation that retrieves data from InfluxDB. +See [Data Exploration](/enterprise_influxdb/v1.9/query_language/explore-data/), [Schema Exploration](/enterprise_influxdb/v1.9/query_language/explore-schema/), [Database Management](/enterprise_influxdb/v1.9/query_language/manage-database/). + +## replication factor + +The attribute of the retention policy that determines how many copies of data to concurrently store (or retain) in the cluster. Replicating copies ensures that data is available when a data node (or more) is unavailable. + +For three nodes or less, the default replication factor equals the number of data nodes. +For more than three nodes, the default replication factor is 3. To change the default replication factor, specify the replication factor `n` in the retention policy. + +Related entries: [duration](#duration), [node](#node), +[retention policy](#retention-policy-rp) + +## retention policy (RP) + +Describes how long InfluxDB keeps data (duration), how many copies of the data to store in the cluster (replication factor), and the time range covered by shard groups (shard group duration). RPs are unique per database and along with the measurement and tag set define a series. + +When you create a database, InfluxDB creates a retention policy called `autogen` with an infinite duration, a replication factor set to one, and a shard group duration set to seven days. +For more information, see [Retention policy management](/enterprise_influxdb/v1.9/query_language/manage-database/#retention-policy-management). + +Related entries: [duration](#duration), [measurement](#measurement), [replication factor](#replication-factor), [series](#series), [shard duration](#shard-duration), [tag set](#tag-set) + +## schema + +How the data are organized in InfluxDB. +The fundamentals of the InfluxDB schema are databases, retention policies, series, measurements, tag keys, tag values, and field keys. +See [Schema Design](/enterprise_influxdb/v1.9/concepts/schema_and_data_layout/) for more information. + +Related entries: [database](#database), [field key](#field-key), [measurement](#measurement), [retention policy](#retention-policy-rp), [series](#series), [tag key](#tag-key), [tag value](#tag-value) + +## selector + +An InfluxQL function that returns a single point from the range of specified points. +See [InfluxQL Functions](/enterprise_influxdb/v1.9/query_language/functions/#selectors) for a complete list of the available and upcoming selectors. + +Related entries: [aggregation](#aggregation), [function](#function), [transformation](#transformation) + +## series + +A logical grouping of data defined by shared measurement, tag set, and field key. + +Related entries: [field set](#field-set), [measurement](#measurement), [tag set](#tag-set) + +## series cardinality + +The number of unique database, measurement, tag set, and field key combinations in an InfluxDB instance. + +For example, assume that an InfluxDB instance has a single database and one measurement. +The single measurement has two tag keys: `email` and `status`. +If there are three different `email`s, and each email address is associated with two +different `status`es then the series cardinality for the measurement is 6 +(3 * 2 = 6): + +| email | status | +| :-------------------- | :----- | +| lorr@influxdata.com | start | +| lorr@influxdata.com | finish | +| marv@influxdata.com | start | +| marv@influxdata.com | finish | +| cliff@influxdata.com | start | +| cliff@influxdata.com | finish | + +Note that, in some cases, simply performing that multiplication may overestimate series cardinality because of the presence of dependent tags. +Dependent tags are tags that are scoped by another tag and do not increase series +cardinality. +If we add the tag `firstname` to the example above, the series cardinality +would not be 18 (3 * 2 * 3 = 18). +It would remain unchanged at 6, as `firstname` is already scoped by the `email` tag: + +| email | status | firstname | +| :-------------------- | :----- | :-------- | +| lorr@influxdata.com | start | lorraine | +| lorr@influxdata.com | finish | lorraine | +| marv@influxdata.com | start | marvin | +| marv@influxdata.com | finish | marvin | +| cliff@influxdata.com | start | clifford | +| cliff@influxdata.com | finish | clifford | + +See [SHOW CARDINALITY](/enterprise_influxdb/v1.9/query_language/spec/#show-cardinality) to learn about the InfluxQL commands for series cardinality. + +Related entries: [field key](#field-key),[measurement](#measurement), [tag key](#tag-key), [tag set](#tag-set) + +## series key + +A series key identifies a particular series by measurement, tag set, and field key. + +For example: + +``` +# measurement, tag set, field key +h2o_level, location=santa_monica, h2o_feet +``` + +Related entries: [series](#series) + +## server + +A machine, virtual or physical, that is running InfluxDB. +There should only be one InfluxDB process per server. + +Related entries: [node](#node) + +## shard + +A shard contains the actual encoded and compressed data, and is represented by a TSM file on disk. +Every shard belongs to one and only one shard group. +Multiple shards may exist in a single shard group. +Each shard contains a specific set of series. +All points falling on a given series in a given shard group will be stored in the same shard (TSM file) on disk. + +Related entries: [series](#series), [shard duration](#shard-duration), [shard group](#shard-group), [tsm](#tsm-time-structured-merge-tree) + +## shard duration + +The shard duration determines how much time each shard group spans. +The specific interval is determined by the `SHARD DURATION` of the retention policy. +See [Retention Policy management](/enterprise_influxdb/v1.9/query_language/manage-database/#retention-policy-management) for more information. + +For example, given a retention policy with `SHARD DURATION` set to `1w`, each shard group will span a single week and contain all points with timestamps in that week. + +Related entries: [database](#database), [retention policy](#retention-policy-rp), [series](#series), [shard](#shard), [shard group](#shard-group) + +## shard group + +Shard groups are logical containers for shards. +Shard groups are organized by time and retention policy. +Every retention policy that contains data has at least one associated shard group. +A given shard group contains all shards with data for the interval covered by the shard group. +The interval spanned by each shard group is the shard duration. + +Related entries: [database](#database), [retention policy](#retention-policy-rp), [series](#series), [shard](#shard), [shard duration](#shard-duration) + +## subscription + +Subscriptions allow [Kapacitor](/{{< latest "kapacitor" >}}/) to receive data from InfluxDB in a push model rather than the pull model based on querying data. +When Kapacitor is configured to work with InfluxDB, the subscription will automatically push every write for the subscribed database from InfluxDB to Kapacitor. +Subscriptions can use TCP or UDP for transmitting the writes. + +## tag + +The key-value pair in the InfluxDB data structure that records metadata. +Tags are an optional part of the data structure, but they are useful for storing commonly-queried metadata; tags are indexed so queries on tags are performant. +*Query tip:* Compare tags to fields; fields are not indexed. + +Related entries: [field](#field), [tag key](#tag-key), [tag set](#tag-set), [tag value](#tag-value) + +## tag key + +The key part of the key-value pair that makes up a tag. +Tag keys are strings and they store metadata. +Tag keys are indexed so queries on tag keys are performant. + +*Query tip:* Compare tag keys to field keys; field keys are not indexed. + +Related entries: [field key](#field-key), [tag](#tag), [tag set](#tag-set), [tag value](#tag-value) + +## tag set + +The collection of tag keys and tag values on a point. + +Related entries: [point](#point), [series](#series), [tag](#tag), [tag key](#tag-key), [tag value](#tag-value) + +## tag value + +The value part of the key-value pair that makes up a tag. +Tag values are strings and they store metadata. +Tag values are indexed so queries on tag values are performant. + + +Related entries: [tag](#tag), [tag key](#tag-key), [tag set](#tag-set) + +## timestamp + +The date and time associated with a point. +All time in InfluxDB is UTC. + +For how to specify time when writing data, see [Write Syntax](/enterprise_influxdb/v1.9/write_protocols/write_syntax/). +For how to specify time when querying data, see [Data Exploration](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax). + +Related entries: [point](#point) + +## transformation + +An InfluxQL function that returns a value or a set of values calculated from specified points, but does not return an aggregated value across those points. +See [InfluxQL Functions](/enterprise_influxdb/v1.9/query_language/functions/#transformations) for a complete list of the available and upcoming aggregations. + +Related entries: [aggregation](#aggregation), [function](#function), [selector](#selector) + +## TSM (Time Structured Merge tree) + +The purpose-built data storage format for InfluxDB. TSM allows for greater compaction and higher write and read throughput than existing B+ or LSM tree implementations. See [Storage Engine](/enterprise_influxdb/v1.9/concepts/storage_engine/) for more. + +## user + +There are two kinds of users in InfluxDB: + +* *Admin users* have `READ` and `WRITE` access to all databases and full access to administrative queries and user management commands. +* *Non-admin users* have `READ`, `WRITE`, or `ALL` (both `READ` and `WRITE`) access per database. + +When authentication is enabled, InfluxDB only executes HTTP requests that are sent with a valid username and password. +See [Authentication and Authorization](/enterprise_influxdb/v1.9/administration/authentication_and_authorization/). + +## values per second + +The preferred measurement of the rate at which data are persisted to InfluxDB. Write speeds are generally quoted in values per second. + +To calculate the values per second rate, multiply the number of points written per second by the number of values stored per point. For example, if the points have four fields each, and a batch of 5000 points is written 10 times per second, then the values per second rate is `4 field values per point * 5000 points per batch * 10 batches per second = 200,000 values per second`. + +Related entries: [batch](#batch), [field](#field), [point](#point), [points per second](#points-per-second) + +## WAL (Write Ahead Log) + +The temporary cache for recently written points. To reduce the frequency with which the permanent storage files are accessed, InfluxDB caches new points in the WAL until their total size or age triggers a flush to more permanent storage. This allows for efficient batching of the writes into the TSM. + +Points in the WAL can be queried, and they persist through a system reboot. On process start, all points in the WAL must be flushed before the system accepts new writes. + +Related entries: [tsm](#tsm-time-structured-merge-tree) + + diff --git a/content/enterprise_influxdb/v1.9/concepts/influxdb-enterprise-startup.md b/content/enterprise_influxdb/v1.9/concepts/influxdb-enterprise-startup.md new file mode 100644 index 000000000..f63d93623 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/concepts/influxdb-enterprise-startup.md @@ -0,0 +1,85 @@ +--- +title: InfluxDB Enterprise startup process +description: > + On startup, InfluxDB Enterprise starts all subsystems and services in a deterministic order. +menu: + enterprise_influxdb_1_9: + weight: 10 + name: Startup process + parent: Concepts +--- + +On startup, InfluxDB Enterprise starts all subsystems and services in the following order: + +1. [TSDBStore](#tsdbstore) +2. [Monitor](#monitor) +3. [Cluster](#cluster) +4. [Precreator](#precreator) +5. [Snapshotter](#snapshotter) +6. [Continuous Query](#continuous-query) +7. [Announcer](#announcer) +8. [Retention](#retention) +9. [Stats](#stats) +10. [Anti-entropy](#anti-entropy) +11. [HTTP API](#http-api) + +A **subsystem** is a collection of related services managed together as part of a greater whole. +A **service** is a process that provides specific functionality. + +## Subsystems and services + +### TSDBStore +The TSDBStore subsystem starts and manages the TSM storage engine. +This includes services such as the points writer (write), reads (query), +and [hinted handoff (HH)](/enterprise_influxdb/v1.9/concepts/clustering/#hinted-handoff). +TSDBSTore first opens all the shards and loads write-ahead log (WAL) data into the in-memory write cache. +If `influxd` was cleanly shutdown previously, there will not be any WAL data. +It then loads a portion of each shard's index. + +{{% note %}} +#### Index versions and startup times +If using `inmem` indexing, InfluxDB loads all shard indexes into memory, which, +depending on the number of series in the database, can take time. +If using `tsi1` indexing, InfluxDB only loads hot shard indexes +(the most recent shards or shards currently being written to) into memory and +stores cold shard indexes on disk. +Use `tsi1` indexing to see shorter startup times. +{{% /note %}} + +### Monitor +The Monitor service provides statistical and diagnostic information to InfluxDB about InfluxDB itself. +This information helps with database troubleshooting and performance analysis. + +### Cluster +The Cluster service provides implementations of InfluxDB OSS v1.8 interfaces +that operate on an InfluxDB Enterprise v1.8 cluster. + +### Precreator +The Precreator service creates shards before they are needed. +This ensures necessary shards exist before new time series data arrives and that +write-throughput is not affected the creation of a new shard. + +### Snapshotter +The Snapshotter service routinely creates snapshots of InfluxDB Enterprise metadata. + +### Continuous Query +The Continuous Query (CQ) subsystem manages all InfluxDB CQs. + +### Announcer +The Announcer service announces a data node's status to meta nodes. + +### Retention +The Retention service enforces [retention policies](/enterprise_influxdb/v1.9/concepts/glossary/#retention-policy-rp) +and drops data as it expires. + +### Stats +The Stats service monitors cluster-level statistics. + +### Anti-entropy +The Anti-entropy (AE) subsystem is responsible for reconciling differences between shards. +For more information, see [Use anti-entropy](/enterprise_influxdb/v1.9/administration/anti-entropy/). + +### HTTP API +The InfluxDB HTTP API service provides a public facing interface to interact with +InfluxDB Enterprise and internal interfaces used within the InfluxDB Enterprise cluster. + diff --git a/content/enterprise_influxdb/v1.9/concepts/insights_tradeoffs.md b/content/enterprise_influxdb/v1.9/concepts/insights_tradeoffs.md new file mode 100644 index 000000000..857a0b3f9 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/concepts/insights_tradeoffs.md @@ -0,0 +1,60 @@ +--- +title: InfluxDB design insights and tradeoffs +description: > + Optimizing for time series use case entails some tradeoffs, primarily to increase performance at the cost of functionality. +menu: + enterprise_influxdb_1_9: + name: InfluxDB design insights and tradeoffs + weight: 40 + parent: Concepts +v2: /influxdb/v2.0/reference/key-concepts/design-principles/ +--- + +InfluxDB is a time series database. +Optimizing for this use case entails some tradeoffs, primarily to increase performance at the cost of functionality. +Below is a list of some of those design insights that lead to tradeoffs: + +1. For the time series use case, we assume that if the same data is sent multiple times, it is the exact same data that a client just sent several times. + + _**Pro:**_ Simplified [conflict resolution](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#how-does-influxdb-handle-duplicate-points) increases write performance. + _**Con:**_ Cannot store duplicate data; may overwrite data in rare circumstances. + +2. Deletes are a rare occurrence. + When they do occur it is almost always against large ranges of old data that are cold for writes. + + _**Pro:**_ Restricting access to deletes allows for increased query and write performance. + _**Con:**_ Delete functionality is significantly restricted. + +3. Updates to existing data are a rare occurrence and contentious updates never happen. + Time series data is predominantly new data that is never updated. + + _**Pro:**_ Restricting access to updates allows for increased query and write performance. + _**Con:**_ Update functionality is significantly restricted. + +4. The vast majority of writes are for data with very recent timestamps and the data is added in time ascending order. + + _**Pro:**_ Adding data in time ascending order is significantly more performant. + _**Con:**_ Writing points with random times or with time not in ascending order is significantly less performant. + +5. Scale is critical. + The database must be able to handle a *high* volume of reads and writes. + + _**Pro:**_ The database can handle a *high* volume of reads and writes. + _**Con:**_ The InfluxDB development team was forced to make tradeoffs to increase performance. + +6. Being able to write and query the data is more important than having a strongly consistent view. + + _**Pro:**_ Writing and querying the database can be done by multiple clients and at high loads. + _**Con:**_ Query returns may not include the most recent points if database is under heavy load. + +7. Many time [series](/enterprise_influxdb/v1.9/concepts/glossary/#series) are ephemeral. + There are often time series that appear only for a few hours and then go away, e.g. + a new host that gets started and reports for a while and then gets shut down. + + _**Pro:**_ InfluxDB is good at managing discontinuous data. + _**Con:**_ Schema-less design means that some database functions are not supported e.g. there are no cross table joins. + +8. No one point is too important. + + _**Pro:**_ InfluxDB has very powerful tools to deal with aggregate data and large data sets. + _**Con:**_ Points don't have IDs in the traditional sense, they are differentiated by timestamp and series. diff --git a/content/enterprise_influxdb/v1.9/concepts/key_concepts.md b/content/enterprise_influxdb/v1.9/concepts/key_concepts.md new file mode 100644 index 000000000..caa494d93 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/concepts/key_concepts.md @@ -0,0 +1,202 @@ +--- +title: InfluxDB key concepts +description: Covers key concepts to learn about InfluxDB. +menu: + enterprise_influxdb_1_9: + name: Key concepts + weight: 10 + parent: Concepts +v2: /influxdb/v2.0/reference/key-concepts/ +--- + +Before diving into InfluxDB, it's good to get acquainted with some key concepts of the database. This document introduces key InfluxDB concepts and elements. To introduce the key concepts, we’ll cover how the following elements work together in InfluxDB: + +- [database](/enterprise_influxdb/v1.9/concepts/glossary/#database) +- [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +- [field set](/enterprise_influxdb/v1.9/concepts/glossary/#field-set) +- [field value](/enterprise_influxdb/v1.9/concepts/glossary/#field-value) +- [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) +- [point](/enterprise_influxdb/v1.9/concepts/glossary/#point) +- [retention policy](/enterprise_influxdb/v1.9/concepts/glossary/#retention-policy-rp) +- [series](/enterprise_influxdb/v1.9/concepts/glossary/#series) +- [tag key](/enterprise_influxdb/v1.9/concepts/glossary/#tag-key) +- [tag set](/enterprise_influxdb/v1.9/concepts/glossary/#tag-set) +- [tag value](/enterprise_influxdb/v1.9/concepts/glossary/#tag-value) +- [timestamp](/enterprise_influxdb/v1.9/concepts/glossary/#timestamp) + + +### Sample data + +The next section references the data printed out below. +The data is fictional, but represents a believable setup in InfluxDB. +They show the number of butterflies and honeybees counted by two scientists (`langstroth` and `perpetua`) in two locations (location `1` and location `2`) over the time period from August 18, 2015 at midnight through August 18, 2015 at 6:12 AM. +Assume that the data lives in a database called `my_database` and are subject to the `autogen` retention policy (more on databases and retention policies to come). + +*Hint:* Hover over the links for tooltips to get acquainted with InfluxDB terminology and the layout. + +**name:** census + +| time | butterflies | honeybees | location | scientist | +| ---- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------- | ------------------------------------------------------------------- | -------------------------------------------------------------------- | +| 2015-08-18T00:00:00Z | 12 | 23 | 1 | langstroth | +| 2015-08-18T00:00:00Z | 1 | 30 | 1 | perpetua | +| 2015-08-18T00:06:00Z | 11 | 28 | 1 | langstroth | +| 2015-08-18T00:06:00Z | 3 | 28 | 1 | perpetua | +| 2015-08-18T05:54:00Z | 2 | 11 | 2 | langstroth | +| 2015-08-18T06:00:00Z | 1 | 10 | 2 | langstroth | +| 2015-08-18T06:06:00Z | 8 | 23 | 2 | perpetua | +| 2015-08-18T06:12:00Z | 7 | 22 | 2 | perpetua | + +### Discussion + +Now that you've seen some sample data in InfluxDB this section covers what it all means. + +InfluxDB is a time series database so it makes sense to start with what is at the root of everything we do: time. +In the data above there's a column called `time` - all data in InfluxDB have that column. +`time` stores timestamps, and the _**timestamp**_ shows the date and time, in [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) UTC, associated with particular data. + +The next two columns, called `butterflies` and `honeybees`, are fields. +Fields are made up of field keys and field values. +_**Field keys**_ (`butterflies` and `honeybees`) are strings; the field key `butterflies` tells us that the field values `12`-`7` refer to butterflies and the field key `honeybees` tells us that the field values `23`-`22` refer to, well, honeybees. + +_**Field values**_ are your data; they can be strings, floats, integers, or Booleans, and, because InfluxDB is a time series database, a field value is always associated with a timestamp. +The field values in the sample data are: + +``` +12 23 +1 30 +11 28 +3 28 +2 11 +1 10 +8 23 +7 22 +``` + +In the data above, the collection of field-key and field-value pairs make up a _**field set**_. +Here are all eight field sets in the sample data: + +* `butterflies = 12 honeybees = 23` +* `butterflies = 1 honeybees = 30` +* `butterflies = 11 honeybees = 28` +* `butterflies = 3 honeybees = 28` +* `butterflies = 2 honeybees = 11` +* `butterflies = 1 honeybees = 10` +* `butterflies = 8 honeybees = 23` +* `butterflies = 7 honeybees = 22` + +Fields are a required piece of the InfluxDB data structure - you cannot have data in InfluxDB without fields. +It's also important to note that fields are not indexed. +[Queries](/enterprise_influxdb/v1.9/concepts/glossary/#query) that use field values as filters must scan all values that match the other conditions in the query. +As a result, those queries are not performant relative to queries on tags (more on tags below). +In general, fields should not contain commonly-queried metadata. + +The last two columns in the sample data, called `location` and `scientist`, are tags. +Tags are made up of tag keys and tag values. +Both _**tag keys**_ and _**tag values**_ are stored as strings and record metadata. +The tag keys in the sample data are `location` and `scientist`. +The tag key `location` has two tag values: `1` and `2`. +The tag key `scientist` also has two tag values: `langstroth` and `perpetua`. + +In the data above, the _**tag set**_ is the different combinations of all the tag key-value pairs. +The four tag sets in the sample data are: + +* `location = 1`, `scientist = langstroth` +* `location = 2`, `scientist = langstroth` +* `location = 1`, `scientist = perpetua` +* `location = 2`, `scientist = perpetua` + +Tags are optional. +You don't need to have tags in your data structure, but it's generally a good idea to make use of them because, unlike fields, tags are indexed. +This means that queries on tags are faster and that tags are ideal for storing commonly-queried metadata. + +Avoid using the following reserved keys: + +* `_field` +* `_measurement` +* `time` + +If reserved keys are included as a tag or field key, the associated point is discarded. + +> **Why indexing matters: The schema case study** + +> Say you notice that most of your queries focus on the values of the field keys `honeybees` and `butterflies`: + +> `SELECT * FROM "census" WHERE "butterflies" = 1` +> `SELECT * FROM "census" WHERE "honeybees" = 23` + +> Because fields aren't indexed, InfluxDB scans every value of `butterflies` in the first query and every value of `honeybees` in the second query before it provides a response. +That behavior can hurt query response times - especially on a much larger scale. +To optimize your queries, it may be beneficial to rearrange your [schema](/enterprise_influxdb/v1.9/concepts/glossary/#schema) such that the fields (`butterflies` and `honeybees`) become the tags and the tags (`location` and `scientist`) become the fields: + +> **name:** census +> +| time | location | scientist | butterflies | honeybees | +| ---- | --------------------------------------------------------------------- | ---------------------------------------------------------------------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------- | +| 2015-08-18T00:00:00Z | 1 | langstroth | 12 | 23 | +| 2015-08-18T00:00:00Z | 1 | perpetua | 1 | 30 | +| 2015-08-18T00:06:00Z | 1 | langstroth | 11 | 28 | +| 2015-08-18T00:06:00Z | 1 | perpetua | 3 | 28 | +| 2015-08-18T05:54:00Z | 2 | langstroth | 2 | 11 | +| 2015-08-18T06:00:00Z | 2 | langstroth | 1 | 10 | +| 2015-08-18T06:06:00Z | 2 | perpetua | 8 | 23 | +| 2015-08-18T06:12:00Z | 2 | perpetua | 7 | 22 | + +> Now that `butterflies` and `honeybees` are tags, InfluxDB won't have to scan every one of their values when it performs the queries above - this means that your queries are even faster. + +The _**measurement**_ acts as a container for tags, fields, and the `time` column, and the measurement name is the description of the data that are stored in the associated fields. +Measurement names are strings, and, for any SQL users out there, a measurement is conceptually similar to a table. +The only measurement in the sample data is `census`. +The name `census` tells us that the field values record the number of `butterflies` and `honeybees` - not their size, direction, or some sort of happiness index. + +A single measurement can belong to different retention policies. +A _**retention policy**_ describes how long InfluxDB keeps data (`DURATION`) and how many copies of this data is stored in the cluster (`REPLICATION`). +If you're interested in reading more about retention policies, check out [Database Management](/enterprise_influxdb/v1.9/query_language/manage-database/#retention-policy-management). + +{{% warn %}} Replication factors do not serve a purpose with single node instances. +{{% /warn %}} + +In the sample data, everything in the `census` measurement belongs to the `autogen` retention policy. +InfluxDB automatically creates that retention policy; it has an infinite duration and a replication factor set to one. + +Now that you're familiar with measurements, tag sets, and retention policies, let's discuss series. +In InfluxDB, a _**series**_ is a collection of points that share a measurement, tag set, and field key. +The data above consist of eight series: + +| Series number | Measurement | Tag set | Field key | +|:------------------------ | ----------- | ------- | --------- | +| series 1 | `census` | `location = 1`,`scientist = langstroth` | `butterflies` | +| series 2 | `census` | `location = 2`,`scientist = langstroth` | `butterflies` | +| series 3 | `census` | `location = 1`,`scientist = perpetua` | `butterflies` | +| series 4 | `census` | `location = 2`,`scientist = perpetua` | `butterflies` | +| series 5 | `census` | `location = 1`,`scientist = langstroth` | `honeybees` | +| series 6 | `census` | `location = 2`,`scientist = langstroth` | `honeybees` | +| series 7 | `census` | `location = 1`,`scientist = perpetua` | `honeybees` | +| series 8 | `census` | `location = 2`,`scientist = perpetua` | `honeybees` | + +Understanding the concept of a series is essential when designing your [schema](/enterprise_influxdb/v1.9/concepts/glossary/#schema) and when working with your data in InfluxDB. + +A _**point**_ represents a single data record that has four components: a measurement, tag set, field set, and a timestamp. A point is uniquely identified by its series and timestamp. + +For example, here's a single point: +``` +name: census +----------------- +time butterflies honeybees location scientist +2015-08-18T00:00:00Z 1 30 1 perpetua +``` + +The point in this example is part of series 3 and 7 and defined by the measurement (`census`), the tag set (`location = 1`, `scientist = perpetua`), the field set (`butterflies = 1`, `honeybees = 30`), and the timestamp `2015-08-18T00:00:00Z`. + +All of the stuff we've just covered is stored in a database - the sample data are in the database `my_database`. +An InfluxDB _**database**_ is similar to traditional relational databases and serves as a logical container for users, retention policies, continuous queries, and, of course, your time series data. +See [Authentication and Authorization](/enterprise_influxdb/v1.9/administration/authentication_and_authorization/) and [Continuous Queries](/enterprise_influxdb/v1.9/query_language/continuous_queries/) for more on those topics. + +Databases can have several users, continuous queries, retention policies, and measurements. +InfluxDB is a schemaless database which means it's easy to add new measurements, tags, and fields at any time. +It's designed to make working with time series data awesome. + +You made it! +You've covered the fundamental concepts and terminology in InfluxDB. +If you're just starting out, we recommend taking a look at [Getting Started](/enterprise_influxdb/v1.9/introduction/getting_started/) and the [Writing Data](/enterprise_influxdb/v1.9/guides/writing_data/) and [Querying Data](/enterprise_influxdb/v1.9/guides/querying_data/) guides. +May our time series database serve you well 🕔. diff --git a/content/enterprise_influxdb/v1.9/concepts/schema_and_data_layout.md b/content/enterprise_influxdb/v1.9/concepts/schema_and_data_layout.md new file mode 100644 index 000000000..ddd82c15d --- /dev/null +++ b/content/enterprise_influxdb/v1.9/concepts/schema_and_data_layout.md @@ -0,0 +1,251 @@ +--- +title: InfluxDB schema design and data layout +description: > + General guidelines for InfluxDB schema design and data layout. +menu: + enterprise_influxdb_1_9: + name: Schema design and data layout + weight: 50 + parent: Concepts +--- + +Every InfluxDB use case is special and your [schema](/enterprise_influxdb/v1.9/concepts/glossary/#schema) will reflect that uniqueness. +There are, however, general guidelines to follow and pitfalls to avoid when designing your schema. + + + + + + + + +
        General RecommendationsEncouraged Schema DesignDiscouraged Schema DesignShard Group Duration Management
        + +## General recommendations + +### Encouraged schema design + +We recommend that you: + +- [Encode meta data in tags](#encode-meta-data-in-tags) +- [Avoid using keywords as tag or field names](#avoid-using-keywords-as-tag-or-field-names) + +#### Encode meta data in tags + +[Tags](/enterprise_influxdb/v1.9/concepts/glossary/#tag) are indexed and [fields](/enterprise_influxdb/v1.9/concepts/glossary/#field) are not indexed. +This means that queries on tags are more performant than those on fields. + +In general, your queries should guide what gets stored as a tag and what gets stored as a field: + +- Store commonly-queried meta data in tags +- Store data in tags if you plan to use them with the InfluxQL `GROUP BY` clause +- Store data in fields if you plan to use them with an [InfluxQL](/enterprise_influxdb/v1.9/query_language/functions/) function +- Store numeric values as fields ([tag values](/enterprise_influxdb/v1.9/concepts/glossary/#tag-value) only support string values) + +#### Avoid using keywords as tag or field names + +Not required, but simplifies writing queries because you won't have to wrap tag or field names in double quotes. +See [InfluxQL](https://github.com/influxdata/influxql/blob/master/README.md#keywords) and [Flux](https://github.com/influxdata/flux/blob/master/docs/SPEC.md#keywords) keywords to avoid. + +Also, if a tag or field name contains characters other than `[A-z,_]`, you must wrap it in double quotes in InfluxQL or use [bracket notation](/{{< latest "influxdb" "v2" >}}/query-data/get-started/syntax-basics/#records) in Flux. + +### Discouraged schema design + +We recommend that you: + +- [Avoid too many series](#avoid-too-many-series) +- [Avoid the same name for a tag and a field](#avoid-the-same-name-for-a-tag-and-a-field) +- [Avoid encoding data in measurement names](#avoid-encoding-data-in-measurement-names) +- [Avoid putting more than one piece of information in one tag](#avoid-putting-more-than-one-piece-of-information-in-one-tag) + +#### Avoid too many series + +[Tags](/enterprise_influxdb/v1.9/concepts/glossary/#tag) containing highly variable information like UUIDs, hashes, and random strings lead to a large number of [series](/enterprise_influxdb/v1.9/concepts/glossary/#series) in the database, also known as high series cardinality. High series cardinality is a primary driver of high memory usage for many database workloads. + +See [Hardware sizing guidelines](/enterprise_influxdb/v1.9/reference/hardware_sizing/) for [series cardinality](/enterprise_influxdb/v1.9/concepts/glossary/#series-cardinality) recommendations based on your hardware. If the system has memory constraints, consider storing high-cardinality data as a field rather than a tag. + +#### Avoid the same name for a tag and a field + +Avoid using the same name for a tag and field key. +This often results in unexpected behavior when querying data. + +If you inadvertently add the same name for a tag and field key, see +[Frequently asked questions](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#tag-and-field-key-with-the-same-name) +for information about how to query the data predictably and how to fix the issue. + +#### Avoid encoding data in measurement names + +InfluxDB queries merge data that falls within the same [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement); it's better to differentiate data with [tags](/enterprise_influxdb/v1.9/concepts/glossary/#tag) than with detailed measurement names. If you encode data in a measurement name, you must use a regular expression to query the data, making some queries more complicated or impossible. + +_Example:_ + +Consider the following schema represented by line protocol. + +``` +Schema 1 - Data encoded in the measurement name +------------- +blueberries.plot-1.north temp=50.1 1472515200000000000 +blueberries.plot-2.midwest temp=49.8 1472515200000000000 +``` + +The long measurement names (`blueberries.plot-1.north`) with no tags are similar to Graphite metrics. +Encoding the `plot` and `region` in the measurement name makes the data more difficult to query. + +For example, calculating the average temperature of both plots 1 and 2 is not possible with schema 1. +Compare this to schema 2: + +``` +Schema 2 - Data encoded in tags +------------- +weather_sensor,crop=blueberries,plot=1,region=north temp=50.1 1472515200000000000 +weather_sensor,crop=blueberries,plot=2,region=midwest temp=49.8 1472515200000000000 +``` + +Use Flux or InfluxQL to calculate the average `temp` for blueberries in the `north` region: + +##### Flux + +```js +// Schema 1 - Query for data encoded in the measurement name +from(bucket:"/") + |> range(start:2016-08-30T00:00:00Z) + |> filter(fn: (r) => r._measurement =~ /\.north$/ and r._field == "temp") + |> mean() + +// Schema 2 - Query for data encoded in tags +from(bucket:"/") + |> range(start:2016-08-30T00:00:00Z) + |> filter(fn: (r) => r._measurement == "weather_sensor" and r.region == "north" and r._field == "temp") + |> mean() +``` + +##### InfluxQL + +``` +# Schema 1 - Query for data encoded in the measurement name +> SELECT mean("temp") FROM /\.north$/ + +# Schema 2 - Query for data encoded in tags +> SELECT mean("temp") FROM "weather_sensor" WHERE "region" = 'north' +``` + +### Avoid putting more than one piece of information in one tag + +Splitting a single tag with multiple pieces into separate tags simplifies your queries and reduces the need for regular expressions. + +Consider the following schema represented by line protocol. + +``` +Schema 1 - Multiple data encoded in a single tag +------------- +weather_sensor,crop=blueberries,location=plot-1.north temp=50.1 1472515200000000000 +weather_sensor,crop=blueberries,location=plot-2.midwest temp=49.8 1472515200000000000 +``` + +The Schema 1 data encodes multiple separate parameters, the `plot` and `region` into a long tag value (`plot-1.north`). +Compare this to the following schema represented in line protocol. + +``` +Schema 2 - Data encoded in multiple tags +------------- +weather_sensor,crop=blueberries,plot=1,region=north temp=50.1 1472515200000000000 +weather_sensor,crop=blueberries,plot=2,region=midwest temp=49.8 1472515200000000000 +``` + +Use Flux or InfluxQL to calculate the average `temp` for blueberries in the `north` region. +Schema 2 is preferable because using multiple tags, you don't need a regular expression. + +##### Flux + +```js +// Schema 1 - Query for multiple data encoded in a single tag +from(bucket:"/") + |> range(start:2016-08-30T00:00:00Z) + |> filter(fn: (r) => r._measurement == "weather_sensor" and r.location =~ /\.north$/ and r._field == "temp") + |> mean() + +// Schema 2 - Query for data encoded in multiple tags +from(bucket:"/") + |> range(start:2016-08-30T00:00:00Z) + |> filter(fn: (r) => r._measurement == "weather_sensor" and r.region == "north" and r._field == "temp") + |> mean() +``` + +##### InfluxQL + +``` +# Schema 1 - Query for multiple data encoded in a single tag +> SELECT mean("temp") FROM "weather_sensor" WHERE location =~ /\.north$/ + +# Schema 2 - Query for data encoded in multiple tags +> SELECT mean("temp") FROM "weather_sensor" WHERE region = 'north' +``` + +## Shard group duration management + +### Shard group duration overview + +InfluxDB stores data in shard groups. +Shard groups are organized by [retention policy](/enterprise_influxdb/v1.9/concepts/glossary/#retention-policy-rp) (RP) and store data with timestamps that fall within a specific time interval called the [shard duration](/enterprise_influxdb/v1.9/concepts/glossary/#shard-duration). + +If no shard group duration is provided, the shard group duration is determined by the RP [duration](/enterprise_influxdb/v1.9/concepts/glossary/#duration) at the time the RP is created. The default values are: + +| RP Duration | Shard Group Duration | +|---|---| +| < 2 days | 1 hour | +| >= 2 days and <= 6 months | 1 day | +| > 6 months | 7 days | + +The shard group duration is also configurable per RP. +To configure the shard group duration, see [Retention Policy Management](/enterprise_influxdb/v1.9/query_language/manage-database/#retention-policy-management). + +### Shard group duration tradeoffs + +Determining the optimal shard group duration requires finding the balance between: + +- Better overall performance with longer shards +- Flexibility provided by shorter shards + +#### Long shard group duration + +Longer shard group durations let InfluxDB store more data in the same logical location. +This reduces data duplication, improves compression efficiency, and improves query speed in some cases. + +#### Short shard group duration + +Shorter shard group durations allow the system to more efficiently drop data and record incremental backups. +When InfluxDB enforces an RP it drops entire shard groups, not individual data points, even if the points are older than the RP duration. +A shard group will only be removed once a shard group's duration *end time* is older than the RP duration. + +For example, if your RP has a duration of one day, InfluxDB will drop an hour's worth of data every hour and will always have 25 shard groups. One for each hour in the day and an extra shard group that is partially expiring, but isn't removed until the whole shard group is older than 24 hours. + +>**Note:** A special use case to consider: filtering queries on schema data (such as tags, series, measurements) by time. For example, if you want to filter schema data within a one hour interval, you must set the shard group duration to 1h. For more information, see [filter schema data by time](/enterprise_influxdb/v1.9/query_language/explore-schema/#filter-meta-queries-by-time). + +### Shard group duration recommendations + +The default shard group durations work well for most cases. However, high-throughput or long-running instances will benefit from using longer shard group durations. +Here are some recommendations for longer shard group durations: + +| RP Duration | Shard Group Duration | +|---|---| +| <= 1 day | 6 hours | +| > 1 day and <= 7 days | 1 day | +| > 7 days and <= 3 months | 7 days | +| > 3 months | 30 days | +| infinite | 52 weeks or longer | + +> **Note:** Note that `INF` (infinite) is not a [valid shard group duration](/enterprise_influxdb/v1.9/query_language/manage-database/#retention-policy-management). +In extreme cases where data covers decades and will never be deleted, a long shard group duration like `1040w` (20 years) is perfectly valid. + +Other factors to consider before setting shard group duration: + +* Shard groups should be twice as long as the longest time range of the most frequent queries +* Shard groups should each contain more than 100,000 [points](/enterprise_influxdb/v1.9/concepts/glossary/#point) per shard group +* Shard groups should each contain more than 1,000 points per [series](/enterprise_influxdb/v1.9/concepts/glossary/#series) + +#### Shard group duration for backfilling + +Bulk insertion of historical data covering a large time range in the past will trigger the creation of a large number of shards at once. +The concurrent access and overhead of writing to hundreds or thousands of shards can quickly lead to slow performance and memory exhaustion. + +When writing historical data, we highly recommend temporarily setting a longer shard group duration so fewer shards are created. Typically, a shard group duration of 52 weeks works well for backfilling. diff --git a/content/enterprise_influxdb/v1.9/concepts/storage_engine.md b/content/enterprise_influxdb/v1.9/concepts/storage_engine.md new file mode 100644 index 000000000..f4100abe9 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/concepts/storage_engine.md @@ -0,0 +1,438 @@ +--- +title: In-memory indexing and the Time-Structured Merge Tree (TSM) +description: > + InfluxDB storage engine, in-memory indexing, and the Time-Structured Merge Tree (TSM) in InfluxDB OSS. +menu: + enterprise_influxdb_1_9: + name: In-memory indexing with TSM + weight: 60 + parent: Concepts +v2: /influxdb/v2.0/reference/internals/storage-engine/ +--- + +## The InfluxDB storage engine and the Time-Structured Merge Tree (TSM) + +The InfluxDB storage engine looks very similar to a LSM Tree. +It has a write ahead log and a collection of read-only data files which are similar in concept to SSTables in an LSM Tree. +TSM files contain sorted, compressed series data. + +InfluxDB will create a [shard](/enterprise_influxdb/v1.9/concepts/glossary/#shard) for each block of time. +For example, if you have a [retention policy](/enterprise_influxdb/v1.9/concepts/glossary/#retention-policy-rp) with an unlimited duration, shards will be created for each 7 day block of time. +Each of these shards maps to an underlying storage engine database. +Each of these databases has its own [WAL](/enterprise_influxdb/v1.9/concepts/glossary/#wal-write-ahead-log) and TSM files. + +We'll dig into each of these parts of the storage engine. + +## Storage engine + +The storage engine ties a number of components together and provides the external interface for storing and querying series data. It is composed of a number of components that each serve a particular role: + +* In-Memory Index - The in-memory index is a shared index across shards that provides the quick access to [measurements](/enterprise_influxdb/v1.9/concepts/glossary/#measurement), [tags](/enterprise_influxdb/v1.9/concepts/glossary/#tag), and [series](/enterprise_influxdb/v1.9/concepts/glossary/#series). The index is used by the engine, but is not specific to the storage engine itself. +* WAL - The WAL is a write-optimized storage format that allows for writes to be durable, but not easily queryable. Writes to the WAL are appended to segments of a fixed size. +* Cache - The Cache is an in-memory representation of the data stored in the WAL. It is queried at runtime and merged with the data stored in TSM files. +* TSM Files - TSM files store compressed series data in a columnar format. +* FileStore - The FileStore mediates access to all TSM files on disk. It ensures that TSM files are installed atomically when existing ones are replaced as well as removing TSM files that are no longer used. +* Compactor - The Compactor is responsible for converting less optimized Cache and TSM data into more read-optimized formats. It does this by compressing series, removing deleted data, optimizing indices and combining smaller files into larger ones. +* Compaction Planner - The Compaction Planner determines which TSM files are ready for a compaction and ensures that multiple concurrent compactions do not interfere with each other. +* Compression - Compression is handled by various Encoders and Decoders for specific data types. Some encoders are fairly static and always encode the same type the same way; others switch their compression strategy based on the shape of the data. +* Writers/Readers - Each file type (WAL segment, TSM files, tombstones, etc..) has Writers and Readers for working with the formats. + +### Write Ahead Log (WAL) + +The WAL is organized as a bunch of files that look like `_000001.wal`. +The file numbers are monotonically increasing and referred to as WAL segments. +When a segment reaches 10MB in size, it is closed and a new one is opened. Each WAL segment stores multiple compressed blocks of writes and deletes. + +When a write comes in the new points are serialized, compressed using Snappy, and written to a WAL file. +The file is `fsync`'d and the data is added to an in-memory index before a success is returned. +This means that batching points together is required to achieve high throughput performance. +(Optimal batch size seems to be 5,000-10,000 points per batch for many use cases.) + +Each entry in the WAL follows a [TLV standard](https://en.wikipedia.org/wiki/Type-length-value) with a single byte representing the type of entry (write or delete), a 4 byte `uint32` for the length of the compressed block, and then the compressed block. + +### Cache + +The Cache is an in-memory copy of all data points current stored in the WAL. +The points are organized by the key, which is the measurement, [tag set](/enterprise_influxdb/v1.9/concepts/glossary/#tag-set), and unique [field](/enterprise_influxdb/v1.9/concepts/glossary/#field). +Each field is kept as its own time-ordered range. +The Cache data is not compressed while in memory. + +Queries to the storage engine will merge data from the Cache with data from the TSM files. +Queries execute on a copy of the data that is made from the cache at query processing time. +This way writes that come in while a query is running won't affect the result. + +Deletes sent to the Cache will clear out the given key or the specific time range for the given key. + +The Cache exposes a few controls for snapshotting behavior. +The two most important controls are the memory limits. +There is a lower bound, [`cache-snapshot-memory-size`](/enterprise_influxdb/v1.9/administration/config#cache-snapshot-memory-size-25m), which when exceeded will trigger a snapshot to TSM files and remove the corresponding WAL segments. +There is also an upper bound, [`cache-max-memory-size`](/enterprise_influxdb/v1.9/administration/config#cache-max-memory-size-1g), which when exceeded will cause the Cache to reject new writes. +These configurations are useful to prevent out of memory situations and to apply back pressure to clients writing data faster than the instance can persist it. +The checks for memory thresholds occur on every write. + +The other snapshot controls are time based. +The idle threshold, [`cache-snapshot-write-cold-duration`](/enterprise_influxdb/v1.9/administration/config#cache-snapshot-write-cold-duration-10m), forces the Cache to snapshot to TSM files if it hasn't received a write within the specified interval. + +The in-memory Cache is recreated on restart by re-reading the WAL files on disk. + +### TSM files + +TSM files are a collection of read-only files that are memory mapped. +The structure of these files looks very similar to an SSTable in LevelDB or other LSM Tree variants. + +A TSM file is composed of four sections: header, blocks, index, and footer. + +``` ++--------+------------------------------------+-------------+--------------+ +| Header | Blocks | Index | Footer | +|5 bytes | N bytes | N bytes | 4 bytes | ++--------+------------------------------------+-------------+--------------+ +``` + +The Header is a magic number to identify the file type and a version number. + +``` ++-------------------+ +| Header | ++-------------------+ +| Magic │ Version | +| 4 bytes │ 1 byte | ++-------------------+ +``` + +Blocks are sequences of pairs of CRC32 checksums and data. +The block data is opaque to the file. +The CRC32 is used for block level error detection. +The length of the blocks is stored in the index. + +``` ++--------------------------------------------------------------------+ +│ Blocks │ ++---------------------+-----------------------+----------------------+ +| Block 1 | Block 2 | Block N | ++---------------------+-----------------------+----------------------+ +| CRC | Data | CRC | Data | CRC | Data | +| 4 bytes | N bytes | 4 bytes | N bytes | 4 bytes | N bytes | ++---------------------+-----------------------+----------------------+ +``` + +Following the blocks is the index for the blocks in the file. +The index is composed of a sequence of index entries ordered lexicographically by key and then by time. +The key includes the measurement name, tag set, and one field. +Multiple fields per point creates multiple index entries in the TSM file. +Each index entry starts with a key length and the key, followed by the block type (float, int, bool, string) and a count of the number of index block entries that follow for that key. +Each index block entry is composed of the min and max time for the block, the offset into the file where the block is located and the size of the block. There is one index block entry for each block in the TSM file that contains the key. + +The index structure can provide efficient access to all blocks as well as the ability to determine the cost associated with accessing a given key. +Given a key and timestamp, we can determine whether a file contains the block for that timestamp. +We can also determine where that block resides and how much data must be read to retrieve the block. +Knowing the size of the block, we can efficiently provision our IO statements. + +``` ++-----------------------------------------------------------------------------+ +│ Index │ ++-----------------------------------------------------------------------------+ +│ Key Len │ Key │ Type │ Count │Min Time │Max Time │ Offset │ Size │...│ +│ 2 bytes │ N bytes │1 byte│2 bytes│ 8 bytes │ 8 bytes │8 bytes │4 bytes │ │ ++-----------------------------------------------------------------------------+ +``` + +The last section is the footer that stores the offset of the start of the index. + +``` ++---------+ +│ Footer │ ++---------+ +│Index Ofs│ +│ 8 bytes │ ++---------+ +``` + +### Compression + +Each block is compressed to reduce storage space and disk IO when querying. +A block contains the timestamps and values for a given series and field. +Each block has one byte header, followed by the compressed timestamps and then the compressed values. + +``` ++--------------------------------------------------+ +| Type | Len | Timestamps | Values | +|1 Byte | VByte | N Bytes | N Bytes │ ++--------------------------------------------------+ +``` + +The timestamps and values are compressed and stored separately using encodings dependent on the data type and its shape. +Storing them independently allows timestamp encoding to be used for all timestamps, while allowing different encodings for different field types. +For example, some points may be able to use run-length encoding whereas other may not. + +Each value type also contains a 1 byte header indicating the type of compression for the remaining bytes. +The four high bits store the compression type and the four low bits are used by the encoder if needed. + +#### Timestamps + +Timestamp encoding is adaptive and based on the structure of the timestamps that are encoded. +It uses a combination of delta encoding, scaling, and compression using simple8b run-length encoding, as well as falling back to no compression if needed. + +Timestamp resolution is variable but can be as granular as a nanosecond, requiring up to 8 bytes to store uncompressed. +During encoding, the values are first delta-encoded. +The first value is the starting timestamp and subsequent values are the differences from the prior value. +This usually converts the values into much smaller integers that are easier to compress. +Many timestamps are also monotonically increasing and fall on even boundaries of time such as every 10s. +When timestamps have this structure, they are scaled by the largest common divisor that is also a factor of 10. +This has the effect of converting very large integer deltas into smaller ones that compress even better. + +Using these adjusted values, if all the deltas are the same, the time range is stored using run-length encoding. +If run-length encoding is not possible and all values are less than (1 << 60) - 1 ([~36.5 years](https://www.wolframalpha.com/input/?i=\(1+%3C%3C+60\)+-+1+nanoseconds+to+years) at nanosecond resolution), then the timestamps are encoded using [simple8b encoding](https://github.com/jwilder/encoding/tree/master/simple8b). +Simple8b encoding is a 64bit word-aligned integer encoding that packs multiple integers into a single 64bit word. +If any value exceeds the maximum the deltas are stored uncompressed using 8 bytes each for the block. +Future encodings may use a patched scheme such as Patched Frame-Of-Reference (PFOR) to handle outliers more effectively. + +#### Floats + +Floats are encoded using an implementation of the [Facebook Gorilla paper](http://www.vldb.org/pvldb/vol8/p1816-teller.pdf). +The encoding XORs consecutive values together to produce a small result when the values are close together. +The delta is then stored using control bits to indicate how many leading and trailing zeroes are in the XOR value. +Our implementation removes the timestamp encoding described in paper and only encodes the float values. + +#### Integers + +Integer encoding uses two different strategies depending on the range of values in the uncompressed data. +Encoded values are first encoded using [ZigZag encoding](https://developers.google.com/protocol-buffers/docs/encoding#signed-integers). +This interleaves positive and negative integers across a range of positive integers. + +For example, [-2,-1,0,1] becomes [3,1,0,2]. +See Google's [Protocol Buffers documentation](https://developers.google.com/protocol-buffers/docs/encoding#signed-integers) for more information. + +If all ZigZag encoded values are less than (1 << 60) - 1, they are compressed using simple8b encoding. +If any values are larger than the maximum then all values are stored uncompressed in the block. +If all values are identical, run-length encoding is used. +This works very well for values that are frequently constant. + +#### Booleans + +Booleans are encoded using a simple bit packing strategy where each Boolean uses 1 bit. +The number of Booleans encoded is stored using variable-byte encoding at the beginning of the block. + +#### Strings +Strings are encoding using [Snappy](http://google.github.io/snappy/) compression. +Each string is packed consecutively and they are compressed as one larger block. + +### Compactions + +Compactions are recurring processes that migrate data stored in a write-optimized format into a more read-optimized format. +There are a number of stages of compaction that take place while a shard is hot for writes: + +* Snapshots - Values in the Cache and WAL must be converted to TSM files to free memory and disk space used by the WAL segments. +These compactions occur based on the cache memory and time thresholds. +* Level Compactions - Level compactions (levels 1-4) occur as the TSM files grow. +TSM files are compacted from snapshots to level 1 files. +Multiple level 1 files are compacted to produce level 2 files. +The process continues until files reach level 4 and the max size for a TSM file. +They will not be compacted further unless deletes, index optimization compactions, or full compactions need to run. +Lower level compactions use strategies that avoid CPU-intensive activities like decompressing and combining blocks. +Higher level (and thus less frequent) compactions will re-combine blocks to fully compact them and increase the compression ratio. +* Index Optimization - When many level 4 TSM files accumulate, the internal indexes become larger and more costly to access. +An index optimization compaction splits the series and indices across a new set of TSM files, sorting all points for a given series into one TSM file. +Before an index optimization, each TSM file contained points for most or all series, and thus each contains the same series index. +After an index optimization, each TSM file contains points from a minimum of series and there is little series overlap between files. +Each TSM file thus has a smaller unique series index, instead of a duplicate of the full series list. +In addition, all points from a particular series are contiguous in a TSM file rather than spread across multiple TSM files. +* Full Compactions - Full compactions run when a shard has become cold for writes for long time, or when deletes have occurred on the shard. +Full compactions produce an optimal set of TSM files and include all optimizations from Level and Index Optimization compactions. +Once a shard is fully compacted, no other compactions will run on it unless new writes or deletes are stored. + +### Writes + +Writes are appended to the current WAL segment and are also added to the Cache. +Each WAL segment has a maximum size. +Writes roll over to a new file once the current file fills up. +The cache is also size bounded; snapshots are taken and WAL compactions are initiated when the cache becomes too full. +If the inbound write rate exceeds the WAL compaction rate for a sustained period, the cache may become too full, in which case new writes will fail until the snapshot process catches up. + +When WAL segments fill up and are closed, the Compactor snapshots the Cache and writes the data to a new TSM file. +When the TSM file is successfully written and `fsync`'d, it is loaded and referenced by the FileStore. + +### Updates + +Updates (writing a newer value for a point that already exists) occur as normal writes. +Since cached values overwrite existing values, newer writes take precedence. +If a write would overwrite a point in a prior TSM file, the points are merged at query runtime and the newer write takes precedence. + + +### Deletes + +Deletes occur by writing a delete entry to the WAL for the measurement or series and then updating the Cache and FileStore. +The Cache evicts all relevant entries. +The FileStore writes a tombstone file for each TSM file that contains relevant data. +These tombstone files are used at startup time to ignore blocks as well as during compactions to remove deleted entries. + +Queries against partially deleted series are handled at query time until a compaction removes the data fully from the TSM files. + +### Queries + +When a query is executed by the storage engine, it is essentially a seek to a given time associated with a specific series key and field. +First, we do a search on the data files to find the files that contain a time range matching the query as well containing matching series. + +Once we have the data files selected, we next need to find the position in the file of the series key index entries. +We run a binary search against each TSM index to find the location of its index blocks. + +In common cases the blocks will not overlap across multiple TSM files and we can search the index entries linearly to find the start block from which to read. +If there are overlapping blocks of time, the index entries are sorted to ensure newer writes will take precedence and that blocks can be processed in order during query execution. + +When iterating over the index entries the blocks are read sequentially from the blocks section. +The block is decompressed and we seek to the specific point. + + +# The new InfluxDB storage engine: from LSM Tree to B+Tree and back again to create the Time Structured Merge Tree + +Writing a new storage format should be a last resort. +So how did InfluxData end up writing our own engine? +InfluxData has experimented with many storage formats and found each lacking in some fundamental way. +The performance requirements for InfluxDB are significant, and eventually overwhelm other storage systems. +The 0.8 line of InfluxDB allowed multiple storage engines, including LevelDB, RocksDB, HyperLevelDB, and LMDB. +The 0.9 line of InfluxDB used BoltDB as the underlying storage engine. +This writeup is about the Time Structured Merge Tree storage engine that was released in 0.9.5 and is the only storage engine supported in InfluxDB 0.11+, including the entire 1.x family. + +The properties of the time series data use case make it challenging for many existing storage engines. +Over the course of InfluxDB development, InfluxData tried a few of the more popular options. +We started with LevelDB, an engine based on LSM Trees, which are optimized for write throughput. +After that we tried BoltDB, an engine based on a memory mapped B+Tree, which is optimized for reads. +Finally, we ended up building our own storage engine that is similar in many ways to LSM Trees. + +With our new storage engine we were able to achieve up to a 45x reduction in disk space usage from our B+Tree setup with even greater write throughput and compression than what we saw with LevelDB and its variants. +This post will cover the details of that evolution and end with an in-depth look at our new storage engine and its inner workings. + +## Properties of time series data + +The workload of time series data is quite different from normal database workloads. +There are a number of factors that conspire to make it very difficult to scale and remain performant: + +* Billions of individual data points +* High write throughput +* High read throughput +* Large deletes (data expiration) +* Mostly an insert/append workload, very few updates + +The first and most obvious problem is one of scale. +In DevOps, IoT, or APM it is easy to collect hundreds of millions or billions of unique data points every day. + +For example, let's say we have 200 VMs or servers running, with each server collecting an average of 100 measurements every 10 seconds. +Given there are 86,400 seconds in a day, a single measurement will generate 8,640 points in a day per server. +That gives us a total of 172,800,000 (`200 * 100 * 8,640`) individual data points per day. +We find similar or larger numbers in sensor data use cases. + +The volume of data means that the write throughput can be very high. +We regularly get requests for setups than can handle hundreds of thousands of writes per second. +Some larger companies will only consider systems that can handle millions of writes per second. + +At the same time, time series data can be a high read throughput use case. +It's true that if you're tracking 700,000 unique metrics or time series you can't hope to visualize all of them. +That leads many people to think that you don't actually read most of the data that goes into the database. +However, other than dashboards that people have up on their screens, there are automated systems for monitoring or combining the large volume of time series data with other types of data. + +Inside InfluxDB, aggregate functions calculated on the fly may combine tens of thousands of distinct time series into a single view. +Each one of those queries must read each aggregated data point, so for InfluxDB the read throughput is often many times higher than the write throughput. + +Given that time series is mostly an append-only workload, you might think that it's possible to get great performance on a B+Tree. +Appends in the keyspace are efficient and you can achieve greater than 100,000 per second. +However, we have those appends happening in individual time series. +So the inserts end up looking more like random inserts than append only inserts. + +One of the biggest problems we found with time series data is that it's very common to delete all data after it gets past a certain age. +The common pattern here is that users have high precision data that is kept for a short period of time like a few days or months. +Users then downsample and aggregate that data into lower precision rollups that are kept around much longer. + +The naive implementation would be to simply delete each record once it passes its expiration time. +However, that means that once the first points written reach their expiration date, the system is processing just as many deletes as writes, which is something most storage engines aren't designed for. + +Let's dig into the details of the two types of storage engines we tried and how these properties had a significant impact on our performance. + +## LevelDB and log structured merge trees + +When the InfluxDB project began, we picked LevelDB as the storage engine because we had used it for time series data storage in the product that was the precursor to InfluxDB. +We knew that it had great properties for write throughput and everything seemed to "just work". + +LevelDB is an implementation of a log structured merge tree (LSM tree) that was built as an open source project at Google. +It exposes an API for a key-value store where the key space is sorted. +This last part is important for time series data as it allowed us to quickly scan ranges of time as long as the timestamp was in the key. + +LSM Trees are based on a log that takes writes and two structures known as Mem Tables and SSTables. +These tables represent the sorted keyspace. +SSTables are read only files that are continuously replaced by other SSTables that merge inserts and updates into the keyspace. + +The two biggest advantages that LevelDB had for us were high write throughput and built in compression. +However, as we learned more about what people needed with time series data, we encountered a few insurmountable challenges. + +The first problem we had was that LevelDB doesn't support hot backups. +If you want to do a safe backup of the database, you have to close it and then copy it. +The LevelDB variants RocksDB and HyperLevelDB fix this problem, but there was another more pressing problem that we didn't think they could solve. + +Our users needed a way to automatically manage data retention. +That meant we needed deletes on a very large scale. +In LSM Trees, a delete is as expensive, if not more so, than a write. +A delete writes a new record known as a tombstone. +After that queries merge the result set with any tombstones to purge the deleted data from the query return. +Later, a compaction runs that removes the tombstone record and the underlying deleted record in the SSTable file. + +To get around doing deletes, we split data across what we call shards, which are contiguous blocks of time. +Shards would typically hold either one day or seven days worth of data. +Each shard mapped to an underlying LevelDB. +This meant that we could drop an entire day of data by just closing out the database and removing the underlying files. + +Users of RocksDB may at this point bring up a feature called ColumnFamilies. +When putting time series data into Rocks, it's common to split blocks of time into column families and then drop those when their time is up. +It's the same general idea: create a separate area where you can just drop files instead of updating indexes when you delete a large block of data. +Dropping a column family is a very efficient operation. +However, column families are a fairly new feature and we had another use case for shards. + +Organizing data into shards meant that it could be moved within a cluster without having to examine billions of keys. +At the time of this writing, it was not possible to move a column family in one RocksDB to another. +Old shards are typically cold for writes so moving them around would be cheap and easy. +We would have the added benefit of having a spot in the keyspace that is cold for writes so it would be easier to do consistency checks later. + +The organization of data into shards worked great for a while, until a large amount of data went into InfluxDB. +LevelDB splits the data out over many small files. +Having dozens or hundreds of these databases open in a single process ended up creating a big problem. +Users that had six months or a year of data would run out of file handles. +It's not something we found with the majority of users, but anyone pushing the database to its limits would hit this problem and we had no fix for it. +There were simply too many file handles open. + +## BoltDB and mmap B+Trees + +After struggling with LevelDB and its variants for a year we decided to move over to BoltDB, a pure Golang database heavily inspired by LMDB, a mmap B+Tree database written in C. +It has the same API semantics as LevelDB: a key value store where the keyspace is ordered. +Many of our users were surprised. +Our own posted tests of the LevelDB variants vs. LMDB (a mmap B+Tree) showed RocksDB as the best performer. + +However, there were other considerations that went into this decision outside of the pure write performance. +At this point our most important goal was to get to something stable that could be run in production and backed up. +BoltDB also had the advantage of being written in pure Go, which simplified our build chain immensely and made it easy to build for other OSes and platforms. + +The biggest win for us was that BoltDB used a single file as the database. +At this point our most common source of bug reports were from people running out of file handles. +Bolt solved the hot backup problem and the file limit problems all at the same time. + +We were willing to take a hit on write throughput if it meant that we'd have a system that was more reliable and stable that we could build on. +Our reasoning was that for anyone pushing really big write loads, they'd be running a cluster anyway. + +We released versions 0.9.0 to 0.9.2 based on BoltDB. +From a development perspective it was delightful. +Clean API, fast and easy to build in our Go project, and reliable. +However, after running for a while we found a big problem with write throughput. +After the database got over a few GB, writes would start spiking IOPS. + +Some users were able to get past this by putting InfluxDB on big hardware with near unlimited IOPS. +However, most users are on VMs with limited resources in the cloud. +We had to figure out a way to reduce the impact of writing a bunch of points into hundreds of thousands of series at a time. + +With the 0.9.3 and 0.9.4 releases our plan was to put a write ahead log (WAL) in front of Bolt. +That way we could reduce the number of random insertions into the keyspace. +Instead, we'd buffer up multiple writes that were next to each other and then flush them at once. +However, that only served to delay the problem. +High IOPS still became an issue and it showed up very quickly for anyone operating at even moderate work loads. + +However, our experience building the first WAL implementation in front of Bolt gave us the confidence we needed that the write problem could be solved. +The performance of the WAL itself was fantastic, the index simply could not keep up. +At this point we started thinking again about how we could create something similar to an LSM Tree that could keep up with our write load. + +Thus was born the Time Structured Merge Tree. diff --git a/content/enterprise_influxdb/v1.9/concepts/time-series-index.md b/content/enterprise_influxdb/v1.9/concepts/time-series-index.md new file mode 100644 index 000000000..68f2cd296 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/concepts/time-series-index.md @@ -0,0 +1,51 @@ +--- +title: Time Series Index (TSI) overview +description: > + The Time Series Index (TSI) storage engine supports high cardinality in time series data. +menu: + enterprise_influxdb_1_9: + name: Time Series Index (TSI) overview + weight: 70 + parent: Concepts +--- + +Find overview and background information on Time Series Index (TSI) in this topic. For detail, including how to enable and configure TSI, see [Time Series Index (TSI) details](/enterprise_influxdb/v1.9/concepts/tsi-details/). + +## Overview + +To support a large number of time series, that is, a very high cardinality in the number of unique time series that the database stores, InfluxData has added the new Time Series Index (TSI). +InfluxData supports customers using InfluxDB with tens of millions of time series. +InfluxData's goal, however, is to expand to hundreds of millions, and eventually billions. +Using InfluxData's TSI storage engine, users should be able to have millions of unique time series. +The goal is that the number of series should be unbounded by the amount of memory on the server hardware. +Importantly, the number of series that exist in the database will have a negligible impact on database startup time. +This work represents the most significant technical advancement in the database since InfluxData released the Time Series Merge Tree (TSM) storage engine in 2016. + +## Background information + +InfluxDB actually looks like two databases in one, a time series data store and an inverted index for the measurement, tag, and field metadata. + +### Time-Structured Merge Tree (TSM) + +The Time-Structured Merge Tree (TSM) engine solves the problem of getting maximum throughput, compression, and query speed for raw time series data. +Up until TSI, the inverted index was an in-memory data structure that was built during startup of the database based on the data in TSM. +This meant that for every measurement, tag key-value pair, and field name, there was a lookup table in-memory to map those bits of metadata to an underlying time series. +For users with a high number of ephemeral series, memory utilization continued increasing as new time series were created. +And, startup times increased since all of that data would have to be loaded onto the heap at start time. + +> For details, see [TSM-based data storage and in-memory indexing](/enterprise_influxdb/v1.9/concepts/storage_engine/). + +### Time Series Index (TSI) + +The new time series index (TSI) moves the index to files on disk that we memory map. +This means that we let the operating system handle being the Least Recently Used (LRU) memory. +Much like the TSM engine for raw time series data we have a write-ahead log with an in-memory structure that gets merged at query time with the memory-mapped index. +Background routines run constantly to compact the index into larger and larger files to avoid having to do too many index merges at query time. +Under the covers, we’re using techniques like Robin Hood Hashing to do fast index lookups and HyperLogLog++ to keep sketches of cardinality estimates. +The latter will give us the ability to add things to the query languages like the [SHOW CARDINALITY](/enterprise_influxdb/v1.9/query_language/spec#show-cardinality) queries. + +### Issues solved by TSI and remaining to be solved + +The primary issue that Time Series Index (TSI) addresses is ephemeral time series. Most frequently, this occurs in use cases that want to track per process metrics or per container metrics by putting identifiers in tags. For example, the [Heapster project for Kubernetes](https://github.com/kubernetes/heapster) does this. For series that are no longer hot for writes or queries, they won’t take up space in memory. + +The issue that the Heapster project and similar use cases did not address is limiting the scope of data returned by the SHOW queries. We’ll have updates to the query language in the future to limit those results by time. We also don’t solve the problem of having all these series hot for reads and writes. For that problem, scale-out clustering is the solution. We’ll have to continue to optimize the query language and engine to work with large sets of series. We’ll need to add guard rails and limits into the language and eventually, add spill-to-disk query processing. That work will be on-going in every release of InfluxDB. diff --git a/content/enterprise_influxdb/v1.9/concepts/tsi-details.md b/content/enterprise_influxdb/v1.9/concepts/tsi-details.md new file mode 100644 index 000000000..10ff73b4a --- /dev/null +++ b/content/enterprise_influxdb/v1.9/concepts/tsi-details.md @@ -0,0 +1,172 @@ +--- +title: Time Series Index (TSI) details +description: Enable and understand the Time Series Index (TSI). +menu: + enterprise_influxdb_1_9: + name: Time Series Index (TSI) details + weight: 80 + parent: Concepts +--- + +When InfluxDB ingests data, we store not only the value but we also index the measurement and tag information so that it can be queried quickly. +In earlier versions, index data could only be stored in-memory, however, that requires a lot of RAM and places an upper bound on the number of series a machine can hold. +This upper bound is usually somewhere between 1 - 4 million series depending on the machine used. + +The Time Series Index (TSI) was developed to allow us to go past that upper bound. +TSI stores index data on disk so that we are no longer restricted by RAM. +TSI uses the operating system's page cache to pull hot data into memory and let cold data rest on disk. + +## Enable TSI + +To enable TSI, set the following line in the InfluxDB configuration file (`influxdb.conf`): + +``` +index-version = "tsi1" +``` + +(Be sure to include the double quotes.) + +### InfluxDB Enterprise + +- To convert your data nodes to support TSI, see [Upgrade InfluxDB Enterprise clusters](/enterprise_influxdb/v1.8/administration/upgrading/). + +- For detail on configuration, see [Configure InfluxDB Enterprise clusters](/enterprise_influxdb/v1.8/administration/configuration/). + +### InfluxDB OSS + +- For detail on configuration, see [Configuring InfluxDB OSS](/enterprise_influxdb/v1.9/administration/config/). + +## Tooling + +### `influx_inspect dumptsi` + +If you are troubleshooting an issue with an index, you can use the `influx_inspect dumptsi` command. +This command allows you to print summary statistics on an index, file, or a set of files. +This command only works on one index at a time. + +For details on this command, see [influx_inspect dumptsi](/enterprise_influxdb/v1.9/tools/influx_inspect/#dumptsi). + +### `influx_inspect buildtsi` + +If you want to convert an existing shard from an in-memory index to a TSI index, or if you have an existing TSI index which has become corrupt, you can use the `buildtsi` command to create the index from the underlying TSM data. +If you have an existing TSI index that you want to rebuild, first delete the `index` directory within your shard. + +This command works at the server-level but you can optionally add database, retention policy and shard filters to only apply to a subset of shards. + +For details on this command, see [influx inspect buildtsi](/enterprise_influxdb/v1.9/tools/influx_inspect/#buildtsi). + + +## Understanding TSI + +### File organization + +TSI (Time Series Index) is a log-structured merge tree-based database for InfluxDB series data. +TSI is composed of several parts: + +* **Index**: Contains the entire index dataset for a single shard. + +* **Partition**: Contains a sharded partition of the data for a shard. + +* **LogFile**: Contains newly written series as an in-memory index and is persisted as a WAL. + +* **IndexFile**: Contains an immutable, memory-mapped index built from a LogFile or merged from two contiguous index files. + +There is also a **SeriesFile** which contains a set of all series keys across the entire database. +Each shard within the database shares the same series file. + +### Writes + +The following occurs when a write comes into the system: + +1. Series is added to the series file or is looked up if it already exists. This returns an auto-incrementing series ID. +2. The series is sent to the Index. The index maintains a roaring bitmap of existing series IDs and ignores series that have already been created. +3. The series is hashed and sent to the appropriate Partition. +4. The Partition writes the series as an entry to the LogFile. +5. The LogFile writes the series to a write-ahead log file on disk and adds the series to a set of in-memory indexes. + +### Compaction + +Once the LogFile exceeds a threshold (5MB), then a new active log file is created and the previous one begins compacting into an IndexFile. +This first index file is at level 1 (L1). +The log file is considered level 0 (L0). + +Index files can also be created by merging two smaller index files together. +For example, if contiguous two L1 index files exist then they can be merged into an L2 index file. + +### Reads + +The index provides several API calls for retrieving sets of data such as: + +* `MeasurementIterator()`: Returns a sorted list of measurement names. +* `TagKeyIterator()`: Returns a sorted list of tag keys in a measurement. +* `TagValueIterator()`: Returns a sorted list of tag values for a tag key. +* `MeasurementSeriesIDIterator()`: Returns a sorted list of all series IDs for a measurement. +* `TagKeySeriesIDIterator()`: Returns a sorted list of all series IDs for a tag key. +* `TagValueSeriesIDIterator()`: Returns a sorted list of all series IDs for a tag value. + +These iterators are all composable using several merge iterators. +For each type of iterator (measurement, tag key, tag value, series id), there are multiple merge iterator types: + +* **Merge**: Deduplicates items from two iterators. +* **Intersect**: Returns only items that exist in two iterators. +* **Difference**: Only returns items from first iterator that don't exist in the second iterator. + +For example, a query with a WHERE clause of `region != 'us-west'` that operates across two shards will construct a set of iterators like this: + +``` +DifferenceSeriesIDIterators( + MergeSeriesIDIterators( + Shard1.MeasurementSeriesIDIterator("m"), + Shard2.MeasurementSeriesIDIterator("m"), + ), + MergeSeriesIDIterators( + Shard1.TagValueSeriesIDIterator("m", "region", "us-west"), + Shard2.TagValueSeriesIDIterator("m", "region", "us-west"), + ), +) +``` + +### Log File Structure + +The log file is simply structured as a list of LogEntry objects written to disk in sequential order. Log files are written until they reach 5MB and then they are compacted into index files. +The entry objects in the log can be of any of the following types: + +* AddSeries +* DeleteSeries +* DeleteMeasurement +* DeleteTagKey +* DeleteTagValue + +The in-memory index on the log file tracks the following: + +* Measurements by name +* Tag keys by measurement +* Tag values by tag key +* Series by measurement +* Series by tag value +* Tombstones for series, measurements, tag keys, and tag values. + +The log file also maintains bitsets for series ID existence and tombstones. +These bitsets are merged with other log files and index files to regenerate the full index bitset on startup. + +### Index File Structure + +The index file is an immutable file that tracks similar information to the log file, but all data is indexed and written to disk so that it can be directly accessed from a memory-map. + +The index file has the following sections: + +* **TagBlocks:** Maintains an index of tag values for a single tag key. +* **MeasurementBlock:** Maintains an index of measurements and their tag keys. +* **Trailer:** Stores offset information for the file as well as HyperLogLog sketches for cardinality estimation. + +### Manifest + +The MANIFEST file is stored in the index directory and lists all the files that belong to the index and the order in which they should be accessed. +This file is updated every time a compaction occurs. +Any files that are in the directory that are not in the index file are index files that are in the process of being compacted. + +### FileSet + +A file set is an in-memory snapshot of the manifest that is obtained while the InfluxDB process is running. +This is required to provide a consistent view of the index at a point-in-time. +The file set also facilitates reference counting for all of its files so that no file will be deleted via compaction until all readers of the file are done with it. diff --git a/content/enterprise_influxdb/v1.9/features/_index.md b/content/enterprise_influxdb/v1.9/features/_index.md new file mode 100644 index 000000000..14c68de17 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/features/_index.md @@ -0,0 +1,12 @@ +--- +title: InfluxDB Enterprise features +description: Users, clustering, and other InfluxDB Enterprise features. +aliases: + - /enterprise/v1.8/features/ +menu: + enterprise_influxdb_1_9: + name: Enterprise features + weight: 60 +--- + +{{< children hlevel="h2" >}} diff --git a/content/enterprise_influxdb/v1.9/features/clustering-features.md b/content/enterprise_influxdb/v1.9/features/clustering-features.md new file mode 100644 index 000000000..c24d45b17 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/features/clustering-features.md @@ -0,0 +1,128 @@ +--- +title: InfluxDB Enterprise cluster features +description: Overview of features related to InfluxDB Enterprise clustering. +aliases: + - /enterprise/v1.8/features/clustering-features/ +menu: + enterprise_influxdb_1_9: + name: Cluster features + weight: 20 + parent: Enterprise features +--- + +## Entitlements + +A valid license key is required in order to start `influxd-meta` or `influxd`. +License keys restrict the number of data nodes that can be added to a cluster as well as the number of CPU cores a data node can use. +Without a valid license, the process will abort startup. + +Access your license expiration date with the `/debug/vars` endpoint. + +{{< keep-url >}} +```sh +$ curl http://localhost:8086/debug/vars | jq '.entitlements' +{ + "name": "entitlements", + "tags": null, + "values": { + "licenseExpiry": "2022-02-15T00:00:00Z", + "licenseType": "license-key" + } +} +``` +{{% caption %}} +This examples uses `curl` and [`jq`](https://stedolan.github.io/jq/). +{{% /caption %}} + +## Query management + +Query management works cluster wide. Specifically, `SHOW QUERIES` and `KILL QUERY ` on `""` can be run on any data node. `SHOW QUERIES` will report all queries running across the cluster and the node which is running the query. +`KILL QUERY` can abort queries running on the local node or any other remote data node. For details on using the `SHOW QUERIES` and `KILL QUERY` on InfluxDB Enterprise clusters, +see [Query Management](/enterprise_influxdb/v1.9/troubleshooting/query_management/). + +## Subscriptions + +Subscriptions used by Kapacitor work in a cluster. Writes to any node will be forwarded to subscribers across all supported subscription protocols. + +## Continuous queries + +### Configuration and operational considerations on a cluster + +It is important to understand how to configure InfluxDB Enterprise and how this impacts the continuous queries (CQ) engine’s behavior: + +- **Data node configuration** `[continuous queries]` +[run-interval](/enterprise_influxdb/v1.9/administration/config-data-nodes#run-interval-1s) +-- The interval at which InfluxDB checks to see if a CQ needs to run. Set this option to the lowest interval +at which your CQs run. For example, if your most frequent CQ runs every minute, set run-interval to 1m. +- **Meta node configuration** `[meta]` +[lease-duration](/enterprise_influxdb/v1.9/administration/config-meta-nodes#lease-duration-1m0s) +-- The default duration of the leases that data nodes acquire from the meta nodes. Leases automatically expire after the +lease-duration is met. Leases ensure that only one data node is running something at a given time. For example, Continuous +Queries use a lease so that all data nodes aren’t running the same CQs at once. +- **Execution time of CQs** – CQs are sequentially executed. Depending on the amount of work that they need to accomplish +in order to complete, the configuration parameters mentioned above can have an impact on the observed behavior of CQs. + +The CQ service is running on every node, but only a single node is granted exclusive access to execute CQs at any one time. +However, every time the `run-interval` elapses (and assuming a node isn't currently executing CQs), a node attempts to +acquire the CQ lease. By default the `run-interval` is one second – so the data nodes are aggressively checking to see +if they can acquire the lease. On clusters where all CQs execute in an amount of time less than `lease-duration` +(default is 1m), there's a good chance that the first data node to acquire the lease will still hold the lease when +the `run-interval` elapses. Other nodes will be denied the lease and when the node holding the lease requests it again, +the lease is renewed with the expiration extended to `lease-duration`. So in a typical situation, we observe that a +single data node acquires the CQ lease and holds on to it. It effectively becomes the executor of CQs until it is +recycled (for any reason). + +Now consider the the following case, CQs take longer to execute than the `lease-duration`, so when the lease expires, +~1 second later another data node requests and is granted the lease. The original holder of the lease is busily working +on sequentially executing the list of CQs it was originally handed and the data node now holding the lease begins +executing CQs from the top of the list. + +Based on this scenario, it may appear that CQs are “executing in parallel” because multiple data nodes are +essentially “rolling” sequentially through the registered CQs and the lease is rolling from node to node. +The “long pole” here is effectively your most complex CQ – and it likely means that at some point all nodes +are attempting to execute that same complex CQ (and likely competing for resources as they overwrite points +generated by that query on each node that is executing it --- likely with some phased offset). + +To avoid this behavior, and this is desirable because it reduces the overall load on your cluster, +you should set the lease-duration to a value greater than the aggregate execution time for ALL the CQs that you are running. + +Based on the current way in which CQs are configured to execute, the way to address parallelism is by using +Kapacitor for the more complex CQs that you are attempting to run. +[See Kapacitor as a continuous query engine](/{{< latest "kapacitor" >}}/guides/continuous_queries/). +However, you can keep the more simplistic and highly performant CQs within the database – +but ensure that the lease duration is greater than their aggregate execution time to ensure that +“extra” load is not being unnecessarily introduced on your cluster. + + +## PProf endpoints + +Meta nodes expose the `/debug/pprof` endpoints for profiling and troubleshooting. + +## Shard movement + +* [Copy shard](/enterprise_influxdb/v1.9/tools/influxd-ctl/#copy-shard) support - copy a shard from one node to another +* [Copy shard status](/enterprise_influxdb/v1.9/tools/influxd-ctl/#copy-shard-status) - query the status of a copy shard request +* [Kill copy shard](/enterprise_influxdb/v1.9/tools/influxd-ctl/#kill-copy-shard) - kill a running shard copy +* [Remove shard](/enterprise_influxdb/v1.9/tools/influxd-ctl/#remove-shard) - remove a shard from a node (this deletes data) +* [Truncate shards](/enterprise_influxdb/v1.9/tools/influxd-ctl/#truncate-shards) - truncate all active shard groups and start new shards immediately (This is useful when adding nodes or changing replication factors.) + +This functionality is exposed via an API on the meta service and through [`influxd-ctl` sub-commands](/enterprise_influxdb/v1.9/tools/influxd-ctl/). + +## OSS conversion + +Importing a OSS single server as the first data node is supported. + +See [OSS to cluster migration](/enterprise_influxdb/v1.9/guides/migration/) for +step-by-step instructions. + +## Query routing + +The query engine skips failed nodes that hold a shard needed for queries. +If there is a replica on another node, it will retry on that node. + +## Backup and restore + +InfluxDB Enterprise clusters support backup and restore functionality starting with +version 0.7.1. +See [Backup and restore](/enterprise_influxdb/v1.9/administration/backup-and-restore/) for +more information. diff --git a/content/enterprise_influxdb/v1.9/features/users.md b/content/enterprise_influxdb/v1.9/features/users.md new file mode 100644 index 000000000..587f4470b --- /dev/null +++ b/content/enterprise_influxdb/v1.9/features/users.md @@ -0,0 +1,197 @@ +--- +title: InfluxDB Enterprise users +description: Overview of users in InfluxDB Enterprise. +aliases: + - /enterprise/v1.8/features/users/ +menu: + enterprise_influxdb_1_9: + weight: 0 + parent: Enterprise features +--- + +InfluxDB Enterprise users have functions that are either specific to the web +console or specific to the cluster: + +``` +Users Cluster Permissions + +Penelope + O + \|/ + | ----------------------> Dev Account --------> Manage Queries + / \ --------> Monitor + --------> Add/Remove Nodes +Jim + O + \|/ + | ----------------------> Marketing Account ---> View Admin + / \ ---> Graph Role ---> Read + ---> View Chronograf +``` + +## Cluster user information + +In the cluster, individual users are assigned to an account. +Cluster accounts have permissions and roles. + +In the diagram above, Penelope is assigned to the Dev Account and +Jim is assigned to the Marketing Account. +The Dev Account includes the permissions to manage queries, monitor the +cluster, and add/remove nodes from the cluster. +The Marketing Account includes the permission to view and edit the admin screens +as well as the Graph Role which contains the permissions to read data and +view Chronograf. + +### Roles + +Roles are groups of permissions. +A single role can belong to several cluster accounts. + +InfluxDB Enterprise clusters have two built-in roles: + +#### Global Admin + +The Global Admin role has all 16 [cluster permissions](#permissions). + +#### Admin + +The Admin role has all [cluster permissions](#permissions) except for the +permissions to: + +* Add/Remove Nodes +* Copy Shard +* Manage Shards +* Rebalance + +### Permissions + +InfluxDB Enterprise clusters have 16 permissions: + +#### View Admin + +Permission to view or edit admin screens. + +#### View Chronograf + +Permission to use Chronograf tools. + +#### Create Databases + +Permission to create databases. + +#### Create Users & Roles + +Permission to create users and roles. + +#### Add/Remove nodes + +Permission to add/remove nodes from a cluster. + +#### Drop Databases + +Permission to drop databases. + +#### Drop Data + +Permission to drop measurements and series. + +#### Read + +Permission to read data. + +#### Write + +Permission to write data. + +#### Rebalance + +Permission to rebalance a cluster. + +#### Manage Shards + +Permission to copy and delete shards. + +#### Manage continuous queries + +Permission to create, show, and drop continuous queries. + +#### Manage Queries + +Permission to show and kill queries. + +#### Manage Subscriptions + +Permission to show, add, and drop subscriptions. + +#### Monitor + +Permission to show stats and diagnostics. + +#### Copy Shard + +Permission to copy shards. + +### Permission to Statement + +The following table describes permissions required to execute the associated database statement. It also describes whether these permissions apply just to InfluxDB (Database) or InfluxDB Enterprise (Cluster). + +|Permission|Statement| +|---|---| +|CreateDatabasePermission|AlterRetentionPolicyStatement, CreateDatabaseStatement, CreateRetentionPolicyStatement, ShowRetentionPoliciesStatement| +|ManageContinuousQueryPermission|CreateContinuousQueryStatement, DropContinuousQueryStatement, ShowContinuousQueriesStatement| +|ManageSubscriptionPermission|CreateSubscriptionStatement, DropSubscriptionStatement, ShowSubscriptionsStatement| +|CreateUserAndRolePermission|CreateUserStatement, DropUserStatement, GrantAdminStatement, GrantStatement, RevokeAdminStatement, RevokeStatement, SetPasswordUserStatement, ShowGrantsForUserStatement, ShowUsersStatement| +|DropDataPermission|DeleteSeriesStatement, DeleteStatement, DropMeasurementStatement, DropSeriesStatement| +|DropDatabasePermission|DropDatabaseStatement, DropRetentionPolicyStatement| +|ManageShardPermission|DropShardStatement,ShowShardGroupsStatement, ShowShardsStatement| +|ManageQueryPermission|KillQueryStatement, ShowQueriesStatement| +|MonitorPermission|ShowDiagnosticsStatement, ShowStatsStatement| +|ReadDataPermission|ShowFieldKeysStatement, ShowMeasurementsStatement, ShowSeriesStatement, ShowTagKeysStatement, ShowTagValuesStatement, ShowRetentionPoliciesStatement| +|NoPermissions|ShowDatabasesStatement| +|Determined by type of select statement|SelectStatement| + +### Statement to Permission + +The following table describes database statements and the permissions required to execute them. It also describes whether these permissions apply just to InfluxDB (Database) or InfluxDB Enterprise (Cluster). + +|Statment|Permissions|Scope| +|---|---|---| +|AlterRetentionPolicyStatement|CreateDatabasePermission|Database| +|CreateContinuousQueryStatement|ManageContinuousQueryPermission|Database| +|CreateDatabaseStatement|CreateDatabasePermission|Cluster| +|CreateRetentionPolicyStatement|CreateDatabasePermission|Database| +|CreateSubscriptionStatement|ManageSubscriptionPermission|Database| +|CreateUserStatement|CreateUserAndRolePermission|Database| +|DeleteSeriesStatement|DropDataPermission|Database| +|DeleteStatement|DropDataPermission|Database| +|DropContinuousQueryStatement|ManageContinuousQueryPermission|Database| +|DropDatabaseStatement|DropDatabasePermission|Cluster| +|DropMeasurementStatement|DropDataPermission|Database| +|DropRetentionPolicyStatement|DropDatabasePermission|Database| +|DropSeriesStatement|DropDataPermission|Database| +|DropShardStatement|ManageShardPermission|Cluster| +|DropSubscriptionStatement|ManageSubscriptionPermission|Database| +|DropUserStatement|CreateUserAndRolePermission|Database| +|GrantAdminStatement|CreateUserAndRolePermission|Database| +|GrantStatement|CreateUserAndRolePermission|Database| +|KillQueryStatement|ManageQueryPermission|Database| +|RevokeAdminStatement|CreateUserAndRolePermission|Database| +|RevokeStatement|CreateUserAndRolePermission|Database| +|SelectStatement|Determined by type of select statement|n/a| +|SetPasswordUserStatement|CreateUserAndRolePermission|Database| +|ShowContinuousQueriesStatement|ManageContinuousQueryPermission|Database| +|ShowDatabasesStatement|NoPermissions|Cluster|The user's grants determine which databases are returned in the results.| +|ShowDiagnosticsStatement|MonitorPermission|Database| +|ShowFieldKeysStatement|ReadDataPermission|Database| +|ShowGrantsForUserStatement|CreateUserAndRolePermission|Database| +|ShowMeasurementsStatement|ReadDataPermission|Database| +|ShowQueriesStatement|ManageQueryPermission|Database| +|ShowRetentionPoliciesStatement|CreateDatabasePermission|Database| +|ShowSeriesStatement|ReadDataPermission|Database| +|ShowShardGroupsStatement|ManageShardPermission|Cluster| +|ShowShardsStatement|ManageShardPermission|Cluster| +|ShowStatsStatement|MonitorPermission|Database| +|ShowSubscriptionsStatement|ManageSubscriptionPermission|Database| +|ShowTagKeysStatement|ReadDataPermission|Database| +|ShowTagValuesStatement|ReadDataPermission|Database| +|ShowUsersStatement|CreateUserAndRolePermission|Database| diff --git a/content/enterprise_influxdb/v1.9/flux/_index.md b/content/enterprise_influxdb/v1.9/flux/_index.md new file mode 100644 index 000000000..2c74b958f --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/_index.md @@ -0,0 +1,38 @@ +--- +title: Flux data scripting language +description: > + Flux is a functional data scripting language designed for querying, analyzing, and acting on time series data. +menu: + enterprise_influxdb_1_9: + name: Flux + weight: 71 +v2: /influxdb/v2.0/query-data/get-started/ +--- + +Flux is a functional data scripting language designed for querying, analyzing, and acting on time series data. +Its takes the power of [InfluxQL](/enterprise_influxdb/v1.9/query_language/spec/) and the functionality of [TICKscript](/{{< latest "kapacitor" >}}/tick/introduction/) and combines them into a single, unified syntax. + +> Flux v0.65 is production-ready and included with [InfluxDB v1.8](/enterprise_influxdb/v1.9). +> The InfluxDB v1.8 implementation of Flux is read-only and does not support +> writing data back to InfluxDB. + +## Flux design principles +Flux is designed to be usable, readable, flexible, composable, testable, contributable, and shareable. +Its syntax is largely inspired by [2018's most popular scripting language](https://insights.stackoverflow.com/survey/2018#technology), +Javascript, and takes a functional approach to data exploration and processing. + +The following example illustrates pulling data from a bucket (similar to an InfluxQL database) for the last five minutes, +filtering that data by the `cpu` measurement and the `cpu=cpu-total` tag, windowing the data in 1 minute intervals, +and calculating the average of each window: + +```js +from(bucket:"telegraf/autogen") + |> range(start:-1h) + |> filter(fn:(r) => + r._measurement == "cpu" and + r.cpu == "cpu-total" + ) + |> aggregateWindow(every: 1m, fn: mean) +``` + +{{< children >}} diff --git a/content/enterprise_influxdb/v1.9/flux/flux-vs-influxql.md b/content/enterprise_influxdb/v1.9/flux/flux-vs-influxql.md new file mode 100644 index 000000000..e03b7f140 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/flux-vs-influxql.md @@ -0,0 +1,419 @@ +--- +title: Flux vs InfluxQL +description: +menu: + enterprise_influxdb_1_9: + name: Flux vs InfluxQL + parent: Flux + weight: 5 +--- + +Flux is an alternative to [InfluxQL](/enterprise_influxdb/v1.9/query_language/) and other SQL-like query languages for querying and analyzing data. +Flux uses functional language patterns making it incredibly powerful, flexible, and able to overcome many of the limitations of InfluxQL. +This article outlines many of the tasks possible with Flux but not InfluxQL and provides information about Flux and InfluxQL parity. + +- [Possible with Flux](#possible-with-flux) +- [InfluxQL and Flux parity](#influxql-and-flux-parity) + +## Possible with Flux + +- [Joins](#joins) +- [Math across measurements](#math-across-measurements) +- [Sort by tags](#sort-by-tags) +- [Group by any column](#group-by-any-column) +- [Window by calendar months and years](#window-by-calendar-months-and-years) +- [Work with multiple data sources](#work-with-multiple-data-sources) +- [DatePart-like queries](#datepart-like-queries) +- [Pivot](#pivot) +- [Histograms](#histograms) +- [Covariance](#covariance) +- [Cast booleans to integers](#cast-booleans-to-integers) +- [String manipulation and data shaping](#string-manipulation-and-data-shaping) +- [Work with geo-temporal data](#work-with-geo-temporal-data) + +### Joins +InfluxQL has never supported joins. They can be accomplished using [TICKscript](/{{< latest "kapacitor" >}}/tick/introduction/), +but even TICKscript's join capabilities are limited. +Flux's [`join()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/join/) allows you +to join data **from any bucket, any measurement, and on any columns** as long as +each data set includes the columns on which they are to be joined. +This opens the door for really powerful and useful operations. + +```js +dataStream1 = from(bucket: "bucket1") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "network" and + r._field == "bytes-transferred" + ) + +dataStream2 = from(bucket: "bucket1") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "httpd" and + r._field == "requests-per-sec" + ) + +join( + tables: {d1:dataStream1, d2:dataStream2}, + on: ["_time", "_stop", "_start", "host"] + ) +``` + + +--- + +_For an in-depth walkthrough of using the `join()` function, see [How to join data with Flux](/enterprise_influxdb/v1.9/flux/guides/join)._ + +--- + +### Math across measurements +Being able to perform cross-measurement joins also allows you to run calculations using +data from separate measurements – a highly requested feature from the InfluxData community. +The example below takes two data streams from separate measurements, `mem` and `processes`, +joins them, then calculates the average amount of memory used per running process: + +```js +// Memory used (in bytes) +memUsed = from(bucket: "telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field == "used" + ) + +// Total processes running +procTotal = from(bucket: "telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "processes" and + r._field == "total" + ) + +// Join memory used with total processes and calculate +// the average memory (in MB) used for running processes. +join( + tables: {mem:memUsed, proc:procTotal}, + on: ["_time", "_stop", "_start", "host"] + ) + |> map(fn: (r) => ({ + _time: r._time, + _value: (r._value_mem / r._value_proc) / 1000000 + }) +) +``` + +### Sort by tags +InfluxQL's sorting capabilities are very limited, allowing you only to control the +sort order of `time` using the `ORDER BY time` clause. +Flux's [`sort()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/sort) sorts records based on list of columns. +Depending on the column type, records are sorted lexicographically, numerically, or chronologically. + +```js +from(bucket:"telegraf/autogen") + |> range(start:-12h) + |> filter(fn: (r) => + r._measurement == "system" and + r._field == "uptime" + ) + |> sort(columns:["region", "host", "_value"]) +``` + +### Group by any column +InfluxQL lets you group by tags or by time intervals, but nothing else. +Flux lets you group by any column in the dataset, including `_value`. +Use the Flux [`group()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/group/) +to define which columns to group data by. + +```js +from(bucket:"telegraf/autogen") + |> range(start:-12h) + |> filter(fn: (r) => r._measurement == "system" and r._field == "uptime" ) + |> group(columns:["host", "_value"]) +``` + +### Window by calendar months and years +InfluxQL does not support windowing data by calendar months and years due to their varied lengths. +Flux supports calendar month and year duration units (`1mo`, `1y`) and lets you +window and aggregate data by calendar month and year. + +```js +from(bucket:"telegraf/autogen") + |> range(start:-1y) + |> filter(fn: (r) => r._measurement == "mem" and r._field == "used_percent" ) + |> aggregateWindow(every: 1mo, fn: mean) +``` + +### Work with multiple data sources +InfluxQL can only query data stored in InfluxDB. +Flux can query data from other data sources such as CSV, PostgreSQL, MySQL, Google BigTable, and more. +Join that data with data in InfluxDB to enrich query results. + +- [Flux CSV package](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/csv/) +- [Flux SQL package](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/sql/) +- [Flux BigTable package](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/bigtable/) + + +```js +import "csv" +import "sql" + +csvData = csv.from(csv: rawCSV) +sqlData = sql.from( + driverName: "postgres", + dataSourceName: "postgresql://user:password@localhost", + query:"SELECT * FROM example_table" +) +data = from(bucket: "telegraf/autogen") + |> range(start: -24h) + |> filter(fn: (r) => r._measurement == "sensor") + +auxData = join(tables: {csv: csvData, sql: sqlData}, on: ["sensor_id"]) +enrichedData = join(tables: {data: data, aux: auxData}, on: ["sensor_id"]) + +enrichedData + |> yield(name: "enriched_data") +``` + +--- + +_For an in-depth walkthrough of querying SQL data, see [Query SQL data sources](/enterprise_influxdb/v1.9/flux/guides/sql)._ + +--- + +### DatePart-like queries +InfluxQL doesn't support DatePart-like queries that only return results during specified hours of the day. +The Flux [`hourSelection` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/hourselection/) +returns only data with time values in a specified hour range. + +```js +from(bucket: "telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "cpu" and + r.cpu == "cpu-total" + ) + |> hourSelection(start: 9, stop: 17) +``` + +### Pivot +Pivoting data tables has never been supported in InfluxQL. +The Flux [`pivot()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/pivot) provides the ability +to pivot data tables by specifying `rowKey`, `columnKey`, and `valueColumn` parameters. + +```js +from(bucket: "telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "cpu" and + r.cpu == "cpu-total" + ) + |> pivot( + rowKey:["_time"], + columnKey: ["_field"], + valueColumn: "_value" + ) +``` + +### Histograms +The ability to generate histograms has been a highly requested feature for InfluxQL, but has never been supported. +Flux's [`histogram()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/histogram) uses input +data to generate a cumulative histogram with support for other histogram types coming in the future. + +```js +from(bucket: "telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field == "used_percent" + ) + |> histogram( + buckets: [10, 20, 30, 40, 50, 60, 70, 80, 90, 100] + ) +``` + +--- + +_For an example of using Flux to create a cumulative histogram, see [Create histograms](/enterprise_influxdb/v1.9/flux/guides/histograms)._ + +--- + +### Covariance +Flux provides functions for simple covariance calculation. +The [`covariance()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/covariance) +calculates the covariance between two columns and the [`cov()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/cov) +calculates the covariance between two data streams. + +###### Covariance between two columns +```js +from(bucket: "telegraf/autogen") + |> range(start:-5m) + |> covariance(columns: ["x", "y"]) +``` + +###### Covariance between two streams of data +```js +table1 = from(bucket: "telegraf/autogen") + |> range(start: -15m) + |> filter(fn: (r) => + r._measurement == "measurement_1" + ) + +table2 = from(bucket: "telegraf/autogen") + |> range(start: -15m) + |> filter(fn: (r) => + r._measurement == "measurement_2" + ) + +cov(x: table1, y: table2, on: ["_time", "_field"]) +``` + +### Cast booleans to integers +InfluxQL supports type casting, but only for numeric data types (floats to integers and vice versa). +[Flux type conversion functions](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/type-conversions/) +provide much broader support for type conversions and let you perform some long-requested +operations like casting a boolean values to integers. + +##### Cast boolean field values to integers +```js +from(bucket: "telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "m" and + r._field == "bool_field" + ) + |> toInt() +``` + +### String manipulation and data shaping +InfluxQL doesn't support string manipulation when querying data. +The [Flux Strings package](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/strings/) is a collection of functions that operate on string data. +When combined with the [`map()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/map/), +functions in the string package allow for operations like string sanitization and normalization. + +```js +import "strings" + +from(bucket: "telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "weather" and + r._field == "temp" + ) + |> map(fn: (r) => ({ + r with + location: strings.toTitle(v: r.location), + sensor: strings.replaceAll(v: r.sensor, t: " ", u: "-"), + status: strings.substring(v: r.status, start: 0, end: 8) + })) +``` + +### Work with geo-temporal data +InfluxQL doesn't provide functionality for working with geo-temporal data. +The [Flux Geo package](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/geo/) is a collection of functions that +let you shape, filter, and group geo-temporal data. + +```js +import "experimental/geo" + +from(bucket: "geo/autogen") + |> range(start: -1w) + |> filter(fn: (r) => r._measurement == "taxi") + |> geo.shapeData(latField: "latitude", lonField: "longitude", level: 20) + |> geo.filterRows( + region: {lat: 40.69335938, lon: -73.30078125, radius: 20.0}, + strict: true + ) + |> geo.asTracks(groupBy: ["fare-id"]) +``` + + +## InfluxQL and Flux parity +Flux is working towards complete parity with InfluxQL and new functions are being added to that end. +The table below shows InfluxQL statements, clauses, and functions along with their equivalent Flux functions. + +_For a complete list of Flux functions, [view all Flux functions](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/all-functions)._ + +### InfluxQL and Flux parity + +| InfluxQL | Flux Functions | +| -------- | -------------- | +| [SELECT](/enterprise_influxdb/v1.9/query_language/explore-data/#the-basic-select-statement) | [filter()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/filter/) | +| [WHERE](/enterprise_influxdb/v1.9/query_language/explore-data/#the-where-clause) | [filter()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/filter/), [range()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/range/) | +| [GROUP BY](/enterprise_influxdb/v1.9/query_language/explore-data/#the-group-by-clause) | [group()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/group/) | +| [INTO](/enterprise_influxdb/v1.9/query_language/explore-data/#the-into-clause) | [to()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/outputs/to/) * | +| [ORDER BY](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc) | [sort()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/sort/) | +| [LIMIT](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-clause) | [limit()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/limit/) | +| [SLIMIT](/enterprise_influxdb/v1.9/query_language/explore-data/#the-slimit-clause) | -- | +| [OFFSET](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-clause) | -- | +| [SOFFSET](/enterprise_influxdb/v1.9/query_language/explore-data/#the-soffset-clause) | -- | +| [SHOW DATABASES](/enterprise_influxdb/v1.9/query_language/explore-schema/#show-databases) | [buckets()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/inputs/buckets/) | +| [SHOW MEASUREMENTS](/enterprise_influxdb/v1.9/query_language/explore-schema/#show-measurements) | [v1.measurements](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/influxdb-v1/measurements) | +| [SHOW FIELD KEYS](/enterprise_influxdb/v1.9/query_language/explore-schema/#show-field-keys) | [keys()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/keys/) | +| [SHOW RETENTION POLICIES](/enterprise_influxdb/v1.9/query_language/explore-schema/#show-retention-policies) | [buckets()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/inputs/buckets/) | +| [SHOW TAG KEYS](/enterprise_influxdb/v1.9/query_language/explore-schema/#show-tag-keys) | [v1.tagKeys()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/influxdb-v1/tagkeys), [v1.measurementTagKeys()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/influxdb-v1/measurementtagkeys) | +| [SHOW TAG VALUES](/enterprise_influxdb/v1.9/query_language/explore-schema/#show-tag-values) | [v1.tagValues()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/influxdb-v1/tagvalues), [v1.measurementTagValues()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/influxdb-v1/measurementtagvalues) | +| [SHOW SERIES](/enterprise_influxdb/v1.9/query_language/explore-schema/#show-series) | -- | +| [CREATE DATABASE](/enterprise_influxdb/v1.9/query_language/manage-database/#create-database) | -- | +| [DROP DATABASE](/enterprise_influxdb/v1.9/query_language/manage-database/#delete-a-database-with-drop-database) | -- | +| [DROP SERIES](/enterprise_influxdb/v1.9/query_language/manage-database/#drop-series-from-the-index-with-drop-series) | -- | +| [DELETE](/enterprise_influxdb/v1.9/query_language/manage-database/#delete-series-with-delete) | -- | +| [DROP MEASUREMENT](/enterprise_influxdb/v1.9/query_language/manage-database/#delete-measurements-with-drop-measurement) | -- | +| [DROP SHARD](/enterprise_influxdb/v1.9/query_language/manage-database/#delete-a-shard-with-drop-shard) | -- | +| [CREATE RETENTION POLICY](/enterprise_influxdb/v1.9/query_language/manage-database/#create-retention-policies-with-create-retention-policy) | -- | +| [ALTER RETENTION POLICY](/enterprise_influxdb/v1.9/query_language/manage-database/#modify-retention-policies-with-alter-retention-policy) | -- | +| [DROP RETENTION POLICY](/enterprise_influxdb/v1.9/query_language/manage-database/#delete-retention-policies-with-drop-retention-policy) | -- | +| [COUNT](/enterprise_influxdb/v1.9/query_language/functions#count) | [count()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/count/) | +| [DISTINCT](/enterprise_influxdb/v1.9/query_language/functions#distinct) | [distinct()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/distinct/) | +| [INTEGRAL](/enterprise_influxdb/v1.9/query_language/functions#integral) | [integral()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/integral/) | +| [MEAN](/enterprise_influxdb/v1.9/query_language/functions#mean) | [mean()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/mean/) | +| [MEDIAN](/enterprise_influxdb/v1.9/query_language/functions#median) | [median()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/median/) | +| [MODE](/enterprise_influxdb/v1.9/query_language/functions#mode) | [mode()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/mode/) | +| [SPREAD](/enterprise_influxdb/v1.9/query_language/functions#spread) | [spread()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/spread/) | +| [STDDEV](/enterprise_influxdb/v1.9/query_language/functions#stddev) | [stddev()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/stddev/) | +| [SUM](/enterprise_influxdb/v1.9/query_language/functions#sum) | [sum()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/sum/) | +| [BOTTOM](/enterprise_influxdb/v1.9/query_language/functions#bottom) | [bottom()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/bottom/) | +| [FIRST](/enterprise_influxdb/v1.9/query_language/functions#first) | [first()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/first/) | +| [LAST](/enterprise_influxdb/v1.9/query_language/functions#last) | [last()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/last/) | +| [MAX](/enterprise_influxdb/v1.9/query_language/functions#max) | [max()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/max/) | +| [MIN](/enterprise_influxdb/v1.9/query_language/functions#min) | [min()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/min/) | +| [PERCENTILE](/enterprise_influxdb/v1.9/query_language/functions#percentile) | [quantile()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/quantile/) | +| [SAMPLE](/enterprise_influxdb/v1.9/query_language/functions#sample) | [sample()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/sample/) | +| [TOP](/enterprise_influxdb/v1.9/query_language/functions#top) | [top()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/top/) | +| [ABS](/enterprise_influxdb/v1.9/query_language/functions#abs) | [math.abs()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/abs/) | +| [ACOS](/enterprise_influxdb/v1.9/query_language/functions#acos) | [math.acos()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/acos/) | +| [ASIN](/enterprise_influxdb/v1.9/query_language/functions#asin) | [math.asin()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/asin/) | +| [ATAN](/enterprise_influxdb/v1.9/query_language/functions#atan) | [math.atan()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/atan/) | +| [ATAN2](/enterprise_influxdb/v1.9/query_language/functions#atan2) | [math.atan2()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/atan2/) | +| [CEIL](/enterprise_influxdb/v1.9/query_language/functions#ceil) | [math.ceil()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/ceil/) | +| [COS](/enterprise_influxdb/v1.9/query_language/functions#cos) | [math.cos()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/cos/) | +| [CUMULATIVE_SUM](/enterprise_influxdb/v1.9/query_language/functions#cumulative-sum) | [cumulativeSum()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/cumulativesum/) | +| [DERIVATIVE](/enterprise_influxdb/v1.9/query_language/functions#derivative) | [derivative()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/derivative/) | +| [DIFFERENCE](/enterprise_influxdb/v1.9/query_language/functions#difference) | [difference()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/difference/) | +| [ELAPSED](/enterprise_influxdb/v1.9/query_language/functions#elapsed) | [elapsed()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/elapsed/) | +| [EXP](/enterprise_influxdb/v1.9/query_language/functions#exp) | [math.exp()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/exp/) | +| [FLOOR](/enterprise_influxdb/v1.9/query_language/functions#floor) | [math.floor()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/floor/) | +| [HISTOGRAM](/enterprise_influxdb/v1.9/query_language/functions#histogram) | [histogram()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/histogram/) | +| [LN](/enterprise_influxdb/v1.9/query_language/functions#ln) | [math.log()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/log/) | +| [LOG](/enterprise_influxdb/v1.9/query_language/functions#log) | [math.logb()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/logb/) | +| [LOG2](/enterprise_influxdb/v1.9/query_language/functions#log2) | [math.log2()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/log2/) | +| [LOG10](/enterprise_influxdb/v1.9/query_language/functions/#log10) | [math.log10()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/log10/) | +| [MOVING_AVERAGE](/enterprise_influxdb/v1.9/query_language/functions#moving-average) | [movingAverage()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/movingaverage/) | +| [NON_NEGATIVE_DERIVATIVE](/enterprise_influxdb/v1.9/query_language/functions#non-negative-derivative) | [derivative(nonNegative:true)](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/derivative/) | +| [NON_NEGATIVE_DIFFERENCE](/enterprise_influxdb/v1.9/query_language/functions#non-negative-difference) | [difference(nonNegative:true)](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/derivative/) | +| [POW](/enterprise_influxdb/v1.9/query_language/functions#pow) | [math.pow()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/pow/) | +| [ROUND](/enterprise_influxdb/v1.9/query_language/functions#round) | [math.round()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/round/) | +| [SIN](/enterprise_influxdb/v1.9/query_language/functions#sin) | [math.sin()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/sin/) | +| [SQRT](/enterprise_influxdb/v1.9/query_language/functions#sqrt) | [math.sqrt()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/sqrt/) | +| [TAN](/enterprise_influxdb/v1.9/query_language/functions#tan) | [math.tan()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/math/tan/) | +| [HOLT_WINTERS](/enterprise_influxdb/v1.9/query_language/functions#holt-winters) | [holtWinters()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/holtwinters/) | +| [CHANDE_MOMENTUM_OSCILLATOR](/enterprise_influxdb/v1.9/query_language/functions#chande-momentum-oscillator) | [chandeMomentumOscillator()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/chandemomentumoscillator/) | +| [EXPONENTIAL_MOVING_AVERAGE](/enterprise_influxdb/v1.9/query_language/functions#exponential-moving-average) | [exponentialMovingAverage()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/exponentialmovingaverage/) | +| [DOUBLE_EXPONENTIAL_MOVING_AVERAGE](/enterprise_influxdb/v1.9/query_language/functions#double-exponential-moving-average) | [doubleEMA()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/doubleema/) | +| [KAUFMANS_EFFICIENCY_RATIO](/enterprise_influxdb/v1.9/query_language/functions#kaufmans-efficiency-ratio) | [kaufmansER()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/kaufmanser/) | +| [KAUFMANS_ADAPTIVE_MOVING_AVERAGE](/enterprise_influxdb/v1.9/query_language/functions#kaufmans-adaptive-moving-average) | [kaufmansAMA()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/kaufmansama/) | +| [TRIPLE_EXPONENTIAL_MOVING_AVERAGE](/enterprise_influxdb/v1.9/query_language/functions#triple-exponential-moving-average) | [tripleEMA()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/tripleema/) | +| [TRIPLE_EXPONENTIAL_DERIVATIVE](/enterprise_influxdb/v1.9/query_language/functions#triple-exponential-derivative) | [tripleExponentialDerivative()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/tripleexponentialderivative/) | +| [RELATIVE_STRENGTH_INDEX](/enterprise_influxdb/v1.9/query_language/functions#relative-strength-index) | [relativeStrengthIndex()](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/relativestrengthindex/) | + +_* The to() function only writes to InfluxDB 2.0._ diff --git a/content/enterprise_influxdb/v1.9/flux/get-started/_index.md b/content/enterprise_influxdb/v1.9/flux/get-started/_index.md new file mode 100644 index 000000000..4428e6420 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/get-started/_index.md @@ -0,0 +1,115 @@ +--- +title: Get started with Flux +description: > + Get started with Flux, InfluxData's new functional data scripting language. + This step-by-step guide will walk you through the basics and get you on your way. +menu: + enterprise_influxdb_1_9: + name: Get started with Flux + identifier: get-started + parent: Flux + weight: 2 +aliases: + - /enterprise_influxdb/v1.9/flux/getting-started/ + - /enterprise_influxdb/v1.9/flux/introduction/getting-started/ +canonical: /{{< latest "influxdb" "v2" >}}/query-data/get-started/ +v2: /influxdb/v2.0/query-data/get-started/ +--- + +Flux is InfluxData's new functional data scripting language designed for querying, +analyzing, and acting on data. + +This multi-part getting started guide walks through important concepts related to Flux. +It covers querying time series data from InfluxDB using Flux, and introduces Flux syntax and functions. + +## What you will need + +##### InfluxDB v1.8+ +Flux v0.65 is built into InfluxDB v1.8 and can be used to query data stored in InfluxDB. + +--- + +_For information about downloading and installing InfluxDB, see [InfluxDB installation](/enterprise_influxdb/v1.9/introduction/installation)._ + +--- + +##### Chronograf v1.8+ +**Not required but strongly recommended**. +Chronograf v1.8's Data Explorer provides a user interface (UI) for writing Flux scripts and visualizing results. +Dashboards in Chronograf v1.8+ also support Flux queries. + +--- + +_For information about downloading and installing Chronograf, see [Chronograf installation](/{{< latest "chronograf" >}}/introduction/installation)._ + +--- + +## Key concepts +Flux introduces important new concepts you should understand as you get started. + +### Buckets +Flux introduces "buckets," a new data storage concept for InfluxDB. +A **bucket** is a named location where data is stored that has a retention policy. +It's similar to an InfluxDB v1.x "database," but is a combination of both a database and a retention policy. +When using multiple retention policies, each retention policy is treated as is its own bucket. + +Flux's [`from()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/inputs/from), which defines an InfluxDB data source, requires a `bucket` parameter. +When using Flux with InfluxDB v1.x, use the following bucket naming convention which combines +the database name and retention policy into a single bucket name: + +###### InfluxDB v1.x bucket naming convention +```js +// Pattern +from(bucket:"/") + +// Example +from(bucket:"telegraf/autogen") +``` + +### Pipe-forward operator +Flux uses pipe-forward operators (`|>`) extensively to chain operations together. +After each function or operation, Flux returns a table or collection of tables containing data. +The pipe-forward operator pipes those tables into the next function or operation where +they are further processed or manipulated. + +### Tables +Flux structures all data in tables. +When data is streamed from data sources, Flux formats it as annotated comma-separated values (CSV), representing tables. +Functions then manipulate or process them and output new tables. +This makes it easy to chain together functions to build sophisticated queries. + +#### Group keys +Every table has a **group key** which describes the contents of the table. +It's a list of columns for which every row in the table will have the same value. +Columns with unique values in each row are **not** part of the group key. + +As functions process and transform data, each modifies the group keys of output tables. +Understanding how tables and group keys are modified by functions is key to properly +shaping your data for the desired output. + +###### Example group key +```js +[_start, _stop, _field, _measurement, host] +``` + +Note that `_time` and `_value` are excluded from the example group key because they +are unique to each row. + +## Tools for working with Flux + +You have multiple [options for writing and running Flux queries](/enterprise_influxdb/v1.9/flux/guides/execute-queries/), +but as you're getting started, we recommend using the following: + +### Chronograf's Data Explorer +Chronograf's Data Explorer makes it easy to write your first Flux script and visualize the results. +To use Chronograf's Flux UI, open the **Data Explorer** and to the right of the source +dropdown above the graph placeholder, select **Flux** as the source type. + +This will provide **Schema**, **Script**, and **Functions** panes. +The Schema pane allows you to explore your data. +The Script pane is where you write your Flux script. +The Functions pane provides a list of functions available in your Flux queries. + +
        + +
        diff --git a/content/enterprise_influxdb/v1.9/flux/get-started/query-influxdb.md b/content/enterprise_influxdb/v1.9/flux/get-started/query-influxdb.md new file mode 100644 index 000000000..f5fb26a7d --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/get-started/query-influxdb.md @@ -0,0 +1,134 @@ +--- +title: Query InfluxDB with Flux +description: Learn the basics of using Flux to query data from InfluxDB. +menu: + enterprise_influxdb_1_9: + name: Query InfluxDB + parent: get-started + weight: 1 +aliases: + - /enterprise_influxdb/v1.9/flux/getting-started/query-influxdb/ +canonical: /{{< latest "influxdb" "v2" >}}/query-data/get-started/query-influxdb/ +v2: /influxdb/v2.0/query-data/get-started/query-influxdb/ +--- + +This guide walks through the basics of using Flux to query data from InfluxDB. +_**If you haven't already, make sure to install InfluxDB v1.8+, [enable Flux](/enterprise_influxdb/v1.9/flux/installation), +and choose a [tool for writing Flux queries](/enterprise_influxdb/v1.9/flux/get-started#tools-for-working-with-flux).**_ + +Every Flux query needs the following: + +1. [A data source](#1-define-your-data-source) +2. [A time range](#2-specify-a-time-range) +3. [Data filters](#3-filter-your-data) + + +## 1. Define your data source +Flux's [`from()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/inputs/from) function defines an InfluxDB data source. +It requires a [`bucket`](/enterprise_influxdb/v1.9/flux/get-started/#buckets) parameter. +For this example, use `telegraf/autogen`, a combination of the default database and retention policy provided by the TICK stack. + +```js +from(bucket:"telegraf/autogen") +``` + +## 2. Specify a time range +Flux requires a time range when querying time series data. +"Unbounded" queries are very resource-intensive and as a protective measure, +Flux will not query the database without a specified range. + +Use the pipe-forward operator (`|>`) to pipe data from your data source into the [`range()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/range) +function, which specifies a time range for your query. +It accepts two properties: `start` and `stop`. +Ranges can be **relative** using negative [durations](/{{< latest "influxdb" "v2" >}}/reference/flux/language/lexical-elements#duration-literals) +or **absolute** using [timestamps](/{{< latest "influxdb" "v2" >}}/reference/flux/language/lexical-elements#date-and-time-literals). + +###### Example relative time ranges +```js +// Relative time range with start only. Stop defaults to now. +from(bucket:"telegraf/autogen") + |> range(start: -1h) + +// Relative time range with start and stop +from(bucket:"telegraf/autogen") + |> range(start: -1h, stop: -10m) +``` + +> Relative ranges are relative to "now." + +###### Example absolute time range +```js +from(bucket:"telegraf/autogen") + |> range(start: 2018-11-05T23:30:00Z, stop: 2018-11-06T00:00:00Z) +``` + +#### Use the following: +For this guide, use the relative time range, `-15m`, to limit query results to data from the last 15 minutes: + +```js +from(bucket:"telegraf/autogen") + |> range(start: -15m) +``` + +## 3. Filter your data +Pass your ranged data into the `filter()` function to narrow results based on data attributes or columns. +The `filter()` function has one parameter, `fn`, which expects an anonymous function +with logic that filters data based on columns or attributes. + +Flux's anonymous function syntax is very similar to Javascript's. +Records or rows are passed into the `filter()` function as a record (`r`). +The anonymous function takes the record and evaluates it to see if it matches the defined filters. +Use the `AND` relational operator to chain multiple filters. + +```js +// Pattern +(r) => (r.recordProperty comparisonOperator comparisonExpression) + +// Example with single filter +(r) => (r._measurement == "cpu") + +// Example with multiple filters +(r) => (r._measurement == "cpu") and (r._field != "usage_system" ) +``` + +#### Use the following: +For this example, filter by the `cpu` measurement, the `usage_system` field, and the `cpu-total` tag value: + +```js +from(bucket:"telegraf/autogen") + |> range(start: -15m) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_system" and + r.cpu == "cpu-total" + ) +``` + +## 4. Yield your queried data +Use Flux's `yield()` function to output the filtered tables as the result of the query. + +```js +from(bucket:"telegraf/autogen") + |> range(start: -15m) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_system" and + r.cpu == "cpu-total" + ) + |> yield() +``` + +> Chronograf and the `influx` CLI automatically assume a `yield()` function at +> the end of each script in order to output and visualize the data. +> Best practice is to include a `yield()` function, but it is not always necessary. + +## Congratulations! +You have now queried data from InfluxDB using Flux. + +The query shown here is a barebones example. +Flux queries can be extended in many ways to form powerful scripts. + +
        + + +
        diff --git a/content/enterprise_influxdb/v1.9/flux/get-started/syntax-basics.md b/content/enterprise_influxdb/v1.9/flux/get-started/syntax-basics.md new file mode 100644 index 000000000..36c983beb --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/get-started/syntax-basics.md @@ -0,0 +1,215 @@ +--- +title: Flux syntax basics +description: An introduction to the basic elements of the Flux syntax with real-world application examples. +menu: + enterprise_influxdb_1_9: + name: Syntax basics + parent: get-started + weight: 3 +aliases: + - /enterprise_influxdb/v1.9/flux/getting-started/syntax-basics/ +canonical: /{{< latest "influxdb" "v2" >}}/query-data/get-started/syntax-basics/ +v2: /influxdb/v2.0/query-data/get-started/syntax-basics/ +--- + + +Flux, at its core, is a scripting language designed specifically for working with data. +This guide walks through a handful of simple expressions and how they are handled in Flux. + +### Simple expressions +Flux is a scripting language that supports basic expressions. +For example, simple addition: + +```js +> 1 + 1 +2 +``` + +### Variables +Assign an expression to a variable using the assignment operator, `=`. + +```js +> s = "this is a string" +> i = 1 // an integer +> f = 2.0 // a floating point number +``` + +Type the name of a variable to print its value: + +```js +> s +this is a string +> i +1 +> f +2 +``` + +### Records +Flux also supports records. Each value in a record can be a different data type. + +```js +> o = {name:"Jim", age: 42, "favorite color": "red"} +``` + +Use **dot notation** to access a properties of a record: + +```js +> o.name +Jim +> o.age +42 +``` + +Or **bracket notation**: + +```js +> o["name"] +Jim +> o["age"] +42 +> o["favorite color"] +red +``` + +{{% note %}} +Use bracket notation to reference record properties with special or +white space characters in the property key. +{{% /note %}} + +### Lists +Flux supports lists. List values must be the same type. + +```js +> n = 4 +> l = [1,2,3,n] +> l +[1, 2, 3, 4] +``` + +### Functions +Flux uses functions for most of its heavy lifting. +Below is a simple function that squares a number, `n`. + +```js +> square = (n) => n * n +> square(n:3) +9 +``` + +> Flux does not support positional arguments or parameters. +> Parameters must always be named when calling a function. + +### Pipe-forward operator +Flux uses the pipe-forward operator (`|>`) extensively to chain operations together. +After each function or operation, Flux returns a table or collection of tables containing data. +The pipe-forward operator pipes those tables into the next function where they are further processed or manipulated. + +```js +data |> someFunction() |> anotherFunction() +``` + +## Real-world application of basic syntax +This likely seems familiar if you've already been through through the other [getting started guides](/enterprise_influxdb/v1.9/flux/get-started). +Flux's syntax is inspired by Javascript and other functional scripting languages. +As you begin to apply these basic principles in real-world use cases such as creating data stream variables, +custom functions, etc., the power of Flux and its ability to query and process data will become apparent. + +The examples below provide both multi-line and single-line versions of each input command. +Carriage returns in Flux aren't necessary, but do help with readability. +Both single- and multi-line commands can be copied and pasted into the `influx` CLI running in Flux mode. + +{{< tabs-wrapper >}} +{{% tabs %}} +[Multi-line inputs](#) +[Single-line inputs](#) +{{% /tabs %}} +{{% tab-content %}} +### Define data stream variables +A common use case for variable assignments in Flux is creating variables for one +or more input data streams. + +```js +timeRange = -1h + +cpuUsageUser = + from(bucket:"telegraf/autogen") + |> range(start: timeRange) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_user" and + r.cpu == "cpu-total" + ) + +memUsagePercent = + from(bucket:"telegraf/autogen") + |> range(start: timeRange) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field == "used_percent" + ) +``` + +These variables can be used in other functions, such as `join()`, while keeping the syntax minimal and flexible. + +### Define custom functions +Create a function that returns the `N` number rows in the input stream with the highest `_value`s. +To do this, pass the input stream (`tables`) and the number of results to return (`n`) into a custom function. +Then using Flux's `sort()` and `limit()` functions to find the top `n` results in the data set. + +```js +topN = (tables=<-, n) => + tables + |> sort(desc: true) + |> limit(n: n) +``` + +_More information about creating custom functions is available in the [Custom functions](/{{< latest "influxdb" "v2" >}}/query-data/flux/custom-functions) documentation._ + +Using this new custom function `topN` and the `cpuUsageUser` data stream variable defined above, +find the top five data points and yield the results. + +```js +cpuUsageUser + |> topN(n:5) + |> yield() +``` +{{% /tab-content %}} + +{{% tab-content %}} +### Define data stream variables +A common use case for variable assignments in Flux is creating variables for multiple filtered input data streams. + +```js +timeRange = -1h +cpuUsageUser = from(bucket:"telegraf/autogen") |> range(start: timeRange) |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_user" and r.cpu == "cpu-total") +memUsagePercent = from(bucket:"telegraf/autogen") |> range(start: timeRange) |> filter(fn: (r) => r._measurement == "mem" and r._field == "used_percent") +``` + +These variables can be used in other functions, such as `join()`, while keeping the syntax minimal and flexible. + +### Define custom functions +Let's create a function that returns the `N` number rows in the input data stream with the highest `_value`s. +To do this, pass the input stream (`tables`) and the number of results to return (`n`) into a custom function. +Then using Flux's `sort()` and `limit()` functions to find the top `n` results in the data set. + +```js +topN = (tables=<-, n) => tables |> sort(desc: true) |> limit(n: n) +``` + +_More information about creating custom functions is available in the [Custom functions](/{{< latest "influxdb" "v2" >}}/query-data/flux/custom-functions) documentation._ + +Using the `cpuUsageUser` data stream variable defined [above](#define-data-stream-variables), +find the top five data points with the custom `topN` function and yield the results. + +```js +cpuUsageUser |> topN(n:5) |> yield() +``` +{{% /tab-content %}} +{{< /tabs-wrapper >}} + +This query will return the five data points with the highest user CPU usage over the last hour. + +
        + +
        diff --git a/content/enterprise_influxdb/v1.9/flux/get-started/transform-data.md b/content/enterprise_influxdb/v1.9/flux/get-started/transform-data.md new file mode 100644 index 000000000..125793532 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/get-started/transform-data.md @@ -0,0 +1,184 @@ +--- +title: Transform data with Flux +description: Learn the basics of using Flux to transform data queried from InfluxDB. +menu: + enterprise_influxdb_1_9: + name: Transform your data + parent: get-started + weight: 2 +aliases: + - /enterprise_influxdb/v1.9/flux/getting-started/transform-data/ +canonical: /{{< latest "influxdb" "v2" >}}/query-data/get-started/transform-data/ +v2: /influxdb/v2.0/query-data/get-started/transform-data/ +--- + +When [querying data from InfluxDB](/enterprise_influxdb/v1.9/flux/get-started/query-influxdb), +you often need to transform that data in some way. +Common examples are aggregating data into averages, downsampling data, etc. + +This guide demonstrates using [Flux functions](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib) to transform your data. +It walks through creating a Flux script that partitions data into windows of time, +averages the `_value`s in each window, and outputs the averages as a new table. + +It's important to understand how the "shape" of your data changes through each of these operations. + +## Query data +Use the query built in the previous [Query data from InfluxDB](/enterprise_influxdb/v1.9/flux/get-started/query-influxdb) +guide, but update the range to pull data from the last hour: + +```js +from(bucket:"telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_system" and + r.cpu == "cpu-total" + ) +``` + +## Flux functions +Flux provides a number of functions that perform specific operations, transformations, and tasks. +You can also [create custom functions](/{{< latest "influxdb" "v2" >}}/query-data/flux/custom-functions) in your Flux queries. +_Functions are covered in detail in the [Flux functions](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib) documentation._ + +A common type of function used when transforming data queried from InfluxDB is an aggregate function. +Aggregate functions take a set of `_value`s in a table, aggregate them, and transform +them into a new value. + +This example uses the [`mean()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/mean) +to average values within time windows. + +> The following example walks through the steps required to window and aggregate data, +> but there is a [`aggregateWindow()` helper function](#helper-functions) that does it for you. +> It's just good to understand the steps in the process. + +## Window your data +Flux's [`window()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/window) partitions records based on a time value. +Use the `every` parameter to define a duration of time for each window. + +{{% note %}} +#### Calendar months and years +`every` supports all [valid duration units](/{{< latest "influxdb" "v2" >}}/reference/flux/language/types/#duration-types), +including **calendar months (`1mo`)** and **years (`1y`)**. +{{% /note %}} + +For this example, window data in five minute intervals (`5m`). + +```js +from(bucket:"telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_system" and + r.cpu == "cpu-total" + ) + |> window(every: 5m) +``` + +As data is gathered into windows of time, each window is output as its own table. +When visualized, each table is assigned a unique color. + +![Windowed data tables](/img/flux/windowed-data.png) + +## Aggregate windowed data +Flux aggregate functions take the `_value`s in each table and aggregate them in some way. +Use the [`mean()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/mean) to average the `_value`s of each table. + +```js +from(bucket:"telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_system" and + r.cpu == "cpu-total" + ) + |> window(every: 5m) + |> mean() +``` + +As rows in each window are aggregated, their output table contains only a single row with the aggregate value. +Windowed tables are all still separate and, when visualized, will appear as single, unconnected points. + +![Windowed aggregate data](/img/flux/windowed-aggregates.png) + +## Add times to your aggregates +As values are aggregated, the resulting tables do not have a `_time` column because +the records used for the aggregation all have different timestamps. +Aggregate functions don't infer what time should be used for the aggregate value. +Therefore the `_time` column is dropped. + +A `_time` column is required in the [next operation](#unwindow-aggregate-tables). +To add one, use the [`duplicate()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/duplicate) +to duplicate the `_stop` column as the `_time` column for each windowed table. + +```js +from(bucket:"telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_system" and + r.cpu == "cpu-total" + ) + |> window(every: 5m) + |> mean() + |> duplicate(column: "_stop", as: "_time") +``` + +## Unwindow aggregate tables + +Use the `window()` function with the `every: inf` parameter to gather all points +into a single, infinite window. + +```js +from(bucket:"telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_system" and + r.cpu == "cpu-total" + ) + |> window(every: 5m) + |> mean() + |> duplicate(column: "_stop", as: "_time") + |> window(every: inf) +``` + +Once ungrouped and combined into a single table, the aggregate data points will appear connected in your visualization. + +![Unwindowed aggregate data](/img/flux/windowed-aggregates-ungrouped.png) + +## Helper functions +This may seem like a lot of coding just to build a query that aggregates data, however going through the +process helps to understand how data changes "shape" as it is passed through each function. + +Flux provides (and allows you to create) "helper" functions that abstract many of these steps. +The same operation performed in this guide can be accomplished using the +[`aggregateWindow()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/aggregatewindow). + +```js +from(bucket:"telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_system" and + r.cpu == "cpu-total" + ) + |> aggregateWindow(every: 5m, fn: mean) +``` + +## Congratulations! +You have now constructed a Flux query that uses Flux functions to transform your data. +There are many more ways to manipulate your data using both Flux's primitive functions +and your own custom functions, but this is a good introduction into the basic syntax and query structure. + +--- + +_For a deeper dive into windowing and aggregating data with example data output for each transformation, +view the [Windowing and aggregating data](/enterprise_influxdb/v1.9/flux/guides/window-aggregate) guide._ + +--- + +
        + + +
        diff --git a/content/enterprise_influxdb/v1.9/flux/guides/_index.md b/content/enterprise_influxdb/v1.9/flux/guides/_index.md new file mode 100644 index 000000000..58127501c --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/_index.md @@ -0,0 +1,40 @@ +--- +title: Query data with Flux +description: Guides that walk through both common and complex queries and use cases for Flux. +weight: 3 +aliases: + - /flux/latest/ + - /flux/latest/introduction +menu: + enterprise_influxdb_1_9: + name: Query with Flux + parent: Flux +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/ +v2: /influxdb/v2.0/query-data/flux/ +--- + +The following guides walk through both common and complex queries and use cases for Flux. + +{{% note %}} +#### Example data variable +Many of the examples provided in the following guides use a `data` variable, +which represents a basic query that filters data by measurement and field. +`data` is defined as: + +```js +data = from(bucket: "db/rp") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "example-measurement" and + r._field == "example-field" + ) +``` +{{% /note %}} + +## Flux query guides + +{{< children type="anchored-list" pages="all" >}} + +--- + +{{< children pages="all" readmore="true" hr="true" >}} diff --git a/content/enterprise_influxdb/v1.9/flux/guides/calculate-percentages.md b/content/enterprise_influxdb/v1.9/flux/guides/calculate-percentages.md new file mode 100644 index 000000000..272c8b179 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/calculate-percentages.md @@ -0,0 +1,210 @@ +--- +title: Calculate percentages with Flux +list_title: Calculate percentages +description: > + Use `pivot()` or `join()` and the `map()` function to align operand values into rows and calculate a percentage. +menu: + enterprise_influxdb_1_9: + name: Calculate percentages + identifier: flux-calc-perc + parent: Query with Flux +weight: 6 +list_query_example: percentages +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/calculate-percentages/ +v2: /influxdb/v2.0/query-data/flux/calculate-percentages/ +--- + +Calculating percentages from queried data is a common use case for time series data. +To calculate a percentage in Flux, operands must be in each row. +Use `map()` to re-map values in the row and calculate a percentage. + +**To calculate percentages** + +1. Use [`from()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/inputs/from/), + [`range()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/range/) and + [`filter()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/filter/) to query operands. +2. Use [`pivot()` or `join()`](/enterprise_influxdb/v1.9/flux/guides/mathematic-operations/#pivot-vs-join) + to align operand values into rows. +3. Use [`map()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/map/) + to divide the numerator operand value by the denominator operand value and multiply by 100. + +{{% note %}} +The following examples use `pivot()` to align operands into rows because +`pivot()` works in most cases and is more performant than `join()`. +_See [Pivot vs join](/enterprise_influxdb/v1.9/flux/guides/mathematic-operations/#pivot-vs-join)._ +{{% /note %}} + +```js +from(bucket: "db/rp") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "m1" and r._field =~ /field[1-2]/ ) + |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value") + |> map(fn: (r) => ({ r with _value: r.field1 / r.field2 * 100.0 })) +``` + +## GPU monitoring example +The following example queries data from the gpu-monitor bucket and calculates the +percentage of GPU memory used over time. +Data includes the following: + +- **`gpu` measurement** +- **`mem_used` field**: used GPU memory in bytes +- **`mem_total` field**: total GPU memory in bytes + +### Query mem_used and mem_total fields +```js +from(bucket: "gpu-monitor") + |> range(start: 2020-01-01T00:00:00Z) + |> filter(fn: (r) => r._measurement == "gpu" and r._field =~ /mem_/) +``` + +###### Returns the following stream of tables: + +| _time | _measurement | _field | _value | +|:----- |:------------:|:------: | ------: | +| 2020-01-01T00:00:00Z | gpu | mem_used | 2517924577 | +| 2020-01-01T00:00:10Z | gpu | mem_used | 2695091978 | +| 2020-01-01T00:00:20Z | gpu | mem_used | 2576980377 | +| 2020-01-01T00:00:30Z | gpu | mem_used | 3006477107 | +| 2020-01-01T00:00:40Z | gpu | mem_used | 3543348019 | +| 2020-01-01T00:00:50Z | gpu | mem_used | 4402341478 | + +

        + +| _time | _measurement | _field | _value | +|:----- |:------------:|:------: | ------: | +| 2020-01-01T00:00:00Z | gpu | mem_total | 8589934592 | +| 2020-01-01T00:00:10Z | gpu | mem_total | 8589934592 | +| 2020-01-01T00:00:20Z | gpu | mem_total | 8589934592 | +| 2020-01-01T00:00:30Z | gpu | mem_total | 8589934592 | +| 2020-01-01T00:00:40Z | gpu | mem_total | 8589934592 | +| 2020-01-01T00:00:50Z | gpu | mem_total | 8589934592 | + +### Pivot fields into columns +Use `pivot()` to pivot the `mem_used` and `mem_total` fields into columns. +Output includes `mem_used` and `mem_total` columns with values for each corresponding `_time`. + +```js +// ... + |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value") +``` + +###### Returns the following: + +| _time | _measurement | mem_used | mem_total | +|:----- |:------------:| --------: | ---------: | +| 2020-01-01T00:00:00Z | gpu | 2517924577 | 8589934592 | +| 2020-01-01T00:00:10Z | gpu | 2695091978 | 8589934592 | +| 2020-01-01T00:00:20Z | gpu | 2576980377 | 8589934592 | +| 2020-01-01T00:00:30Z | gpu | 3006477107 | 8589934592 | +| 2020-01-01T00:00:40Z | gpu | 3543348019 | 8589934592 | +| 2020-01-01T00:00:50Z | gpu | 4402341478 | 8589934592 | + +### Map new values +Each row now contains the values necessary to calculate a percentage. +Use `map()` to re-map values in each row. +Divide `mem_used` by `mem_total` and multiply by 100 to return the percentage. + +{{% note %}} +To return a precise float percentage value that includes decimal points, the example +below casts integer field values to floats and multiplies by a float value (`100.0`). +{{% /note %}} + +```js +// ... + |> map(fn: (r) => ({ + _time: r._time, + _measurement: r._measurement, + _field: "mem_used_percent", + _value: float(v: r.mem_used) / float(v: r.mem_total) * 100.0 + })) +``` +##### Query results: + +| _time | _measurement | _field | _value | +|:----- |:------------:|:------: | ------: | +| 2020-01-01T00:00:00Z | gpu | mem_used_percent | 29.31 | +| 2020-01-01T00:00:10Z | gpu | mem_used_percent | 31.37 | +| 2020-01-01T00:00:20Z | gpu | mem_used_percent | 30.00 | +| 2020-01-01T00:00:30Z | gpu | mem_used_percent | 35.00 | +| 2020-01-01T00:00:40Z | gpu | mem_used_percent | 41.25 | +| 2020-01-01T00:00:50Z | gpu | mem_used_percent | 51.25 | + +### Full query +```js +from(bucket: "gpu-monitor") + |> range(start: 2020-01-01T00:00:00Z) + |> filter(fn: (r) => r._measurement == "gpu" and r._field =~ /mem_/ ) + |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value") + |> map(fn: (r) => ({ + _time: r._time, + _measurement: r._measurement, + _field: "mem_used_percent", + _value: float(v: r.mem_used) / float(v: r.mem_total) * 100.0 + })) +``` + +## Examples + +#### Calculate percentages using multiple fields +```js +from(bucket: "db/rp") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "example-measurement") + |> filter(fn: (r) => + r._field == "used_system" or + r._field == "used_user" or + r._field == "total" + ) + |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value") + |> map(fn: (r) => ({ r with + _value: float(v: r.used_system + r.used_user) / float(v: r.total) * 100.0 + })) +``` + +#### Calculate percentages using multiple measurements + +1. Ensure measurements are in the same [bucket](/enterprise_influxdb/v1.9/flux/get-started/#buckets). +2. Use `filter()` to include data from both measurements. +3. Use `group()` to ungroup data and return a single table. +4. Use `pivot()` to pivot fields into columns. +5. Use `map()` to re-map rows and perform the percentage calculation. + + +```js +from(bucket: "db/rp") + |> range(start: -1h) + |> filter(fn: (r) => + (r._measurement == "m1" or r._measurement == "m2") and + (r._field == "field1" or r._field == "field2") + ) + |> group() + |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value") + |> map(fn: (r) => ({ r with _value: r.field1 / r.field2 * 100.0 })) +``` + +#### Calculate percentages using multiple data sources +```js +import "sql" +import "influxdata/influxdb/secrets" + +pgUser = secrets.get(key: "POSTGRES_USER") +pgPass = secrets.get(key: "POSTGRES_PASSWORD") +pgHost = secrets.get(key: "POSTGRES_HOST") + +t1 = sql.from( + driverName: "postgres", + dataSourceName: "postgresql://${pgUser}:${pgPass}@${pgHost}", + query:"SELECT id, name, available FROM exampleTable" +) + +t2 = from(bucket: "db/rp") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "example-measurement" and + r._field == "example-field" + ) + +join(tables: {t1: t1, t2: t2}, on: ["id"]) + |> map(fn: (r) => ({ r with _value: r._value_t2 / r.available_t1 * 100.0 })) +``` diff --git a/content/enterprise_influxdb/v1.9/flux/guides/conditional-logic.md b/content/enterprise_influxdb/v1.9/flux/guides/conditional-logic.md new file mode 100644 index 000000000..94765c2fd --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/conditional-logic.md @@ -0,0 +1,197 @@ +--- +title: Query using conditional logic +seotitle: Query using conditional logic in Flux +list_title: Conditional logic +description: > + This guide describes how to use Flux conditional expressions, such as `if`, + `else`, and `then`, to query and transform data. **Flux evaluates statements from left to right and stops evaluating once a condition matches.** +menu: + enterprise_influxdb_1_9: + name: Conditional logic + parent: Query with Flux +weight: 20 +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/conditional-logic/ +v2: /influxdb/v2.0/query-data/flux/conditional-logic/ +list_code_example: | + ```js + if color == "green" then "008000" else "ffffff" + ``` +--- + +Flux provides `if`, `then`, and `else` conditional expressions that allow for powerful and flexible Flux queries. + +##### Conditional expression syntax +```js +// Pattern +if then else + +// Example +if color == "green" then "008000" else "ffffff" +``` + +Conditional expressions are most useful in the following contexts: + +- When defining variables. +- When using functions that operate on a single row at a time ( + [`filter()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/filter/), + [`map()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/map/), + [`reduce()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/reduce) ). + +## Evaluating conditional expressions + +Flux evaluates statements in order and stops evaluating once a condition matches. + +For example, given the following statement: + +```js +if r._value > 95.0000001 and r._value <= 100.0 then "critical" +else if r._value > 85.0000001 and r._value <= 95.0 then "warning" +else if r._value > 70.0000001 and r._value <= 85.0 then "high" +else "normal" +``` + +When `r._value` is 96, the output is "critical" and the remaining conditions are not evaluated. + +## Examples + +- [Conditionally set the value of a variable](#conditionally-set-the-value-of-a-variable) +- [Create conditional filters](#create-conditional-filters) +- [Conditionally transform column values with map()](#conditionally-transform-column-values-with-map) +- [Conditionally increment a count with reduce()](#conditionally-increment-a-count-with-reduce) + +### Conditionally set the value of a variable +The following example sets the `overdue` variable based on the +`dueDate` variable's relation to `now()`. + +```js +dueDate = 2019-05-01 +overdue = if dueDate < now() then true else false +``` + +### Create conditional filters +The following example uses an example `metric` variable to change how the query filters data. +`metric` has three possible values: + +- Memory +- CPU +- Disk + +```js +metric = "Memory" + +from(bucket: "telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => + if v.metric == "Memory" + then r._measurement == "mem" and r._field == "used_percent" + else if v.metric == "CPU" + then r._measurement == "cpu" and r._field == "usage_user" + else if v.metric == "Disk" + then r._measurement == "disk" and r._field == "used_percent" + else r._measurement != "" + ) +``` + +### Conditionally transform column values with map() +The following example uses the [`map()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/map/) +to conditionally transform column values. +It sets the `level` column to a specific string based on `_value` column. + +{{< code-tabs-wrapper >}} +{{% code-tabs %}} +[No Comments](#) +[Comments](#) +{{% /code-tabs %}} +{{% code-tab-content %}} +```js +from(bucket: "telegraf/autogen") + |> range(start: -5m) + |> filter(fn: (r) => r._measurement == "mem" and r._field == "used_percent" ) + |> map(fn: (r) => ({ + r with + level: + if r._value >= 95.0000001 and r._value <= 100.0 then "critical" + else if r._value >= 85.0000001 and r._value <= 95.0 then "warning" + else if r._value >= 70.0000001 and r._value <= 85.0 then "high" + else "normal" + }) + ) +``` +{{% /code-tab-content %}} +{{% code-tab-content %}} +```js +from(bucket: "telegraf/autogen") + |> range(start: -5m) + |> filter(fn: (r) => r._measurement == "mem" and r._field == "used_percent" ) + |> map(fn: (r) => ({ + // Retain all existing columns in the mapped row + r with + // Set the level column value based on the _value column + level: + if r._value >= 95.0000001 and r._value <= 100.0 then "critical" + else if r._value >= 85.0000001 and r._value <= 95.0 then "warning" + else if r._value >= 70.0000001 and r._value <= 85.0 then "high" + else "normal" + }) + ) +``` + +{{% /code-tab-content %}} +{{< /code-tabs-wrapper >}} + +### Conditionally increment a count with reduce() +The following example uses the [`aggregateWindow()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/aggregatewindow/) +and [`reduce()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/reduce/) +functions to count the number of records in every five minute window that exceed a defined threshold. + +{{< code-tabs-wrapper >}} +{{% code-tabs %}} +[No Comments](#) +[Comments](#) +{{% /code-tabs %}} +{{% code-tab-content %}} +```js +threshold = 65.0 + +from(bucket: "telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "mem" and r._field == "used_percent" ) + |> aggregateWindow( + every: 5m, + fn: (column, tables=<-) => tables |> reduce( + identity: {above_threshold_count: 0.0}, + fn: (r, accumulator) => ({ + above_threshold_count: + if r._value >= threshold then accumulator.above_threshold_count + 1.0 + else accumulator.above_threshold_count + 0.0 + }) + ) + ) +``` +{{% /code-tab-content %}} +{{% code-tab-content %}} +```js +threshold = 65.0 + +from(bucket: "telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "mem" and r._field == "used_percent" ) + // Aggregate data into 5 minute windows using a custom reduce() function + |> aggregateWindow( + every: 5m, + // Use a custom function in the fn parameter. + // The aggregateWindow fn parameter requires 'column' and 'tables' parameters. + fn: (column, tables=<-) => tables |> reduce( + identity: {above_threshold_count: 0.0}, + fn: (r, accumulator) => ({ + // Conditionally increment above_threshold_count if + // r.value exceeds the threshold + above_threshold_count: + if r._value >= threshold then accumulator.above_threshold_count + 1.0 + else accumulator.above_threshold_count + 0.0 + }) + ) + ) +``` +{{% /code-tab-content %}} +{{< /code-tabs-wrapper >}} diff --git a/content/enterprise_influxdb/v1.9/flux/guides/cumulativesum.md b/content/enterprise_influxdb/v1.9/flux/guides/cumulativesum.md new file mode 100644 index 000000000..57b7fab4c --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/cumulativesum.md @@ -0,0 +1,69 @@ +--- +title: Query cumulative sum +seotitle: Query cumulative sum in Flux +list_title: Cumulative sum +description: > + Use the `cumulativeSum()` function to calculate a running total of values. +weight: 10 +menu: + enterprise_influxdb_1_9: + parent: Query with Flux + name: Cumulative sum +list_query_example: cumulative_sum +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/cumulativesum/ +v2: /influxdb/v2.0/query-data/flux/cumulativesum/ +--- + +Use the [`cumulativeSum()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/cumulativesum/) +to calculate a running total of values. +`cumulativeSum` sums the values of subsequent records and returns each row updated with the summed total. + +{{< flex >}} +{{% flex-content "half" %}} +**Given the following input table:** + +| _time | _value | +| ----- |:------:| +| 0001 | 1 | +| 0002 | 2 | +| 0003 | 1 | +| 0004 | 3 | +{{% /flex-content %}} +{{% flex-content "half" %}} +**`cumulativeSum()` returns:** + +| _time | _value | +| ----- |:------:| +| 0001 | 1 | +| 0002 | 3 | +| 0003 | 4 | +| 0004 | 7 | +{{% /flex-content %}} +{{< /flex >}} + +{{% note %}} +The examples below use the [example data variable](/enterprise_influxdb/v1.9/flux/guides/#example-data-variable). +{{% /note %}} + +##### Calculate the running total of values +```js +data + |> cumulativeSum() +``` + +## Use cumulativeSum() with aggregateWindow() +[`aggregateWindow()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/aggregatewindow/) +segments data into windows of time, aggregates data in each window into a single +point, then removes the time-based segmentation. +It is primarily used to downsample data. + +`aggregateWindow()` expects an aggregate function that returns a single row for each time window. +To use `cumulativeSum()` with `aggregateWindow`, use `sum` in `aggregateWindow()`, +then calculate the running total of the aggregate values with `cumulativeSum()`. + + +```js +data + |> aggregateWindow(every: 5m, fn: sum) + |> cumulativeSum() +``` diff --git a/content/enterprise_influxdb/v1.9/flux/guides/execute-queries.md b/content/enterprise_influxdb/v1.9/flux/guides/execute-queries.md new file mode 100644 index 000000000..88f58b70f --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/execute-queries.md @@ -0,0 +1,96 @@ +--- +title: Execute Flux queries +description: Use the InfluxDB CLI, API, and the Chronograf Data Explorer to execute Flux queries. +menu: + enterprise_influxdb_1_9: + name: Execute Flux queries + parent: Query with Flux +weight: 1 +aliases: + - /enterprise_influxdb/v1.9/flux/guides/executing-queries/ +v2: /influxdb/v2.0/query-data/execute-queries/ +--- + +There are multiple ways to execute Flux queries with InfluxDB and Chronograf v1.8+. +This guide covers the different options: + +1. [Chronograf's Data Explorer](#chronograf-s-data-explorer) +2. [Influx CLI](#influx-cli) +3. [InfluxDB API](#influxdb-api) + +> Before attempting these methods, make sure Flux is enabled by setting +> `flux-enabled = true` in the `[http]` section of your InfluxDB configuration file. + +## Chronograf's Data Explorer +Chronograf v1.8+ supports Flux in its Data Explorer. +Flux queries can be built, executed, and visualized from within the Chronograf user interface. + +{{% note %}} +If [authentication is enabled](/enterprise_influxdb/v1.9/administration/authentication_and_authorization) +on your InfluxDB instance, use the `-username` flag to provide your InfluxDB username and +the `-password` flag to provide your password. +{{% /note %}} + +### Submit a Flux query via via STDIN +Flux queries an be piped into the `influx` CLI via STDIN. +Query results are otuput in your terminal. + +{{< code-tabs-wrapper >}} +{{% code-tabs %}} +[No Auth](#) +[Auth Enabled](#) +{{% /code-tabs %}} +{{% code-tab-content %}} +```bash +echo '' | influx -type=flux +``` +{{% /code-tab-content %}} +{{% code-tab-content %}} +```bash +echo '' | influx -type=flux -username myuser -password PasSw0rd +``` +{{% /code-tab-content %}} +{{< /code-tabs-wrapper >}} + +## InfluxDB API +Flux can be used to query InfluxDB through InfluxDB's `/api/v2/query` endpoint. +Queried data is returned in annotated CSV format. + +In your request, set the following: + +- `Accept` header to `application/csv` +- `Content-type` header to `application/vnd.flux` +- If [authentication is enabled](/enterprise_influxdb/v1.9/administration/authentication_and_authorization) + on your InfluxDB instance, `Authorization` header to `Token :` + +This allows you to POST the Flux query in plain text and receive the annotated CSV response. + +Below is an example `curl` command that queries InfluxDB using Flux: + +{{< code-tabs-wrapper >}} +{{% code-tabs %}} +[No Auth](#) +[Auth Enabled](#) +{{% /code-tabs %}} +{{% code-tab-content %}} +```bash +curl -XPOST localhost:8086/api/v2/query -sS \ + -H 'Accept:application/csv' \ + -H 'Content-type:application/vnd.flux' \ + -d 'from(bucket:"telegraf") + |> range(start:-5m) + |> filter(fn:(r) => r._measurement == "cpu")' +``` +{{% /code-tab-content %}} +{{% code-tab-content %}} +```bash +curl -XPOST localhost:8086/api/v2/query -sS \ + -H 'Accept:application/csv' \ + -H 'Content-type:application/vnd.flux' \ + -H 'Authorization: Token :' \ + -d 'from(bucket:"telegraf") + |> range(start:-5m) + |> filter(fn:(r) => r._measurement == "cpu")' +``` +{{% /code-tab-content %}} +{{< /code-tabs-wrapper >}} diff --git a/content/enterprise_influxdb/v1.9/flux/guides/exists.md b/content/enterprise_influxdb/v1.9/flux/guides/exists.md new file mode 100644 index 000000000..ed7c685e6 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/exists.md @@ -0,0 +1,82 @@ +--- +title: Check if a value exists +seotitle: Use Flux to check if a value exists +list_title: Exists +description: > + Use the Flux `exists` operator to check if a record contains a key or if that + key's value is `null`. +menu: + enterprise_influxdb_1_9: + name: Exists + parent: Query with Flux +weight: 20 +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/exists/ +v2: /influxdb/v2.0/query-data/flux/exists/ +list_code_example: | + ##### Filter null values + ```js + data + |> filter(fn: (r) => exists r._value) + ``` +--- + +Use the Flux `exists` operator to check if a record contains a key or if that +key's value is `null`. + +```js +p = {firstName: "John", lastName: "Doe", age: 42} + +exists p.firstName +// Returns true + +exists p.height +// Returns false +``` + +If you're just getting started with Flux queries, check out the following: + +- [Get started with Flux](/enterprise_influxdb/v1.9/flux/get-started/) for a conceptual overview of Flux and parts of a Flux query. +- [Execute queries](/enterprise_influxdb/v1.9/flux/guides/execute-queries/) to discover a variety of ways to run your queries. + +Use `exists` with row functions ( +[`filter()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/filter/), +[`map()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/map/), +[`reduce()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/reduce/)) +to check if a row includes a column or if the value for that column is `null`. + +#### Filter null values +```js +from(bucket: "db/rp") + |> range(start: -5m) + |> filter(fn: (r) => exists r._value) +``` + +#### Map values based on existence +```js +from(bucket: "default") + |> range(start: -30s) + |> map(fn: (r) => ({ + r with + human_readable: + if exists r._value then "${r._field} is ${string(v:r._value)}." + else "${r._field} has no value." + })) +``` + +#### Ignore null values in a custom aggregate function +```js +customSumProduct = (tables=<-) => + tables + |> reduce( + identity: {sum: 0.0, product: 1.0}, + fn: (r, accumulator) => ({ + r with + sum: + if exists r._value then r._value + accumulator.sum + else accumulator.sum, + product: + if exists r._value then r.value * accumulator.product + else accumulator.product + }) + ) +``` diff --git a/content/enterprise_influxdb/v1.9/flux/guides/fill.md b/content/enterprise_influxdb/v1.9/flux/guides/fill.md new file mode 100644 index 000000000..e03417349 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/fill.md @@ -0,0 +1,112 @@ +--- +title: Fill null values in data +seotitle: Fill null values in data +list_title: Fill +description: > + Use the `fill()` function to replace _null_ values. +weight: 10 +menu: + enterprise_influxdb_1_9: + parent: Query with Flux + name: Fill +list_query_example: fill_null +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/fill/ +v2: /influxdb/v2.0/query-data/flux/fill/ +--- + +Use the [`fill()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/fill/) +to replace _null_ values with: + +- [the previous non-null value](#fill-with-the-previous-value) +- [a specified value](#fill-with-a-specified-value) + + +```js +data + |> fill(usePrevious: true) + +// OR + +data + |> fill(value: 0.0) +``` + +{{% note %}} +#### Fill empty windows of time +The `fill()` function **does not** fill empty windows of time. +It only replaces _null_ values in existing data. +Filling empty windows of time requires time interpolation +_(see [influxdata/flux#2428](https://github.com/influxdata/flux/issues/2428))_. +{{% /note %}} + +## Fill with the previous value +To fill _null_ values with the previous **non-null** value, set the `usePrevious` parameter to `true`. + +{{% note %}} +Values remain _null_ if there is no previous non-null value in the table. +{{% /note %}} + +```js +data + |> fill(usePrevious: true) +``` + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | null | +| 2020-01-01T00:02:00Z | 0.8 | +| 2020-01-01T00:03:00Z | null | +| 2020-01-01T00:04:00Z | null | +| 2020-01-01T00:05:00Z | 1.4 | +{{% /flex-content %}} +{{% flex-content %}} +**`fill(usePrevious: true)` returns:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | null | +| 2020-01-01T00:02:00Z | 0.8 | +| 2020-01-01T00:03:00Z | 0.8 | +| 2020-01-01T00:04:00Z | 0.8 | +| 2020-01-01T00:05:00Z | 1.4 | +{{% /flex-content %}} +{{< /flex >}} + +## Fill with a specified value +To fill _null_ values with a specified value, use the `value` parameter to specify the fill value. +_The fill value must match the [data type](/{{< latest "influxdb" "v2" >}}/reference/flux/language/types/#basic-types) +of the [column](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/fill/#column)._ + +```js +data + |> fill(value: 0.0) +``` + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | null | +| 2020-01-01T00:02:00Z | 0.8 | +| 2020-01-01T00:03:00Z | null | +| 2020-01-01T00:04:00Z | null | +| 2020-01-01T00:05:00Z | 1.4 | +{{% /flex-content %}} +{{% flex-content %}} +**`fill(value: 0.0)` returns:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 0.0 | +| 2020-01-01T00:02:00Z | 0.8 | +| 2020-01-01T00:03:00Z | 0.0 | +| 2020-01-01T00:04:00Z | 0.0 | +| 2020-01-01T00:05:00Z | 1.4 | +{{% /flex-content %}} +{{< /flex >}} diff --git a/content/enterprise_influxdb/v1.9/flux/guides/first-last.md b/content/enterprise_influxdb/v1.9/flux/guides/first-last.md new file mode 100644 index 000000000..b001f492e --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/first-last.md @@ -0,0 +1,149 @@ +--- +title: Query first and last values +seotitle: Query first and last values in Flux +list_title: First and last +description: > + Use the `first()` or `last()` functions to return the first or last point in an input table. +weight: 10 +menu: + enterprise_influxdb_1_9: + parent: Query with Flux + name: First & last +list_query_example: first_last +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/first-last/ +v2: /influxdb/v2.0/query-data/flux/first-last/ +--- + +Use the [`first()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/first/) or +[`last()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/last/) functions +to return the first or last record in an input table. + +```js +data + |> first() + +// OR + +data + |> last() +``` + +{{% note %}} +By default, InfluxDB returns results sorted by time, however you can use the +[`sort()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/sort/) +to change how results are sorted. +`first()` and `last()` respect the sort order of input data and return records +based on the order they are received in. +{{% /note %}} + +### first +`first()` returns the first non-null record in an input table. + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.0 | +| 2020-01-01T00:03:00Z | 2.0 | +| 2020-01-01T00:04:00Z | 3.0 | +{{% /flex-content %}} +{{% flex-content %}} +**The following function returns:** +```js +|> first() +``` + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +{{% /flex-content %}} +{{< /flex >}} + +### last +`last()` returns the last non-null record in an input table. + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.0 | +| 2020-01-01T00:03:00Z | 2.0 | +| 2020-01-01T00:04:00Z | 3.0 | +{{% /flex-content %}} +{{% flex-content %}} +**The following function returns:** + +```js +|> last() +``` + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:04:00Z | 3.0 | +{{% /flex-content %}} +{{< /flex >}} + +## Use first() or last() with aggregateWindow() +Use `first()` and `last()` with [`aggregateWindow()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/aggregatewindow/) +to select the first or last records in time-based groups. +`aggregateWindow()` segments data into windows of time, aggregates data in each window into a single +point using aggregate or selector functions, and then removes the time-based segmentation. + + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:00:00Z | 10 | +| 2020-01-01T00:00:15Z | 12 | +| 2020-01-01T00:00:45Z | 9 | +| 2020-01-01T00:01:05Z | 9 | +| 2020-01-01T00:01:10Z | 15 | +| 2020-01-01T00:02:30Z | 11 | +{{% /flex-content %}} + +{{% flex-content %}} +**The following function returns:** +{{< code-tabs-wrapper >}} +{{% code-tabs %}} +[first](#) +[last](#) +{{% /code-tabs %}} +{{% code-tab-content %}} +```js +|> aggregateWindow( + every: 1h, + fn: first +) +``` +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:00:59Z | 10 | +| 2020-01-01T00:01:59Z | 9 | +| 2020-01-01T00:02:59Z | 11 | +{{% /code-tab-content %}} +{{% code-tab-content %}} +```js +|> aggregateWindow( + every: 1h, + fn: last +) +``` + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:00:59Z | 9 | +| 2020-01-01T00:01:59Z | 15 | +| 2020-01-01T00:02:59Z | 11 | +{{% /code-tab-content %}} +{{< /code-tabs-wrapper >}} +{{%/flex-content %}} +{{< /flex >}} diff --git a/content/enterprise_influxdb/v1.9/flux/guides/flux-in-dashboards.md b/content/enterprise_influxdb/v1.9/flux/guides/flux-in-dashboards.md new file mode 100644 index 000000000..f3b8d7e19 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/flux-in-dashboards.md @@ -0,0 +1,151 @@ +--- +title: Use Flux in Chronograf dashboards +description: > + This guide walks through using Flux queries in Chronograf dashboard cells, + what template variables are available, and how to use them. +menu: + enterprise_influxdb_1_9: + name: Use Flux in dashboards + parent: Query with Flux +weight: 30 +canonical: /{{< latest "influxdb" "v1" >}}/flux/guides/flux-in-dashboards/ +--- + +[Chronograf](/{{< latest "chronograf" >}}/) is the web user interface for managing for the +InfluxData platform that lest you create and customize dashboards that visualize your data. +Visualized data is retrieved using either an InfluxQL or Flux query. +This guide walks through using Flux queries in Chronograf dashboard cells. + +## Using Flux in dashboard cells + +--- + +_**Chronograf v1.8+** and **InfluxDB v1.8 with [Flux enabled](/enterprise_influxdb/v1.9/flux/installation)** +are required to use Flux in dashboards._ + +--- + +To use Flux in a dashboard cell, either create a new cell or edit an existing cell +by clicking the **pencil** icon in the top right corner of the cell. +To the right of the **Source dropdown** above the graph preview, select **Flux** as the source type. + +{{< img-hd src="/img/influxdb/1-7-flux-dashboard-cell.png" alt="Flux in Chronograf dashboard cells" />}} + +> The Flux source type is only available if your data source has +> [Flux enabled](/enterprise_influxdb/v1.9/flux/installation). + +This will provide **Schema**, **Script**, and **Functions** panes. + +### Schema pane +The Schema pane allows you to explore your data and add filters for specific +measurements, fields, and tags to your Flux script. + +{{< img-hd src="/img/influxdb/1-7-flux-dashboard-add-filter.png" title="Add a filter from the Schema panel" />}} + +### Script pane +The Script pane is where you write your Flux script. +In its default state, the **Script** pane includes an optional [Script Wizard](/chronograf/v1.8/guides/querying-data/#explore-data-with-flux) +that uses selected options to build a Flux query for you. +The generated query includes all the relevant functions and [template variables](#template-variables-in-flux) +required to return your desired data. + +### Functions pane +The Functions pane provides a list of functions available in your Flux queries. +Clicking on a function will add it to the end of the script in the Script pane. +Hovering over a function provides documentation for the function as well as links +to deep documentation. + +### Dynamic sources +Chronograf can be configured with multiple data sources. +The **Sources dropdown** allows you to select a specific data source to connect to, +but a **Dynamic Source** options is also available. +With a dynamic source, the cell will query data from whatever data source to which +Chronograf is currently connected. +Connections are managed under Chronograf's **Configuration** tab. + +### View raw data +As you're building your Flux scripts, each function processes or transforms your +data is ways specific to the function. +It can be helpful to view the actual data in order to see how it is being shaped. +The **View Raw Data** toggle above the data visualization switches between graphed +data and raw data shown in table form. + +{{< img-hd src="/img/influxdb/1-7-flux-dashboard-view-raw.png" alt="View raw data" />}} + +_The **View Raw Data** toggle is only available when using Flux._ + +## Template variables in Flux +Chronograf [template variables](/{{< latest "chronograf" >}}/guides/dashboard-template-variables/) +allow you to alter specific components of cells’ queries using elements provided in the +Chronograf user interface. + +In your Flux query, reference template variables just as you would reference defined Flux variables. +The following example uses Chronograf's [predefined template variables](#predefined-template-variables), +`dashboardTime`, `upperDashboardTime`, and `autoInterval`: + +```js +from(bucket: "telegraf/autogen") + |> filter(fn: (r) => r._measurement == "cpu") + |> range( + start: dashboardTime, + stop: upperDashboardTime + ) + window(every: autoInterval) +``` + +### Predefined template variables + +#### dashboardTime +The `dashboardTime` template variable represents the lower time bound of ranged data. +It's value is controlled by the time dropdown in your dashboard. +It should be used to define the `start` parameter of the `range()` function. + +```js +dataSet + |> range( + start: dashboardTime + ) +``` + +#### upperDashboardTime +The `upperDashboardTime` template variable represents the upper time bound of ranged data. +It's value is modified by the time dropdown in your dashboard when using an absolute time range. +It should be used to define the `stop` parameter of the `range()` function. + +```js +dataSet + |> range( + start: dashboardTime, + stop: upperDashboardTime + ) +``` +> As a best practice, always set the `stop` parameter of the `range()` function to `upperDashboardTime` in cell queries. +> Without it, `stop` defaults to "now" and the absolute upper range bound selected in the time dropdown is not honored, +> potentially causing unnecessary load on InfluxDB. + +#### autoInterval +The `autoInterval` template variable represents the refresh interval of the dashboard +and is controlled by the refresh interval dropdown. +It's typically used to align window intervals created in +[windowing and aggregation](/enterprise_influxdb/v1.9/flux/guides/window-aggregate) operations with dashboard refreshes. + +```js +dataSet + |> range( + start: dashboardTime, + stop: upperDashboardTime + ) + |> aggregateWindow( + every: autoInterval, + fn: mean + ) +``` + +### Custom template variables +<% warn %> +Chronograf does not support the use of custom template variables in Flux queries. +<% /warn %> + +## Using Flux and InfluxQL +Within individual dashboard cells, the use of Flux and InfluxQL is mutually exclusive. +However, a dashboard may consist of different cells, each using Flux or InfluxQL. diff --git a/content/enterprise_influxdb/v1.9/flux/guides/geo/_index.md b/content/enterprise_influxdb/v1.9/flux/guides/geo/_index.md new file mode 100644 index 000000000..36eed98a8 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/geo/_index.md @@ -0,0 +1,91 @@ +--- +title: Work with geo-temporal data +list_title: Geo-temporal data +description: > + Use the Flux Geo package to filter geo-temporal data and group by geographic location or track. +menu: + enterprise_influxdb_1_9: + name: Geo-temporal data + parent: Query with Flux +weight: 20 +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/geo/ +v2: /influxdb/v2.0/query-data/flux/geo/ +list_code_example: | + ```js + import "experimental/geo" + + sampleGeoData + |> geo.filterRows(region: {lat: 30.04, lon: 31.23, radius: 200.0}) + |> geo.groupByArea(newColumn: "geoArea", level: 5) + ``` +--- + +Use the [Flux Geo package](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/geo) to +filter geo-temporal data and group by geographic location or track. + +{{% warn %}} +The Geo package is experimental and subject to change at any time. +By using it, you agree to the [risks of experimental functions](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/#use-experimental-functions-at-your-own-risk). +{{% /warn %}} + +**To work with geo-temporal data:** + +1. Import the `experimental/geo` package. + + ```js + import "experimental/geo" + ``` + +2. Load geo-temporal data. _See below for [sample geo-temporal data](#sample-data)._ +3. Do one or more of the following: + + - [Shape data to work with the Geo package](#shape-data-to-work-with-the-geo-package) + - [Filter data by region](#filter-geo-temporal-data-by-region) (using strict or non-strict filters) + - [Group data by area or by track](#group-geo-temporal-data) + +{{< children >}} + +--- + +## Sample data +Many of the examples in this section use a `sampleGeoData` variable that represents +a sample set of geo-temporal data. +The [Bird Migration Sample Data](https://github.com/influxdata/influxdb2-sample-data/tree/master/bird-migration-data) +available on GitHub provides sample geo-temporal data that meets the +[requirements of the Flux Geo package](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/geo/#geo-schema-requirements). + +### Load annotated CSV sample data +Use the [experimental `csv.from()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/csv/from/) +to load the sample bird migration annotated CSV data from GitHub: + +```js +import `experimental/csv` + +sampleGeoData = csv.from( + url: "https://github.com/influxdata/influxdb2-sample-data/blob/master/bird-migration-data/bird-migration.csv" +) +``` + +{{% note %}} +`csv.from(url: ...)` downloads sample data each time you execute the query **(~1.3 MB)**. +If bandwidth is a concern, use the [`to()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/outputs/to/) +to write the data to a bucket, and then query the bucket with [`from()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/inputs/from/). +{{% /note %}} + +### Write sample data to InfluxDB with line protocol +Use `curl` and the `influx write` command to write bird migration line protocol to InfluxDB. +Replace `db/rp` with your destination bucket: + +```sh +curl https://raw.githubusercontent.com/influxdata/influxdb2-sample-data/master/bird-migration-data/bird-migration.line --output ./tmp-data +influx write -b db/rp @./tmp-data +rm -f ./tmp-data +``` + +Use Flux to query the bird migration data and assign it to the `sampleGeoData` variable: + +```js +sampleGeoData = from(bucket: "db/rp") + |> range(start: 2019-01-01T00:00:00Z, stop: 2019-12-31T23:59:59Z) + |> filter(fn: (r) => r._measurement == "migration") +``` diff --git a/content/enterprise_influxdb/v1.9/flux/guides/geo/filter-by-region.md b/content/enterprise_influxdb/v1.9/flux/guides/geo/filter-by-region.md new file mode 100644 index 000000000..88c182823 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/geo/filter-by-region.md @@ -0,0 +1,132 @@ +--- +title: Filter geo-temporal data by region +description: > + Use the `geo.filterRows` function to filter geo-temporal data by box-shaped, circular, or polygonal geographic regions. +menu: + enterprise_influxdb_1_9: + name: Filter by region + parent: Geo-temporal data +weight: 302 +related: + - /{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/geo/ + - /{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/geo/filterrows/ +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/geo/filter-by-region/ +v2: /influxdb/v2.0/query-data/flux/geo/filter-by-region/ +list_code_example: | + ```js + import "experimental/geo" + + sampleGeoData + |> geo.filterRows( + region: {lat: 30.04, lon: 31.23, radius: 200.0}, + strict: true + ) + ``` +--- + +Use the [`geo.filterRows` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/geo/filterrows/) +to filter geo-temporal data by geographic region: + +1. [Define a geographic region](#define-a-geographic-region) +2. [Use strict or non-strict filtering](#strict-and-non-strict-filtering) + +The following example uses the [sample bird migration data](/enterprise_influxdb/v1.9/flux/guides/geo/#sample-data) +and queries data points **within 200km of Cairo, Egypt**: + +```js +import "experimental/geo" + +sampleGeoData + |> geo.filterRows( + region: {lat: 30.04, lon: 31.23, radius: 200.0}, + strict: true + ) +``` + +## Define a geographic region +Many functions in the Geo package filter data based on geographic region. +Define a geographic region using one of the the following shapes: + +- [box](#box) +- [circle](#circle) +- [polygon](#polygon) + +### box +Define a box-shaped region by specifying a record containing the following properties: + +- **minLat:** minimum latitude in decimal degrees (WGS 84) _(Float)_ +- **maxLat:** maximum latitude in decimal degrees (WGS 84) _(Float)_ +- **minLon:** minimum longitude in decimal degrees (WGS 84) _(Float)_ +- **maxLon:** maximum longitude in decimal degrees (WGS 84) _(Float)_ + +##### Example box-shaped region +```js +{ + minLat: 40.51757813, + maxLat: 40.86914063, + minLon: -73.65234375, + maxLon: -72.94921875 +} +``` + +### circle +Define a circular region by specifying a record containing the following properties: + +- **lat**: latitude of the circle center in decimal degrees (WGS 84) _(Float)_ +- **lon**: longitude of the circle center in decimal degrees (WGS 84) _(Float)_ +- **radius**: radius of the circle in kilometers (km) _(Float)_ + +##### Example circular region +```js +{ + lat: 40.69335938, + lon: -73.30078125, + radius: 20.0 +} +``` + +### polygon +Define a polygonal region with a record containing the latitude and longitude for +each point in the polygon: + +- **points**: points that define the custom polygon _(Array of records)_ + + Define each point with a record containing the following properties: + + - **lat**: latitude in decimal degrees (WGS 84) _(Float)_ + - **lon**: longitude in decimal degrees (WGS 84) _(Float)_ + +##### Example polygonal region +```js +{ + points: [ + {lat: 40.671659, lon: -73.936631}, + {lat: 40.706543, lon: -73.749177}, + {lat: 40.791333, lon: -73.880327} + ] +} +``` + +## Strict and non-strict filtering +In most cases, the specified geographic region does not perfectly align with S2 grid cells. + +- **Non-strict filtering** returns points that may be outside of the specified region but + inside S2 grid cells partially covered by the region. +- **Strict filtering** returns only points inside the specified region. + +_Strict filtering is less performant, but more accurate than non-strict filtering._ + + S2 grid cell + Filter region + Returned point + +{{< flex >}} +{{% flex-content %}} +**Strict filtering** +{{< svg "/static/svgs/geo-strict.svg" >}} +{{% /flex-content %}} +{{% flex-content %}} +**Non-strict filtering** +{{< svg "/static/svgs/geo-non-strict.svg" >}} +{{% /flex-content %}} +{{< /flex >}} diff --git a/content/enterprise_influxdb/v1.9/flux/guides/geo/group-geo-data.md b/content/enterprise_influxdb/v1.9/flux/guides/geo/group-geo-data.md new file mode 100644 index 000000000..ce24153ca --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/geo/group-geo-data.md @@ -0,0 +1,76 @@ +--- +title: Group geo-temporal data +description: > + Use the `geo.groupByArea()` to group geo-temporal data by area and `geo.asTracks()` + to group data into tracks or routes. +menu: + enterprise_influxdb_1_9: + parent: Geo-temporal data +weight: 302 +related: + - /{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/geo/ + - /{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/geo/groupbyarea/ + - /{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/geo/astracks/ +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/geo/group-geo-data/ +v2: /influxdb/v2.0/query-data/flux/geo/group-geo-data/ +list_code_example: | + ```js + import "experimental/geo" + + sampleGeoData + |> geo.groupByArea(newColumn: "geoArea", level: 5) + |> geo.asTracks(groupBy: ["id"],sortBy: ["_time"]) + ``` +--- + +Use the `geo.groupByArea()` to group geo-temporal data by area and `geo.asTracks()` +to group data into tracks or routes. + +- [Group data by area](#group-data-by-area) +- [Group data into tracks or routes](#group-data-by-track-or-route) + +### Group data by area +Use the [`geo.groupByArea()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/geo/groupbyarea/) +to group geo-temporal data points by geographic area. +Areas are determined by [S2 grid cells](https://s2geometry.io/devguide/s2cell_hierarchy.html#s2cellid-numbering) + +- Specify a new column to store the unique area identifier for each point with the `newColumn` parameter. +- Specify the [S2 cell level](https://s2geometry.io/resources/s2cell_statistics) + to use when calculating geographic areas with the `level` parameter. + +The following example uses the [sample bird migration data](/enterprise_influxdb/v1.9/flux/guides/geo/#sample-data) +to query data points within 200km of Cairo, Egypt and group them by geographic area: + +```js +import "experimental/geo" + +sampleGeoData + |> geo.filterRows(region: {lat: 30.04, lon: 31.23, radius: 200.0}) + |> geo.groupByArea( + newColumn: "geoArea", + level: 5 + ) +``` + +### Group data by track or route +Use [`geo.asTracks()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/geo/astracks/) +to group data points into tracks or routes and order them by time or other columns. +Data must contain a unique identifier for each track. For example: `id` or `tid`. + +- Specify columns that uniquely identify each track or route with the `groupBy` parameter. +- Specify which columns to sort by with the `sortBy` parameter. Default is `["_time"]`. + +The following example uses the [sample bird migration data](/enterprise_influxdb/v1.9/flux/guides/geo/#sample-data) +to query data points within 200km of Cairo, Egypt and group them into routes unique +to each bird: + +```js +import "experimental/geo" + +sampleGeoData + |> geo.filterRows(region: {lat: 30.04, lon: 31.23, radius: 200.0}) + |> geo.asTracks( + groupBy: ["id"], + sortBy: ["_time"] + ) +``` diff --git a/content/enterprise_influxdb/v1.9/flux/guides/geo/shape-geo-data.md b/content/enterprise_influxdb/v1.9/flux/guides/geo/shape-geo-data.md new file mode 100644 index 000000000..f9d03d5ca --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/geo/shape-geo-data.md @@ -0,0 +1,122 @@ +--- +title: Shape data to work with the Geo package +description: > + Functions in the Flux Geo package require **lat** and **lon** fields and an **s2_cell_id** tag. + Rename latitude and longitude fields and generate S2 cell ID tokens. +menu: + enterprise_influxdb_1_9: + name: Shape geo-temporal data + parent: Geo-temporal data +weight: 301 +related: + - /{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/geo/ + - /{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/geo/shapedata/ +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/geo/shape-geo-data/ +v2: /influxdb/v2.0/query-data/flux/geo/shape-geo-data/ +list_code_example: | + ```js + import "experimental/geo" + + sampleGeoData + |> geo.shapeData(latField: "latitude", lonField: "longitude", level: 10) + ``` +--- + +Functions in the Geo package require the following data schema: + +- an **s2_cell_id** tag containing the [S2 Cell ID](https://s2geometry.io/devguide/s2cell_hierarchy.html#s2cellid-numbering) + **as a token** +- a **`lat` field** field containing the **latitude in decimal degrees** (WGS 84) +- a **`lon` field** field containing the **longitude in decimal degrees** (WGS 84) + +## Shape geo-temporal data +If your data already contains latitude and longitude fields, use the +[`geo.shapeData()`function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/geo/shapedata/) +to rename the fields to match the requirements of the Geo package, pivot the data +into row-wise sets, and generate S2 cell ID tokens for each point. + +```js +import "experimental/geo" + +from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "example-measurement") + |> geo.shapeData( + latField: "latitude", + lonField: "longitude", + level: 10 + ) +``` + +## Generate S2 cell ID tokens +The Geo package uses the [S2 Geometry Library](https://s2geometry.io/) to represent +geographic coordinates on a three-dimensional sphere. +The sphere is divided into [cells](https://s2geometry.io/devguide/s2cell_hierarchy), +each with a unique 64-bit identifier (S2 cell ID). +Grid and S2 cell ID accuracy are defined by a [level](https://s2geometry.io/resources/s2cell_statistics). + +{{% note %}} +To filter more quickly, use higher S2 Cell ID levels, +but know that that higher levels increase [series cardinality](/enterprise_influxdb/v1.9/concepts/glossary/#series-cardinality). +{{% /note %}} + +The Geo package requires S2 cell IDs as tokens. +To generate add S2 cell IDs tokens to your data, use one of the following options: + +- [Generate S2 cell ID tokens with Telegraf](#generate-s2-cell-id-tokens-with-telegraf) +- [Generate S2 cell ID tokens language-specific libraries](#generate-s2-cell-id-tokens-language-specific-libraries) +- [Generate S2 cell ID tokens with Flux](#generate-s2-cell-id-tokens-with-flux) + +### Generate S2 cell ID tokens with Telegraf +Enable the [Telegraf S2 Geo (`s2geo`) processor](https://github.com/influxdata/telegraf/tree/master/plugins/processors/s2geo) +to generate S2 cell ID tokens at a specified `cell_level` using `lat` and `lon` field values. + +Add the `processors.s2geo` configuration to your Telegraf configuration file (`telegraf.conf`): + +```toml +[[processors.s2geo]] + ## The name of the lat and lon fields containing WGS-84 latitude and + ## longitude in decimal degrees. + lat_field = "lat" + lon_field = "lon" + + ## New tag to create + tag_key = "s2_cell_id" + + ## Cell level (see https://s2geometry.io/resources/s2cell_statistics.html) + cell_level = 9 +``` + +Telegraf stores the S2 cell ID token in the `s2_cell_id` tag. + +### Generate S2 cell ID tokens language-specific libraries +Many programming languages offer S2 Libraries with methods for generating S2 cell ID tokens. +Use latitude and longitude with the `s2.CellID.ToToken` endpoint of the S2 Geometry +Library to generate `s2_cell_id` tags. For example: + +- **Go:** [s2.CellID.ToToken()](https://godoc.org/github.com/golang/geo/s2#CellID.ToToken) +- **Python:** [s2sphere.CellId.to_token()](https://s2sphere.readthedocs.io/en/latest/api.html#s2sphere.CellId) +- **JavaScript:** [s2.cellid.toToken()](https://github.com/mapbox/node-s2/blob/master/API.md#cellidtotoken---string) + +### Generate S2 cell ID tokens with Flux +Use the [`geo.s2CellIDToken()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/geo/s2cellidtoken/) +with existing longitude (`lon`) and latitude (`lat`) field values to generate and add the S2 cell ID token. +First, use the [`geo.toRows()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/geo/torows/) +to pivot **lat** and **lon** fields into row-wise sets: + +```js +import "experimental/geo" + +from(bucket: "example-bucket") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "example-measurement") + |> geo.toRows() + |> map(fn: (r) => ({ r with + s2_cell_id: geo.s2CellIDToken(point: {lon: r.lon, lat: r.lat}, level: 10) + })) +``` + +{{% note %}} +The [`geo.shapeData()`function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/geo/shapedata/) +generates S2 cell ID tokens as well. +{{% /note %}} diff --git a/content/enterprise_influxdb/v1.9/flux/guides/group-data.md b/content/enterprise_influxdb/v1.9/flux/guides/group-data.md new file mode 100644 index 000000000..1bb7c9749 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/group-data.md @@ -0,0 +1,676 @@ +--- +title: Group data in InfluxDB with Flux +list_title: Group +description: > + Use the `group()` function to group data with common values in specific columns. +menu: + enterprise_influxdb_1_9: + name: Group + parent: Query with Flux +weight: 2 +aliases: + - /enterprise_influxdb/v1.9/flux/guides/grouping-data/ +list_query_example: group +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/group-data/ +v2: /influxdb/v2.0/query-data/flux/group-data/ +--- + +With Flux, you can group data by any column in your queried data set. +"Grouping" partitions data into tables in which each row shares a common value for specified columns. +This guide walks through grouping data in Flux and provides examples of how data is shaped in the process. + +If you're just getting started with Flux queries, check out the following: + +- [Get started with Flux](/enterprise_influxdb/v1.9/flux/get-started/) for a conceptual overview of Flux and parts of a Flux query. +- [Execute queries](/enterprise_influxdb/v1.9/flux/guides/execute-queries/) to discover a variety of ways to run your queries. + +## Group keys +Every table has a **group key** – a list of columns which for which every row in the table has the same value. + +###### Example group key +```js +[_start, _stop, _field, _measurement, host] +``` + +Grouping data in Flux is essentially defining the group key of output tables. +Understanding how modifying group keys shapes output data is key to successfully +grouping and transforming data into your desired output. + +## group() Function +Flux's [`group()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/group) defines the +group key for output tables, i.e. grouping records based on values for specific columns. + +###### group() example +```js +dataStream + |> group(columns: ["cpu", "host"]) +``` + +###### Resulting group key +```js +[cpu, host] +``` + +The `group()` function has the following parameters: + +### columns +The list of columns to include or exclude (depending on the [mode](#mode)) in the grouping operation. + +### mode +The method used to define the group and resulting group key. +Possible values include `by` and `except`. + + +## Example grouping operations +To illustrate how grouping works, define a `dataSet` variable that queries System +CPU usage from the `db/rp` bucket. +Filter the `cpu` tag so it only returns results for each numbered CPU core. + +### Data set +CPU used by system operations for all numbered CPU cores. +It uses a regular expression to filter only numbered cores. + +```js +dataSet = from(bucket: "db/rp") + |> range(start: -2m) + |> filter(fn: (r) => + r._field == "usage_system" and + r.cpu =~ /cpu[0-9*]/ + ) + |> drop(columns: ["host"]) +``` + +{{% note %}} +This example drops the `host` column from the returned data since the CPU data +is only tracked for a single host and it simplifies the output tables. +Don't drop the `host` column if monitoring multiple hosts. +{{% /note %}} + +{{% truncate %}} +``` +Table: keys: [_start, _stop, _field, _measurement, cpu] + _start:time _stop:time _field:string _measurement:string cpu:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:34:00.000000000Z 7.892107892107892 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:34:10.000000000Z 7.2 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:34:20.000000000Z 7.4 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:34:30.000000000Z 5.5 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:34:40.000000000Z 7.4 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:34:50.000000000Z 7.5 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:35:00.000000000Z 10.3 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:35:10.000000000Z 9.2 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:35:20.000000000Z 8.4 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:35:30.000000000Z 8.5 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:35:40.000000000Z 8.6 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:35:50.000000000Z 10.2 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu0 2018-11-05T21:36:00.000000000Z 10.6 + +Table: keys: [_start, _stop, _field, _measurement, cpu] + _start:time _stop:time _field:string _measurement:string cpu:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:34:00.000000000Z 0.7992007992007992 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:34:10.000000000Z 0.7 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:34:20.000000000Z 0.7 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:34:30.000000000Z 0.4 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:34:40.000000000Z 0.7 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:34:50.000000000Z 0.7 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:35:00.000000000Z 1.4 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:35:10.000000000Z 1.2 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:35:20.000000000Z 0.8 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:35:30.000000000Z 0.8991008991008991 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:35:40.000000000Z 0.8008008008008008 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:35:50.000000000Z 0.999000999000999 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu1 2018-11-05T21:36:00.000000000Z 1.1022044088176353 + +Table: keys: [_start, _stop, _field, _measurement, cpu] + _start:time _stop:time _field:string _measurement:string cpu:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:34:00.000000000Z 4.1 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:34:10.000000000Z 3.6 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:34:20.000000000Z 3.5 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:34:30.000000000Z 2.6 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:34:40.000000000Z 4.5 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:34:50.000000000Z 4.895104895104895 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:35:00.000000000Z 6.906906906906907 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:35:10.000000000Z 5.7 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:35:20.000000000Z 5.1 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:35:30.000000000Z 4.7 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:35:40.000000000Z 5.1 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:35:50.000000000Z 5.9 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu2 2018-11-05T21:36:00.000000000Z 6.4935064935064934 + +Table: keys: [_start, _stop, _field, _measurement, cpu] + _start:time _stop:time _field:string _measurement:string cpu:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:34:00.000000000Z 0.5005005005005005 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:34:10.000000000Z 0.5 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:34:20.000000000Z 0.5 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:34:30.000000000Z 0.3 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:34:40.000000000Z 0.6 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:34:50.000000000Z 0.6 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:35:00.000000000Z 1.3986013986013985 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:35:10.000000000Z 0.9 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:35:20.000000000Z 0.5005005005005005 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:35:30.000000000Z 0.7 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:35:40.000000000Z 0.6 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:35:50.000000000Z 0.8 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z usage_system cpu cpu3 2018-11-05T21:36:00.000000000Z 0.9 +``` +{{% /truncate %}} + +**Note that the group key is output with each table: `Table: keys: `.** + +![Group example data set](/img/flux/grouping-data-set.png) + +### Group by CPU +Group the `dataSet` stream by the `cpu` column. + +```js +dataSet + |> group(columns: ["cpu"]) +``` + +This won't actually change the structure of the data since it already has `cpu` +in the group key and is therefore grouped by `cpu`. +However, notice that it does change the group key: + +{{% truncate %}} +###### Group by CPU output tables +``` +Table: keys: [cpu] + cpu:string _stop:time _time:time _value:float _field:string _measurement:string _start:time +---------------------- ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:00.000000000Z 7.892107892107892 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:10.000000000Z 7.2 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:20.000000000Z 7.4 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:30.000000000Z 5.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:40.000000000Z 7.4 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:50.000000000Z 7.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:00.000000000Z 10.3 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:10.000000000Z 9.2 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:20.000000000Z 8.4 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:30.000000000Z 8.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:40.000000000Z 8.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:50.000000000Z 10.2 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu0 2018-11-05T21:36:00.000000000Z 2018-11-05T21:36:00.000000000Z 10.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [cpu] + cpu:string _stop:time _time:time _value:float _field:string _measurement:string _start:time +---------------------- ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:00.000000000Z 0.7992007992007992 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:10.000000000Z 0.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:20.000000000Z 0.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:30.000000000Z 0.4 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:40.000000000Z 0.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:50.000000000Z 0.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:00.000000000Z 1.4 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:10.000000000Z 1.2 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:20.000000000Z 0.8 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:30.000000000Z 0.8991008991008991 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:40.000000000Z 0.8008008008008008 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:50.000000000Z 0.999000999000999 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu1 2018-11-05T21:36:00.000000000Z 2018-11-05T21:36:00.000000000Z 1.1022044088176353 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [cpu] + cpu:string _stop:time _time:time _value:float _field:string _measurement:string _start:time +---------------------- ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:00.000000000Z 4.1 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:10.000000000Z 3.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:20.000000000Z 3.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:30.000000000Z 2.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:40.000000000Z 4.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:50.000000000Z 4.895104895104895 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:00.000000000Z 6.906906906906907 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:10.000000000Z 5.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:20.000000000Z 5.1 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:30.000000000Z 4.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:40.000000000Z 5.1 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:50.000000000Z 5.9 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu2 2018-11-05T21:36:00.000000000Z 2018-11-05T21:36:00.000000000Z 6.4935064935064934 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [cpu] + cpu:string _stop:time _time:time _value:float _field:string _measurement:string _start:time +---------------------- ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:00.000000000Z 0.5005005005005005 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:10.000000000Z 0.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:20.000000000Z 0.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:30.000000000Z 0.3 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:40.000000000Z 0.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:50.000000000Z 0.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:00.000000000Z 1.3986013986013985 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:10.000000000Z 0.9 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:20.000000000Z 0.5005005005005005 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:30.000000000Z 0.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:40.000000000Z 0.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:35:50.000000000Z 0.8 usage_system cpu 2018-11-05T21:34:00.000000000Z + cpu3 2018-11-05T21:36:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.9 usage_system cpu 2018-11-05T21:34:00.000000000Z +``` +{{% /truncate %}} + +The visualization remains the same. + +![Group by CPU](/img/flux/grouping-data-set.png) + +### Group by time +Grouping data by the `_time` column is a good illustration of how grouping changes the structure of your data. + +```js +dataSet + |> group(columns: ["_time"]) +``` + +When grouping by `_time`, all records that share a common `_time` value are grouped into individual tables. +So each output table represents a single point in time. + +{{% truncate %}} +###### Group by time output tables +``` +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:34:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 7.892107892107892 usage_system cpu cpu0 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.7992007992007992 usage_system cpu cpu1 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 4.1 usage_system cpu cpu2 +2018-11-05T21:34:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.5005005005005005 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:34:10.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 7.2 usage_system cpu cpu0 +2018-11-05T21:34:10.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.7 usage_system cpu cpu1 +2018-11-05T21:34:10.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 3.6 usage_system cpu cpu2 +2018-11-05T21:34:10.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.5 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:34:20.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 7.4 usage_system cpu cpu0 +2018-11-05T21:34:20.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.7 usage_system cpu cpu1 +2018-11-05T21:34:20.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 3.5 usage_system cpu cpu2 +2018-11-05T21:34:20.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.5 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:34:30.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 5.5 usage_system cpu cpu0 +2018-11-05T21:34:30.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.4 usage_system cpu cpu1 +2018-11-05T21:34:30.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 2.6 usage_system cpu cpu2 +2018-11-05T21:34:30.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.3 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:34:40.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 7.4 usage_system cpu cpu0 +2018-11-05T21:34:40.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.7 usage_system cpu cpu1 +2018-11-05T21:34:40.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 4.5 usage_system cpu cpu2 +2018-11-05T21:34:40.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.6 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:34:50.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 7.5 usage_system cpu cpu0 +2018-11-05T21:34:50.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.7 usage_system cpu cpu1 +2018-11-05T21:34:50.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 4.895104895104895 usage_system cpu cpu2 +2018-11-05T21:34:50.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.6 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:35:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 10.3 usage_system cpu cpu0 +2018-11-05T21:35:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 1.4 usage_system cpu cpu1 +2018-11-05T21:35:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 6.906906906906907 usage_system cpu cpu2 +2018-11-05T21:35:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 1.3986013986013985 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:35:10.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 9.2 usage_system cpu cpu0 +2018-11-05T21:35:10.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 1.2 usage_system cpu cpu1 +2018-11-05T21:35:10.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 5.7 usage_system cpu cpu2 +2018-11-05T21:35:10.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.9 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:35:20.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 8.4 usage_system cpu cpu0 +2018-11-05T21:35:20.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.8 usage_system cpu cpu1 +2018-11-05T21:35:20.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 5.1 usage_system cpu cpu2 +2018-11-05T21:35:20.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.5005005005005005 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:35:30.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 8.5 usage_system cpu cpu0 +2018-11-05T21:35:30.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.8991008991008991 usage_system cpu cpu1 +2018-11-05T21:35:30.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 4.7 usage_system cpu cpu2 +2018-11-05T21:35:30.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.7 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:35:40.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 8.6 usage_system cpu cpu0 +2018-11-05T21:35:40.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.8008008008008008 usage_system cpu cpu1 +2018-11-05T21:35:40.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 5.1 usage_system cpu cpu2 +2018-11-05T21:35:40.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.6 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:35:50.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 10.2 usage_system cpu cpu0 +2018-11-05T21:35:50.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.999000999000999 usage_system cpu cpu1 +2018-11-05T21:35:50.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 5.9 usage_system cpu cpu2 +2018-11-05T21:35:50.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.8 usage_system cpu cpu3 + +Table: keys: [_time] + _time:time _start:time _stop:time _value:float _field:string _measurement:string cpu:string +------------------------------ ------------------------------ ------------------------------ ---------------------------- ---------------------- ---------------------- ---------------------- +2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 10.6 usage_system cpu cpu0 +2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 1.1022044088176353 usage_system cpu cpu1 +2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 6.4935064935064934 usage_system cpu cpu2 +2018-11-05T21:36:00.000000000Z 2018-11-05T21:34:00.000000000Z 2018-11-05T21:36:00.000000000Z 0.9 usage_system cpu cpu3 +``` +{{% /truncate %}} + +Because each timestamp is structured as a separate table, when visualized, all +points that share the same timestamp appear connected. + +![Group by time](/img/flux/grouping-by-time.png) + +{{% note %}} +With some further processing, you could calculate the average CPU usage across all CPUs per point +of time and group them into a single table, but we won't cover that in this example. +If you're interested in running and visualizing this yourself, here's what the query would look like: + +```js +dataSet + |> group(columns: ["_time"]) + |> mean() + |> group(columns: ["_value", "_time"], mode: "except") +``` +{{% /note %}} + +## Group by CPU and time +Group by the `cpu` and `_time` columns. + +```js +dataSet + |> group(columns: ["cpu", "_time"]) +``` + +This outputs a table for every unique `cpu` and `_time` combination: + +{{% truncate %}} +###### Group by CPU and time output tables +``` +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:00.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 7.892107892107892 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:00.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 0.7992007992007992 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:00.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 4.1 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:00.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.5005005005005005 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:10.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 7.2 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:10.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 0.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:10.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 3.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:10.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:20.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 7.4 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:20.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 0.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:20.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 3.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:20.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:30.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 5.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:30.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 0.4 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:30.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 2.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:30.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.3 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:40.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 7.4 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:40.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 0.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:40.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 4.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:40.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:50.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 7.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:50.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 0.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:50.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 4.895104895104895 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:34:50.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:00.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 10.3 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:00.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 1.4 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:00.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 6.906906906906907 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:00.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 1.3986013986013985 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:10.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 9.2 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:10.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 1.2 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:10.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 5.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:10.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.9 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:20.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 8.4 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:20.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 0.8 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:20.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 5.1 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:20.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.5005005005005005 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:30.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 8.5 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:30.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 0.8991008991008991 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:30.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 4.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:30.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.7 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:40.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 8.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:40.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 0.8008008008008008 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:40.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 5.1 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:40.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:50.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 10.2 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:50.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 0.999000999000999 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:50.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 5.9 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:35:50.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.8 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:36:00.000000000Z cpu0 2018-11-05T21:36:00.000000000Z 10.6 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:36:00.000000000Z cpu1 2018-11-05T21:36:00.000000000Z 1.1022044088176353 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:36:00.000000000Z cpu2 2018-11-05T21:36:00.000000000Z 6.4935064935064934 usage_system cpu 2018-11-05T21:34:00.000000000Z + +Table: keys: [_time, cpu] + _time:time cpu:string _stop:time _value:float _field:string _measurement:string _start:time +------------------------------ ---------------------- ------------------------------ ---------------------------- ---------------------- ---------------------- ------------------------------ +2018-11-05T21:36:00.000000000Z cpu3 2018-11-05T21:36:00.000000000Z 0.9 usage_system cpu 2018-11-05T21:34:00.000000000Z +``` +{{% /truncate %}} + +When visualized, tables appear as individual, unconnected points. + +![Group by CPU and time](/img/flux/grouping-by-cpu-time.png) + +Grouping by `cpu` and `_time` is a good illustration of how grouping works. + +## In conclusion +Grouping is a powerful way to shape your data into your desired output format. +It modifies the group keys of output tables, grouping records into tables that +all share common values within specified columns. diff --git a/content/enterprise_influxdb/v1.9/flux/guides/histograms.md b/content/enterprise_influxdb/v1.9/flux/guides/histograms.md new file mode 100644 index 000000000..663bbfef7 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/histograms.md @@ -0,0 +1,142 @@ +--- +title: Create histograms with Flux +list_title: Histograms +description: > + Use the `histogram()` function to create cumulative histograms with Flux. +menu: + enterprise_influxdb_1_9: + name: Histograms + parent: Query with Flux +weight: 10 +list_query_example: histogram +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/histograms/ +v2: /influxdb/v2.0/query-data/flux/histograms/ +--- + +Histograms provide valuable insight into the distribution of your data. +This guide walks through using Flux's `histogram()` function to transform your data into a **cumulative histogram**. + +## histogram() function +The [`histogram()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/histogram) approximates the +cumulative distribution of a dataset by counting data frequencies for a list of "bins." +A **bin** is simply a range in which a data point falls. +All data points that are less than or equal to the bound are counted in the bin. +In the histogram output, a column is added (le) that represents the upper bounds of of each bin. +Bin counts are cumulative. + +```js +from(bucket:"telegraf/autogen") + |> range(start: -5m) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field == "used_percent" + ) + |> histogram(bins: [0.0, 10.0, 20.0, 30.0]) +``` + +> Values output by the `histogram` function represent points of data aggregated over time. +> Since values do not represent single points in time, there is no `_time` column in the output table. + +## Bin helper functions +Flux provides two helper functions for generating histogram bins. +Each generates and outputs an array of floats designed to be used in the `histogram()` function's `bins` parameter. + +### linearBins() +The [`linearBins()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/misc/linearbins) generates a list of linearly separated floats. + +```js +linearBins(start: 0.0, width: 10.0, count: 10) + +// Generated list: [0, 10, 20, 30, 40, 50, 60, 70, 80, 90, +Inf] +``` + +### logarithmicBins() +The [`logarithmicBins()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/misc/logarithmicbins) generates a list of exponentially separated floats. + +```js +logarithmicBins(start: 1.0, factor: 2.0, count: 10, infinty: true) + +// Generated list: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512, +Inf] +``` + +## Examples + +### Generating a histogram with linear bins +```js +from(bucket:"telegraf/autogen") + |> range(start: -5m) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field == "used_percent" + ) + |> histogram( + bins: linearBins( + start:65.5, + width: 0.5, + count: 20, + infinity:false + ) + ) +``` + +###### Output table +``` +Table: keys: [_start, _stop, _field, _measurement, host] + _start:time _stop:time _field:string _measurement:string host:string le:float _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------ ---------------------------- ---------------------------- +2018-11-07T22:19:58.423658000Z 2018-11-07T22:24:58.423658000Z used_percent mem Scotts-MacBook-Pro.local 65.5 5 +2018-11-07T22:19:58.423658000Z 2018-11-07T22:24:58.423658000Z used_percent mem Scotts-MacBook-Pro.local 66 6 +2018-11-07T22:19:58.423658000Z 2018-11-07T22:24:58.423658000Z used_percent mem Scotts-MacBook-Pro.local 66.5 8 +2018-11-07T22:19:58.423658000Z 2018-11-07T22:24:58.423658000Z used_percent mem Scotts-MacBook-Pro.local 67 9 +2018-11-07T22:19:58.423658000Z 2018-11-07T22:24:58.423658000Z used_percent mem Scotts-MacBook-Pro.local 67.5 9 +2018-11-07T22:19:58.423658000Z 2018-11-07T22:24:58.423658000Z used_percent mem Scotts-MacBook-Pro.local 68 10 +2018-11-07T22:19:58.423658000Z 2018-11-07T22:24:58.423658000Z used_percent mem Scotts-MacBook-Pro.local 68.5 12 +2018-11-07T22:19:58.423658000Z 2018-11-07T22:24:58.423658000Z used_percent mem Scotts-MacBook-Pro.local 69 12 +2018-11-07T22:19:58.423658000Z 2018-11-07T22:24:58.423658000Z used_percent mem Scotts-MacBook-Pro.local 69.5 15 +2018-11-07T22:19:58.423658000Z 2018-11-07T22:24:58.423658000Z used_percent mem Scotts-MacBook-Pro.local 70 23 +2018-11-07T22:19:58.423658000Z 2018-11-07T22:24:58.423658000Z used_percent mem Scotts-MacBook-Pro.local 70.5 30 +2018-11-07T22:19:58.423658000Z 2018-11-07T22:24:58.423658000Z used_percent mem Scotts-MacBook-Pro.local 71 30 +2018-11-07T22:19:58.423658000Z 2018-11-07T22:24:58.423658000Z used_percent mem Scotts-MacBook-Pro.local 71.5 30 +2018-11-07T22:19:58.423658000Z 2018-11-07T22:24:58.423658000Z used_percent mem Scotts-MacBook-Pro.local 72 30 +2018-11-07T22:19:58.423658000Z 2018-11-07T22:24:58.423658000Z used_percent mem Scotts-MacBook-Pro.local 72.5 30 +2018-11-07T22:19:58.423658000Z 2018-11-07T22:24:58.423658000Z used_percent mem Scotts-MacBook-Pro.local 73 30 +2018-11-07T22:19:58.423658000Z 2018-11-07T22:24:58.423658000Z used_percent mem Scotts-MacBook-Pro.local 73.5 30 +2018-11-07T22:19:58.423658000Z 2018-11-07T22:24:58.423658000Z used_percent mem Scotts-MacBook-Pro.local 74 30 +2018-11-07T22:19:58.423658000Z 2018-11-07T22:24:58.423658000Z used_percent mem Scotts-MacBook-Pro.local 74.5 30 +2018-11-07T22:19:58.423658000Z 2018-11-07T22:24:58.423658000Z used_percent mem Scotts-MacBook-Pro.local 75 30 +``` + +### Generating a histogram with logarithmic bins +```js +from(bucket:"telegraf/autogen") + |> range(start: -5m) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field == "used_percent" + ) + |> histogram( + bins: logarithmicBins( + start:0.5, + factor: 2.0, + count: 10, + infinity:false + ) + ) +``` + +###### Output table +``` +Table: keys: [_start, _stop, _field, _measurement, host] + _start:time _stop:time _field:string _measurement:string host:string le:float _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------ ---------------------------- ---------------------------- +2018-11-07T22:23:36.860664000Z 2018-11-07T22:28:36.860664000Z used_percent mem Scotts-MacBook-Pro.local 0.5 0 +2018-11-07T22:23:36.860664000Z 2018-11-07T22:28:36.860664000Z used_percent mem Scotts-MacBook-Pro.local 1 0 +2018-11-07T22:23:36.860664000Z 2018-11-07T22:28:36.860664000Z used_percent mem Scotts-MacBook-Pro.local 2 0 +2018-11-07T22:23:36.860664000Z 2018-11-07T22:28:36.860664000Z used_percent mem Scotts-MacBook-Pro.local 4 0 +2018-11-07T22:23:36.860664000Z 2018-11-07T22:28:36.860664000Z used_percent mem Scotts-MacBook-Pro.local 8 0 +2018-11-07T22:23:36.860664000Z 2018-11-07T22:28:36.860664000Z used_percent mem Scotts-MacBook-Pro.local 16 0 +2018-11-07T22:23:36.860664000Z 2018-11-07T22:28:36.860664000Z used_percent mem Scotts-MacBook-Pro.local 32 0 +2018-11-07T22:23:36.860664000Z 2018-11-07T22:28:36.860664000Z used_percent mem Scotts-MacBook-Pro.local 64 2 +2018-11-07T22:23:36.860664000Z 2018-11-07T22:28:36.860664000Z used_percent mem Scotts-MacBook-Pro.local 128 30 +2018-11-07T22:23:36.860664000Z 2018-11-07T22:28:36.860664000Z used_percent mem Scotts-MacBook-Pro.local 256 30 +``` diff --git a/content/enterprise_influxdb/v1.9/flux/guides/increase.md b/content/enterprise_influxdb/v1.9/flux/guides/increase.md new file mode 100644 index 000000000..1576c696b --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/increase.md @@ -0,0 +1,56 @@ +--- +title: Calculate the increase +seotitle: Calculate the increase in Flux +list_title: Increase +description: > + Use the `increase()` function to track increases across multiple columns in a table. + This function is especially useful when tracking changes in counter values that + wrap over time or periodically reset. +weight: 10 +menu: + enterprise_influxdb_1_9: + parent: Query with Flux + name: Increase +list_query_example: increase +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/increase/ +v2: /influxdb/v2.0/query-data/flux/increase/ +--- + +Use the [`increase()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/increase/) +to track increases across multiple columns in a table. +This function is especially useful when tracking changes in counter values that +wrap over time or periodically reset. + +```js +data + |> increase() +``` + +`increase()` returns a cumulative sum of **non-negative** differences between rows in a table. +For example: + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1 | +| 2020-01-01T00:02:00Z | 2 | +| 2020-01-01T00:03:00Z | 8 | +| 2020-01-01T00:04:00Z | 10 | +| 2020-01-01T00:05:00Z | 0 | +| 2020-01-01T00:06:00Z | 4 | +{{% /flex-content %}} +{{% flex-content %}} +**`increase()` returns:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:02:00Z | 1 | +| 2020-01-01T00:03:00Z | 7 | +| 2020-01-01T00:04:00Z | 9 | +| 2020-01-01T00:05:00Z | 9 | +| 2020-01-01T00:06:00Z | 13 | +{{% /flex-content %}} +{{< /flex >}} diff --git a/content/enterprise_influxdb/v1.9/flux/guides/join.md b/content/enterprise_influxdb/v1.9/flux/guides/join.md new file mode 100644 index 000000000..ba340a00e --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/join.md @@ -0,0 +1,312 @@ +--- +title: Join data with Flux +seotitle: Join data in InfluxDB with Flux +list_title: Join +description: This guide walks through joining data with Flux and outlines how it shapes your data in the process. +menu: + enterprise_influxdb_1_9: + name: Join + parent: Query with Flux +weight: 10 +list_query_example: join +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/join/ +v2: /influxdb/v2.0/query-data/flux/join/ +--- + +The [`join()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/join) merges two or more +input streams, whose values are equal on a set of common columns, into a single output stream. +Flux allows you to join on any columns common between two data streams and opens the door +for operations such as cross-measurement joins and math across measurements. + +To illustrate a join operation, use data captured by Telegraf and and stored in +InfluxDB - memory usage and processes. + +In this guide, we'll join two data streams, one representing memory usage and the other representing the +total number of running processes, then calculate the average memory usage per running process. + +If you're just getting started with Flux queries, check out the following: + +- [Get started with Flux](/enterprise_influxdb/v1.9/flux/get-started/) for a conceptual overview of Flux and parts of a Flux query. +- [Execute queries](/enterprise_influxdb/v1.9/flux/guides/execute-queries/) to discover a variety of ways to run your queries. + +## Define stream variables +In order to perform a join, you must have two streams of data. +Assign a variable to each data stream. + +### Memory used variable +Define a `memUsed` variable that filters on the `mem` measurement and the `used` field. +This returns the amount of memory (in bytes) used. + +###### memUsed stream definition +```js +memUsed = from(bucket: "db/rp") + |> range(start: -5m) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field == "used" + ) +``` + +{{% truncate %}} +###### memUsed data output +``` +Table: keys: [_start, _stop, _field, _measurement, host] + _start:time _stop:time _field:string _measurement:string host:string _time:time _value:int +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------ ------------------------------ -------------------------- +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:50:00.000000000Z 10956333056 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:50:10.000000000Z 11014008832 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:50:20.000000000Z 11373428736 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:50:30.000000000Z 11001421824 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:50:40.000000000Z 10985852928 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:50:50.000000000Z 10992279552 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:51:00.000000000Z 11053568000 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:51:10.000000000Z 11092242432 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:51:20.000000000Z 11612774400 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:51:30.000000000Z 11131961344 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:51:40.000000000Z 11124805632 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:51:50.000000000Z 11332464640 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:52:00.000000000Z 11176923136 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:52:10.000000000Z 11181068288 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:52:20.000000000Z 11182579712 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:52:30.000000000Z 11238862848 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:52:40.000000000Z 11275296768 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:52:50.000000000Z 11225411584 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:53:00.000000000Z 11252690944 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:53:10.000000000Z 11227029504 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:53:20.000000000Z 11201646592 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:53:30.000000000Z 11227897856 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:53:40.000000000Z 11330428928 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:53:50.000000000Z 11347976192 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:54:00.000000000Z 11368271872 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:54:10.000000000Z 11269623808 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:54:20.000000000Z 11295637504 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:54:30.000000000Z 11354423296 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:54:40.000000000Z 11379687424 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:54:50.000000000Z 11248926720 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z used mem host1.local 2018-11-06T05:55:00.000000000Z 11292524544 +``` +{{% /truncate %}} + +### Total processes variable +Define a `procTotal` variable that filters on the `processes` measurement and the `total` field. +This returns the number of running processes. + +###### procTotal stream definition +```js +procTotal = from(bucket: "db/rp") + |> range(start: -5m) + |> filter(fn: (r) => + r._measurement == "processes" and + r._field == "total" + ) +``` + +{{% truncate %}} +###### procTotal data output +``` +Table: keys: [_start, _stop, _field, _measurement, host] + _start:time _stop:time _field:string _measurement:string host:string _time:time _value:int +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------ ------------------------------ -------------------------- +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:50:00.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:50:10.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:50:20.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:50:30.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:50:40.000000000Z 469 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:50:50.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:51:00.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:51:10.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:51:20.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:51:30.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:51:40.000000000Z 469 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:51:50.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:52:00.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:52:10.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:52:20.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:52:30.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:52:40.000000000Z 472 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:52:50.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:53:00.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:53:10.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:53:20.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:53:30.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:53:40.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:53:50.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:54:00.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:54:10.000000000Z 470 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:54:20.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:54:30.000000000Z 473 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:54:40.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:54:50.000000000Z 471 +2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z total processes host1.local 2018-11-06T05:55:00.000000000Z 471 +``` +{{% /truncate %}} + +## Join the two data streams +With the two data streams defined, use the `join()` function to join them together. +`join()` requires two parameters: + +##### `tables` +A map of tables to join with keys by which they will be aliased. +In the example below, `mem` is the alias for `memUsed` and `proc` is the alias for `procTotal`. + +##### `on` +An array of strings defining the columns on which the tables will be joined. +_**Both tables must have all columns specified in this list.**_ + +```js +join( + tables: {mem:memUsed, proc:procTotal}, + on: ["_time", "_stop", "_start", "host"] +) +``` + +{{% truncate %}} +###### Joined output table +``` +Table: keys: [_field_mem, _field_proc, _measurement_mem, _measurement_proc, _start, _stop, host] + _field_mem:string _field_proc:string _measurement_mem:string _measurement_proc:string _start:time _stop:time host:string _time:time _value_mem:int _value_proc:int +---------------------- ---------------------- ----------------------- ------------------------ ------------------------------ ------------------------------ ------------------------ ------------------------------ -------------------------- -------------------------- + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:00.000000000Z 10956333056 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:10.000000000Z 11014008832 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:20.000000000Z 11373428736 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:30.000000000Z 11001421824 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:40.000000000Z 10985852928 469 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:50.000000000Z 10992279552 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:00.000000000Z 11053568000 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:10.000000000Z 11092242432 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:20.000000000Z 11612774400 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:30.000000000Z 11131961344 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:40.000000000Z 11124805632 469 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:50.000000000Z 11332464640 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:00.000000000Z 11176923136 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:10.000000000Z 11181068288 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:20.000000000Z 11182579712 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:30.000000000Z 11238862848 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:40.000000000Z 11275296768 472 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:50.000000000Z 11225411584 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:00.000000000Z 11252690944 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:10.000000000Z 11227029504 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:20.000000000Z 11201646592 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:30.000000000Z 11227897856 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:40.000000000Z 11330428928 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:50.000000000Z 11347976192 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:00.000000000Z 11368271872 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:10.000000000Z 11269623808 470 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:20.000000000Z 11295637504 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:30.000000000Z 11354423296 473 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:40.000000000Z 11379687424 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:50.000000000Z 11248926720 471 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:55:00.000000000Z 11292524544 471 +``` +{{% /truncate %}} + +Notice the output table includes the following columns: + +- `_field_mem` +- `_field_proc` +- `_measurement_mem` +- `_measurement_proc` +- `_value_mem` +- `_value_proc` + +These represent the columns with values unique to the two input tables. + +## Calculate and create a new table +With the two streams of data joined into a single table, use the +[`map()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/map) +to build a new table by mapping the existing `_time` column to a new `_time` +column and dividing `_value_mem` by `_value_proc` and mapping it to a +new `_value` column. + +```js +join(tables: {mem:memUsed, proc:procTotal}, on: ["_time", "_stop", "_start", "host"]) + |> map(fn: (r) => ({ + _time: r._time, + _value: r._value_mem / r._value_proc + }) + ) +``` + +{{% truncate %}} +###### Mapped table +``` +Table: keys: [_field_mem, _field_proc, _measurement_mem, _measurement_proc, _start, _stop, host] + _field_mem:string _field_proc:string _measurement_mem:string _measurement_proc:string _start:time _stop:time host:string _time:time _value:int +---------------------- ---------------------- ----------------------- ------------------------ ------------------------------ ------------------------------ ------------------------ ------------------------------ -------------------------- + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:00.000000000Z 23311346 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:10.000000000Z 23434061 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:20.000000000Z 24147407 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:30.000000000Z 23407280 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:40.000000000Z 23423993 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:50:50.000000000Z 23338173 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:00.000000000Z 23518229 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:10.000000000Z 23600515 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:20.000000000Z 24708030 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:30.000000000Z 23685024 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:40.000000000Z 23720267 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:51:50.000000000Z 24060434 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:00.000000000Z 23730197 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:10.000000000Z 23789506 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:20.000000000Z 23792722 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:30.000000000Z 23861704 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:40.000000000Z 23888340 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:52:50.000000000Z 23833145 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:00.000000000Z 23941895 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:10.000000000Z 23887296 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:20.000000000Z 23833290 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:30.000000000Z 23838424 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:40.000000000Z 24056112 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:53:50.000000000Z 24093367 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:00.000000000Z 24136458 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:10.000000000Z 23977922 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:20.000000000Z 23982245 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:30.000000000Z 24005123 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:40.000000000Z 24160695 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:54:50.000000000Z 23883071 + used total mem processes 2018-11-06T05:50:00.000000000Z 2018-11-06T05:55:00.000000000Z Scotts-MacBook-Pro.local 2018-11-06T05:55:00.000000000Z 23975635 +``` +{{% /truncate %}} + +This table represents the average amount of memory in bytes per running process. + + +## Real world example +The following function calculates the batch sizes written to an InfluxDB cluster by joining +fields from `httpd` and `write` measurements in order to compare `pointReq` and `writeReq`. +The results are grouped by cluster ID so you can make comparisons across clusters. + +```js +batchSize = (cluster_id, start=-1m, interval=10s) => { + httpd = from(bucket:"telegraf") + |> range(start:start) + |> filter(fn:(r) => + r._measurement == "influxdb_httpd" and + r._field == "writeReq" and + r.cluster_id == cluster_id + ) + |> aggregateWindow(every: interval, fn: mean) + |> derivative(nonNegative:true,unit:60s) + + write = from(bucket:"telegraf") + |> range(start:start) + |> filter(fn:(r) => + r._measurement == "influxdb_write" and + r._field == "pointReq" and + r.cluster_id == cluster_id + ) + |> aggregateWindow(every: interval, fn: max) + |> derivative(nonNegative:true,unit:60s) + + return join( + tables:{httpd:httpd, write:write}, + on:["_time","_stop","_start","host"] + ) + |> map(fn:(r) => ({ + _time: r._time, + _value: r._value_httpd / r._value_write, + })) + |> group(columns: cluster_id) +} + +batchSize(cluster_id: "enter cluster id here") +``` diff --git a/content/enterprise_influxdb/v1.9/flux/guides/manipulate-timestamps.md b/content/enterprise_influxdb/v1.9/flux/guides/manipulate-timestamps.md new file mode 100644 index 000000000..0b87a2f2c --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/manipulate-timestamps.md @@ -0,0 +1,189 @@ +--- +title: Manipulate timestamps with Flux +list_title: Manipulate timestamps +description: > + Use Flux to process and manipulate timestamps. +menu: + enterprise_influxdb_1_9: + name: Manipulate timestamps + parent: Query with Flux +weight: 20 +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/manipulate-timestamps/ +v2: /influxdb/v2.0/query-data/flux/manipulate-timestamps/ +--- + +Every point stored in InfluxDB has an associated timestamp. +Use Flux to process and manipulate timestamps to suit your needs. + +- [Convert timestamp format](#convert-timestamp-format) +- [Calculate the duration between two timestamps](#calculate-the-duration-between-two-timestamps) +- [Retrieve the current time](#retrieve-the-current-time) +- [Normalize irregular timestamps](#normalize-irregular-timestamps) +- [Use timestamps and durations together](#use-timestamps-and-durations-together) + +{{% note %}} +If you're just getting started with Flux queries, check out the following: + +- [Get started with Flux](/enterprise_influxdb/v1.9/flux/get-started/) for a conceptual overview of Flux and parts of a Flux query. +- [Execute queries](/enterprise_influxdb/v1.9/flux/guides/execute-queries/) to discover a variety of ways to run your queries. +{{% /note %}} + + +## Convert timestamp format + +- [Unix nanosecond to RFC3339](#unix-nanosecond-to-rfc3339) +- [RFC3339 to Unix nanosecond](#rfc3339-to-unix-nanosecond) + +### Unix nanosecond to RFC3339 +Use the [`time()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/type-conversions/time/) +to convert a [Unix **nanosecond** timestamp](/{{< latest "influxdb" "v2" >}}/reference/glossary/#unix-timestamp) +to an [RFC3339 timestamp](/{{< latest "influxdb" "v2" >}}/reference/glossary/#rfc3339-timestamp). + +```js +time(v: 1568808000000000000) +// Returns 2019-09-18T12:00:00.000000000Z +``` + +### RFC3339 to Unix nanosecond +Use the [`uint()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/type-conversions/uint/) +to convert an RFC3339 timestamp to a Unix nanosecond timestamp. + +```js +uint(v: 2019-09-18T12:00:00.000000000Z) +// Returns 1568808000000000000 +``` + +## Calculate the duration between two timestamps +Flux doesn't support mathematical operations using [time type](/{{< latest "influxdb" "v2" >}}/reference/flux/language/types/#time-types) values. +To calculate the duration between two timestamps: + +1. Use the `uint()` function to convert each timestamp to a Unix nanosecond timestamp. +2. Subtract one Unix nanosecond timestamp from the other. +3. Use the `duration()` function to convert the result into a duration. + +```js +time1 = uint(v: 2019-09-17T21:12:05Z) +time2 = uint(v: 2019-09-18T22:16:35Z) + +duration(v: time2 - time1) +// Returns 25h4m30s +``` + +{{% note %}} +Flux doesn't support duration column types. +To store a duration in a column, use the [`string()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/type-conversions/string/) +to convert the duration to a string. +{{% /note %}} + +## Retrieve the current time +- [Current UTC time](#current-utc-time) +- [Current system time](#current-system-time) + +### Current UTC time +Use the [`now()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/misc/now/) to +return the current UTC time in RFC3339 format. + +```js +now() +``` + +{{% note %}} +`now()` is cached at runtime, so all instances of `now()` in a Flux script +return the same value. +{{% /note %}} + +### Current system time +Import the `system` package and use the [`system.time()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/system/time/) +to return the current system time of the host machine in RFC3339 format. + +```js +import "system" + +system.time() +``` + +{{% note %}} +`system.time()` returns the time it is executed, so each instance of `system.time()` +in a Flux script returns a unique value. +{{% /note %}} + +## Normalize irregular timestamps +To normalize irregular timestamps, truncate all `_time` values to a specified unit +with the [`truncateTimeColumn()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/truncatetimecolumn/). +This is useful in [`join()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/join/) +and [`pivot()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/pivot/) +operations where points should align by time, but timestamps vary slightly. + +```js +data + |> truncateTimeColumn(unit: 1m) +``` + +{{< flex >}} +{{% flex-content %}} +**Input:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:00:49Z | 2.0 | +| 2020-01-01T00:01:01Z | 1.9 | +| 2020-01-01T00:03:22Z | 1.8 | +| 2020-01-01T00:04:04Z | 1.9 | +| 2020-01-01T00:05:38Z | 2.1 | +{{% /flex-content %}} +{{% flex-content %}} +**Output:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:00:00Z | 2.0 | +| 2020-01-01T00:01:00Z | 1.9 | +| 2020-01-01T00:03:00Z | 1.8 | +| 2020-01-01T00:04:00Z | 1.9 | +| 2020-01-01T00:05:00Z | 2.1 | +{{% /flex-content %}} +{{< /flex >}} + +## Use timestamps and durations together +- [Add a duration to a timestamp](#add-a-duration-to-a-timestamp) +- [Subtract a duration from a timestamp](#subtract-a-duration-from-a-timestamp) + +### Add a duration to a timestamp +The [`experimental.addDuration()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/addduration/) +adds a duration to a specified time and returns the resulting time. + +{{% warn %}} +By using `experimental.addDuration()`, you accept the +[risks of experimental functions](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/#use-experimental-functions-at-your-own-risk). +{{% /warn %}} + +```js +import "experimental" + +experimental.addDuration( + d: 6h, + to: 2019-09-16T12:00:00Z, +) + +// Returns 2019-09-16T18:00:00.000000000Z +``` + +### Subtract a duration from a timestamp +The [`experimental.subDuration()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/subduration/) +subtracts a duration from a specified time and returns the resulting time. + +{{% warn %}} +By using `experimental.subDuration()`, you accept the +[risks of experimental functions](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/#use-experimental-functions-at-your-own-risk). +{{% /warn %}} + +```js +import "experimental" + +experimental.subDuration( + d: 6h, + from: 2019-09-16T12:00:00Z, +) + +// Returns 2019-09-16T06:00:00.000000000Z +``` diff --git a/content/enterprise_influxdb/v1.9/flux/guides/mathematic-operations.md b/content/enterprise_influxdb/v1.9/flux/guides/mathematic-operations.md new file mode 100644 index 000000000..e0b9b2765 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/mathematic-operations.md @@ -0,0 +1,227 @@ +--- +title: Transform data with mathematic operations +seotitle: Transform data with mathematic operations in Flux +list_title: Transform data with math +description: > + Use the `map()` function to remap column values and apply mathematic operations. +menu: + enterprise_influxdb_1_9: + name: Transform data with math + parent: Query with Flux +weight: 5 +list_query_example: map_math +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/mathematic-operations/ +v2: /influxdb/v2.0/query-data/flux/mathematic-operations/ +--- + +Flux supports mathematic expressions in data transformations. +This article describes how to use [Flux arithmetic operators](/{{< latest "influxdb" "v2" >}}/reference/flux/language/operators/#arithmetic-operators) +to "map" over data and transform values using mathematic operations. + +If you're just getting started with Flux queries, check out the following: + +- [Get started with Flux](/enterprise_influxdb/v1.9/flux/get-started/) for a conceptual overview of Flux and parts of a Flux query. +- [Execute queries](/enterprise_influxdb/v1.9/flux/guides/execute-queries/) to discover a variety of ways to run your queries. + +##### Basic mathematic operations +```js +// Examples executed using the Flux REPL +> 9 + 9 +18 +> 22 - 14 +8 +> 6 * 5 +30 +> 21 / 7 +3 +``` + +

        See Flux read-eval-print-loop (REPL).

        + +{{% note %}} +#### Operands must be the same type +Operands in Flux mathematic operations must be the same data type. +For example, integers cannot be used in operations with floats. +Otherwise, you will get an error similar to: + +``` +Error: type error: float != int +``` + +To convert operands to the same type, use [type-conversion functions](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/type-conversions/) +or manually format operands. +The operand data type determines the output data type. +For example: + +```js +100 // Parsed as an integer +100.0 // Parsed as a float + +// Example evaluations +> 20 / 8 +2 + +> 20.0 / 8.0 +2.5 +``` +{{% /note %}} + +## Custom mathematic functions +Flux lets you [create custom functions](/{{< latest "influxdb" "v2" >}}/query-data/flux/custom-functions) that use mathematic operations. +View the examples below. + +###### Custom multiplication function +```js +multiply = (x, y) => x * y + +multiply(x: 10, y: 12) +// Returns 120 +``` + +###### Custom percentage function +```js +percent = (sample, total) => (sample / total) * 100.0 + +percent(sample: 20.0, total: 80.0) +// Returns 25.0 +``` + +### Transform values in a data stream +To transform multiple values in an input stream, your function needs to: + +- [Handle piped-forward data](/{{< latest "influxdb" "v2" >}}/query-data/flux/custom-functions/#functions-that-manipulate-piped-forward-data). +- Each operand necessary for the calculation exists in each row _(see [Pivot vs join](#pivot-vs-join) below)_. +- Use the [`map()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/map) to iterate over each row. + +The example `multiplyByX()` function below includes: + +- A `tables` parameter that represents the input data stream (`<-`). +- An `x` parameter which is the number by which values in the `_value` column are multiplied. +- A `map()` function that iterates over each row in the input stream. + It uses the `with` operator to preserve existing columns in each row. + It also multiples the `_value` column by `x`. + +```js +multiplyByX = (x, tables=<-) => + tables + |> map(fn: (r) => ({ + r with + _value: r._value * x + }) + ) + +data + |> multiplyByX(x: 10) +``` + +## Examples + +### Convert bytes to gigabytes +To convert active memory from bytes to gigabytes (GB), divide the `active` field +in the `mem` measurement by 1,073,741,824. + +The `map()` function iterates over each row in the piped-forward data and defines +a new `_value` by dividing the original `_value` by 1073741824. + +```js +from(bucket: "db/rp") + |> range(start: -10m) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field == "active" + ) + |> map(fn: (r) => ({ + r with + _value: r._value / 1073741824 + }) + ) +``` + +You could turn that same calculation into a function: + +```js +bytesToGB = (tables=<-) => + tables + |> map(fn: (r) => ({ + r with + _value: r._value / 1073741824 + }) + ) + +data + |> bytesToGB() +``` + +#### Include partial gigabytes +Because the original metric (bytes) is an integer, the output of the operation is an integer and does not include partial GBs. +To calculate partial GBs, convert the `_value` column and its values to floats using the +[`float()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/type-conversions/float) +and format the denominator in the division operation as a float. + +```js +bytesToGB = (tables=<-) => + tables + |> map(fn: (r) => ({ + r with + _value: float(v: r._value) / 1073741824.0 + }) + ) +``` + +### Calculate a percentage +To calculate a percentage, use simple division, then multiply the result by 100. + +```js +> 1.0 / 4.0 * 100.0 +25.0 +``` + +_For an in-depth look at calculating percentages, see [Calculate percentates](/enterprise_influxdb/v1.9/flux/guides/calculate-percentages)._ + +## Pivot vs join +To query and use values in mathematical operations in Flux, operand values must +exists in a single row. +Both `pivot()` and `join()` will do this, but there are important differences between the two: + +#### Pivot is more performant +`pivot()` reads and operates on a single stream of data. +`join()` requires two streams of data and the overhead of reading and combining +both streams can be significant, especially for larger data sets. + +#### Use join for multiple data sources +Use `join()` when querying data from different buckets or data sources. + +##### Pivot fields into columns for mathematic calculations +```js +data + |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value") + |> map(fn: (r) => ({ r with + _value: (r.field1 + r.field2) / r.field3 * 100.0 + })) +``` + +##### Join multiple data sources for mathematic calculations +```js +import "sql" +import "influxdata/influxdb/secrets" + +pgUser = secrets.get(key: "POSTGRES_USER") +pgPass = secrets.get(key: "POSTGRES_PASSWORD") +pgHost = secrets.get(key: "POSTGRES_HOST") + +t1 = sql.from( + driverName: "postgres", + dataSourceName: "postgresql://${pgUser}:${pgPass}@${pgHost}", + query:"SELECT id, name, available FROM exampleTable" +) + +t2 = from(bucket: "db/rp") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "example-measurement" and + r._field == "example-field" + ) + +join(tables: {t1: t1, t2: t2}, on: ["id"]) + |> map(fn: (r) => ({ r with _value: r._value_t2 / r.available_t1 * 100.0 })) +``` diff --git a/content/enterprise_influxdb/v1.9/flux/guides/median.md b/content/enterprise_influxdb/v1.9/flux/guides/median.md new file mode 100644 index 000000000..dd7b10ad5 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/median.md @@ -0,0 +1,146 @@ +--- +title: Find median values +seotitle: Find median values in Flux +list_title: Median +description: > + Use the `median()` function to return a value representing the `0.5` quantile (50th percentile) or median of input data. +weight: 10 +menu: + enterprise_influxdb_1_9: + parent: Query with Flux + name: Median +list_query_example: median +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/median/ +v2: /influxdb/v2.0/query-data/flux/median/ +--- + +Use the [`median()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/median/) +to return a value representing the `0.5` quantile (50th percentile) or median of input data. + +## Select a method for calculating the median +Select one of the following methods to calculate the median: + +- [estimate_tdigest](#estimate-tdigest) +- [exact_mean](#exact-mean) +- [exact_selector](#exact-selector) + +### estimate_tdigest +**(Default)** An aggregate method that uses a [t-digest data structure](https://github.com/tdunning/t-digest) +to compute an accurate `0.5` quantile estimate on large data sources. +Output tables consist of a single row containing the calculated median. + +{{< flex >}} +{{% flex-content %}} +**Given the following input table:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.0 | +| 2020-01-01T00:03:00Z | 2.0 | +| 2020-01-01T00:04:00Z | 3.0 | +{{% /flex-content %}} +{{% flex-content %}} +**`estimate_tdigest` returns:** + +| _value | +|:------:| +| 1.5 | +{{% /flex-content %}} +{{< /flex >}} + +### exact_mean +An aggregate method that takes the average of the two points closest to the `0.5` quantile value. +Output tables consist of a single row containing the calculated median. + +{{< flex >}} +{{% flex-content %}} +**Given the following input table:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.0 | +| 2020-01-01T00:03:00Z | 2.0 | +| 2020-01-01T00:04:00Z | 3.0 | +{{% /flex-content %}} +{{% flex-content %}} +**`exact_mean` returns:** + +| _value | +|:------:| +| 1.5 | +{{% /flex-content %}} +{{< /flex >}} + +### exact_selector +A selector method that returns the data point for which at least 50% of points are less than. +Output tables consist of a single row containing the calculated median. + +{{< flex >}} +{{% flex-content %}} +**Given the following input table:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.0 | +| 2020-01-01T00:03:00Z | 2.0 | +| 2020-01-01T00:04:00Z | 3.0 | +{{% /flex-content %}} +{{% flex-content %}} +**`exact_selector` returns:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:02:00Z | 1.0 | +{{% /flex-content %}} +{{< /flex >}} + +{{% note %}} +The examples below use the [example data variable](/enterprise_influxdb/v1.9/flux/guides/#example-data-variable). +{{% /note %}} + +## Find the value that represents the median +Use the default method, `"estimate_tdigest"`, to return all rows in a table that +contain values in the 50th percentile of data in the table. + +```js +data + |> median() +``` + +## Find the average of values closest to the median +Use the `exact_mean` method to return a single row per input table containing the +average of the two values closest to the mathematical median of data in the table. + +```js +data + |> median(method: "exact_mean") +``` + +## Find the point with the median value +Use the `exact_selector` method to return a single row per input table containing the +value that 50% of values in the table are less than. + +```js +data + |> median(method: "exact_selector") +``` + +## Use median() with aggregateWindow() +[`aggregateWindow()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/aggregatewindow/) +segments data into windows of time, aggregates data in each window into a single +point, and then removes the time-based segmentation. +It is primarily used to downsample data. + +To specify the [median calculation method](#select-a-method-for-calculating-the-median) in `aggregateWindow()`, use the +[full function syntax](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/aggregatewindow/#specify-parameters-of-the-aggregate-function): + +```js +data + |> aggregateWindow( + every: 5m, + fn: (tables=<-, column) => tables |> median(method: "exact_selector") + ) +``` diff --git a/content/enterprise_influxdb/v1.9/flux/guides/monitor-states.md b/content/enterprise_influxdb/v1.9/flux/guides/monitor-states.md new file mode 100644 index 000000000..3a82cbdeb --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/monitor-states.md @@ -0,0 +1,151 @@ +--- +title: Monitor states +seotitle: Monitor states and state changes in your events and metrics with Flux. +description: Flux provides several functions to help monitor states and state changes in your data. +menu: + enterprise_influxdb_1_9: + name: Monitor states + parent: Query with Flux +weight: 20 +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/monitor-states/ +v2: /influxdb/v2.0/query-data/flux/monitor-states/ +--- + +Flux helps you monitor states in your metrics and events: + +- [Find how long a state persists](#find-how-long-a-state-persists) +- [Count the number of consecutive states](#count-the-number-of-consecutive-states) +- [Detect state changes](#example-query-to-count-machine-state) + +If you're just getting started with Flux queries, check out the following: + +- [Get started with Flux](/enterprise_influxdb/v1.9/flux/get-started/) for a conceptual overview of Flux. +- [Execute queries](/enterprise_influxdb/v1.9/flux/guides/executing-queries/) to discover a variety of ways to run your queries. + +## Find how long a state persists + +1. Use the [`stateDuration()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/stateduration/) function to calculate how long a column value has remained the same value (or state). Include the following information: + + - **Column to search:** any tag key, tag value, field key, field value, or measurement. + - **Value:** the value (or state) to search for in the specified column. + - **State duration column:** a new column to store the state duration─the length of time that the specified value persists. + - **Unit:** the unit of time (`1s` (by default), `1m`, `1h`) used to increment the state duration. + + + ```js + |> stateDuration( + fn: (r) => + r._column_to_search == "value_to_search_for", + column: "state_duration", + unit: 1s + ) + ``` + +2. Use `stateDuration()` to search each point for the specified value: + + - For the first point that evaluates `true`, the state duration is set to `0`. For each consecutive point that evaluates `true`, the state duration increases by the time interval between each consecutive point (in specified units). + - If the state is `false`, the state duration is reset to `-1`. + +### Example query with stateDuration() + +The following query searches the `doors` bucket over the past 5 minutes to find how many seconds a door has been `closed`. + +```js +from(bucket: "doors") + |> range(start: -5m) + |> stateDuration( + fn: (r) => + r._value == "closed", + column: "door_closed", + unit: 1s + ) +``` + +In this example, `door_closed` is the **State duration** column. If you write data to the `doors` bucket every minute, the state duration increases by `60s` for each consecutive point where `_value` is `closed`. If `_value` is not `closed`, the state duration is reset to `0`. + +#### Query results + +Results for the example query above may look like this (for simplicity, we've omitted the measurement, tag, and field columns): + +```bash +_time _value door_closed +2019-10-26T17:39:16Z closed 0 +2019-10-26T17:40:16Z closed 60 +2019-10-26T17:41:16Z closed 120 +2019-10-26T17:42:16Z open -1 +2019-10-26T17:43:16Z closed 0 +2019-10-26T17:44:27Z closed 60 +``` + +## Count the number of consecutive states + +1. Use the `stateCount()` function and include the following information: + + - **Column to search:** any tag key, tag value, field key, field value, or measurement. + - **Value:** to search for in the specified column. + - **State count column:** a new column to store the state count─the number of consecutive records in which the specified value exists. + + + ```js + |> stateCount + (fn: (r) => + r._column_to_search == "value_to_search_for", + column: "state_count" + ) + ``` + +2. Use `stateCount()` to search each point for the specified value: + + - For the first point that evaluates `true`, the state count is set to `1`. For each consecutive point that evaluates `true`, the state count increases by 1. + - If the state is `false`, the state count is reset to `-1`. + +### Example query with stateCount() + +The following query searches the `doors` bucket over the past 5 minutes and +calculates how many points have `closed` as their `_value`. + +```js +from(bucket: "doors") + |> range(start: -5m) + |> stateDuration( + fn: (r) => + r._value == "closed", + column: "door_closed") +``` + +This example stores the **state count** in the `door_closed` column. +If you write data to the `doors` bucket every minute, the state count increases +by `1` for each consecutive point where `_value` is `closed`. +If `_value` is not `closed`, the state count is reset to `-1`. + +#### Query results + +Results for the example query above may look like this (for simplicity, we've omitted the measurement, tag, and field columns): + +```bash +_time _value door_closed +2019-10-26T17:39:16Z closed 1 +2019-10-26T17:40:16Z closed 2 +2019-10-26T17:41:16Z closed 3 +2019-10-26T17:42:16Z open -1 +2019-10-26T17:43:16Z closed 1 +2019-10-26T17:44:27Z closed 2 +``` + +#### Example query to count machine state + +The following query checks the machine state every minute (idle, assigned, or busy). +InfluxDB searches the `servers` bucket over the past hour and counts records with a machine state of `idle`, `assigned` or `busy`. + +```js +from(bucket: "servers") + |> range(start: -1h) + |> filter(fn: (r) => + r.machine_state == "idle" or + r.machine_state == "assigned" or + r.machine_state == "busy" + ) + |> stateCount(fn: (r) => r.machine_state == "busy", column: "_count") + |> stateCount(fn: (r) => r.machine_state == "assigned", column: "_count") + |> stateCount(fn: (r) => r.machine_state == "idle", column: "_count") +``` diff --git a/content/enterprise_influxdb/v1.9/flux/guides/moving-average.md b/content/enterprise_influxdb/v1.9/flux/guides/moving-average.md new file mode 100644 index 000000000..f7a8da27b --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/moving-average.md @@ -0,0 +1,115 @@ +--- +title: Calculate the moving average +seotitle: Calculate the moving average in Flux +list_title: Moving Average +description: > + Use the `movingAverage()` or `timedMovingAverage()` functions to return the moving average of data. +weight: 10 +menu: + enterprise_influxdb_1_9: + parent: Query with Flux + name: Moving Average +list_query_example: moving_average +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/moving-average/ +v2: /influxdb/v2.0/query-data/flux/moving-average/ +--- + +Use the [`movingAverage()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/movingaverage/) +or [`timedMovingAverage()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/timedmovingaverage/) +functions to return the moving average of data. + +```js +data + |> movingAverage(n: 5) + +// OR + +data + |> timedMovingAverage(every: 5m, period: 10m) +``` + +### movingAverage() +For each row in a table, `movingAverage()` returns the average of the current value and +**previous** values where `n` is the total number of values used to calculate the average. + +If `n = 3`: + +| Row # | Calculation | +|:-----:|:----------- | +| 1 | _Insufficient number of rows_ | +| 2 | _Insufficient number of rows_ | +| 3 | (Row1 + Row2 + Row3) / 3 | +| 4 | (Row2 + Row3 + Row4) / 3 | +| 5 | (Row3 + Row4 + Row5) / 3 | + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.2 | +| 2020-01-01T00:03:00Z | 1.8 | +| 2020-01-01T00:04:00Z | 0.9 | +| 2020-01-01T00:05:00Z | 1.4 | +| 2020-01-01T00:06:00Z | 2.0 | +{{% /flex-content %}} +{{% flex-content %}} +**The following would return:** + +```js +|> movingAverage(n: 3) +``` + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:03:00Z | 1.33 | +| 2020-01-01T00:04:00Z | 1.30 | +| 2020-01-01T00:05:00Z | 1.36 | +| 2020-01-01T00:06:00Z | 1.43 | +{{% /flex-content %}} +{{< /flex >}} + +### timedMovingAverage() +For each row in a table, `timedMovingAverage()` returns the average of the +current value and all row values in the **previous** `period` (duration). +It returns moving averages at a frequency defined by the `every` parameter. + +Each color in the diagram below represents a period of time used to calculate an +average and the time a point representing the average is returned. +If `every = 30m` and `period = 1h`: + +{{< svg "/static/svgs/timed-moving-avg.svg" >}} + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.2 | +| 2020-01-01T00:03:00Z | 1.8 | +| 2020-01-01T00:04:00Z | 0.9 | +| 2020-01-01T00:05:00Z | 1.4 | +| 2020-01-01T00:06:00Z | 2.0 | +{{% /flex-content %}} +{{% flex-content %}} +**The following would return:** + +```js +|> timedMovingAverage( + every: 2m, + period: 4m +) +``` + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:02:00Z | 1.000 | +| 2020-01-01T00:04:00Z | 1.333 | +| 2020-01-01T00:06:00Z | 1.325 | +| 2020-01-01T00:06:00Z | 1.150 | +{{% /flex-content %}} +{{< /flex >}} diff --git a/content/enterprise_influxdb/v1.9/flux/guides/percentile-quantile.md b/content/enterprise_influxdb/v1.9/flux/guides/percentile-quantile.md new file mode 100644 index 000000000..64e9925e0 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/percentile-quantile.md @@ -0,0 +1,163 @@ +--- +title: Find percentile and quantile values +seotitle: Query percentile and quantile values in Flux +list_title: Percentile & quantile +description: > + Use the `quantile()` function to return all values within the `q` quantile or + percentile of input data. +weight: 10 +menu: + enterprise_influxdb_1_9: + parent: Query with Flux + name: Percentile & quantile +list_query_example: quantile +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/percentile-quantile/ +v2: /influxdb/v2.0/query-data/flux/percentile-quantile/ +--- + +Use the [`quantile()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/quantile/) +to return a value representing the `q` quantile or percentile of input data. + +## Percentile versus quantile +Percentiles and quantiles are very similar, differing only in the number used to calculate return values. +A percentile is calculated using numbers between `0` and `100`. +A quantile is calculated using numbers between `0.0` and `1.0`. +For example, the **`0.5` quantile** is the same as the **50th percentile**. + +## Select a method for calculating the quantile +Select one of the following methods to calculate the quantile: + +- [estimate_tdigest](#estimate-tdigest) +- [exact_mean](#exact-mean) +- [exact_selector](#exact-selector) + +### estimate_tdigest +**(Default)** An aggregate method that uses a [t-digest data structure](https://github.com/tdunning/t-digest) +to compute a quantile estimate on large data sources. +Output tables consist of a single row containing the calculated quantile. + +If calculating the `0.5` quantile or 50th percentile: + +{{< flex >}} +{{% flex-content %}} +**Given the following input table:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.0 | +| 2020-01-01T00:03:00Z | 2.0 | +| 2020-01-01T00:04:00Z | 3.0 | +{{% /flex-content %}} +{{% flex-content %}} +**`estimate_tdigest` returns:** + +| _value | +|:------:| +| 1.5 | +{{% /flex-content %}} +{{< /flex >}} + +### exact_mean +An aggregate method that takes the average of the two points closest to the quantile value. +Output tables consist of a single row containing the calculated quantile. + +If calculating the `0.5` quantile or 50th percentile: + +{{< flex >}} +{{% flex-content %}} +**Given the following input table:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.0 | +| 2020-01-01T00:03:00Z | 2.0 | +| 2020-01-01T00:04:00Z | 3.0 | +{{% /flex-content %}} +{{% flex-content %}} +**`exact_mean` returns:** + +| _value | +|:------:| +| 1.5 | +{{% /flex-content %}} +{{< /flex >}} + +### exact_selector +A selector method that returns the data point for which at least `q` points are less than. +Output tables consist of a single row containing the calculated quantile. + +If calculating the `0.5` quantile or 50th percentile: + +{{< flex >}} +{{% flex-content %}} +**Given the following input table:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:01:00Z | 1.0 | +| 2020-01-01T00:02:00Z | 1.0 | +| 2020-01-01T00:03:00Z | 2.0 | +| 2020-01-01T00:04:00Z | 3.0 | +{{% /flex-content %}} +{{% flex-content %}} +**`exact_selector` returns:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:02:00Z | 1.0 | +{{% /flex-content %}} +{{< /flex >}} + +{{% note %}} +The examples below use the [example data variable](/enterprise_influxdb/v1.9/flux/guides/#example-data-variable). +{{% /note %}} + +## Find the value representing the 99th percentile +Use the default method, `"estimate_tdigest"`, to return all rows in a table that +contain values in the 99th percentile of data in the table. + +```js +data + |> quantile(q: 0.99) +``` + +## Find the average of values closest to the quantile +Use the `exact_mean` method to return a single row per input table containing the +average of the two values closest to the mathematical quantile of data in the table. +For example, to calculate the `0.99` quantile: + +```js +data + |> quantile(q: 0.99, method: "exact_mean") +``` + +## Find the point with the quantile value +Use the `exact_selector` method to return a single row per input table containing the +value that `q * 100`% of values in the table are less than. +For example, to calculate the `0.99` quantile: + +```js +data + |> quantile(q: 0.99, method: "exact_selector") +``` + +## Use quantile() with aggregateWindow() +[`aggregateWindow()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/aggregatewindow/) +segments data into windows of time, aggregates data in each window into a single +point, and then removes the time-based segmentation. +It is primarily used to downsample data. + +To specify the [quantile calculation method](#select-a-method-for-calculating-the-quantile) in +`aggregateWindow()`, use the [full function syntax](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/aggregatewindow/#specify-parameters-of-the-aggregate-function): + +```js +data + |> aggregateWindow( + every: 5m, + fn: (tables=<-, column) => + tables + |> quantile(q: 0.99, method: "exact_selector") + ) +``` diff --git a/content/enterprise_influxdb/v1.9/flux/guides/query-fields.md b/content/enterprise_influxdb/v1.9/flux/guides/query-fields.md new file mode 100644 index 000000000..0b043475b --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/query-fields.md @@ -0,0 +1,78 @@ +--- +title: Query fields and tags +seotitle: Query fields and tags in InfluxDB using Flux +description: > + Use the `filter()` function to query data based on fields, tags, or any other column value. + `filter()` performs operations similar to the `SELECT` statement and the `WHERE` + clause in InfluxQL and other SQL-like query languages. +weight: 1 +menu: + enterprise_influxdb_1_9: + parent: Query with Flux +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/query-fields/ +v2: /influxdb/v2.0/query-data/flux/query-fields/ +list_code_example: | + ```js + from(bucket: "db/rp") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "example-measurement" and + r._field == "example-field" and + r.tag == "example-tag" + ) + ``` +--- + +Use the [`filter()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/filter/) +to query data based on fields, tags, or any other column value. +`filter()` performs operations similar to the `SELECT` statement and the `WHERE` +clause in InfluxQL and other SQL-like query languages. + +## The filter() function +`filter()` has an `fn` parameter that expects a **predicate function**, +an anonymous function comprised of one or more **predicate expressions**. +The predicate function evaluates each input row. +Rows that evaluate to `true` are **included** in the output data. +Rows that evaluate to `false` are **excluded** from the output data. + +```js +// ... + |> filter(fn: (r) => r._measurement == "example-measurement" ) +``` + +The `fn` predicate function requires an `r` argument, which represents each row +as `filter()` iterates over input data. +Key-value pairs in the row record represent columns and their values. +Use **dot notation** or **bracket notation** to reference specific column values in the predicate function. +Use [logical operators](/{{< latest "influxdb" "v2" >}}/reference/flux/language/operators/#logical-operators) +to chain multiple predicate expressions together. + +```js +// Row record +r = {foo: "bar", baz: "quz"} + +// Example predicate function +(r) => r.foo == "bar" and r["baz"] == "quz" + +// Evaluation results +(r) => true and true +``` + +## Filter by fields and tags +The combination of [`from()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/inputs/from), +[`range()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/range), +and `filter()` represent the most basic Flux query: + +1. Use `from()` to define your [bucket](/enterprise_influxdb/v1.9/flux/get-started/#buckets). +2. Use `range()` to limit query results by time. +3. Use `filter()` to identify what rows of data to output. + +```js +from(bucket: "db/rp") + |> range(start: -1h) + |> filter(fn: (r) => + r._measurement == "example-measurement" and + r._field == "example-field" and + r.tag == "example-tag" + ) +``` diff --git a/content/enterprise_influxdb/v1.9/flux/guides/rate.md b/content/enterprise_influxdb/v1.9/flux/guides/rate.md new file mode 100644 index 000000000..b44573942 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/rate.md @@ -0,0 +1,175 @@ +--- +title: Calculate the rate of change +seotitle: Calculate the rate of change in Flux +list_title: Rate +description: > + Use the `derivative()` function to calculate the rate of change between subsequent values or the + `aggregate.rate()` function to calculate the average rate of change per window of time. + If time between points varies, these functions normalize points to a common time interval + making values easily comparable. +weight: 10 +menu: + enterprise_influxdb_1_9: + parent: Query with Flux + name: Rate +list_query_example: rate_of_change +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/rate/ +v2: /influxdb/v2.0/query-data/flux/rate/ +--- + + +Use the [`derivative()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/derivative/) +to calculate the rate of change between subsequent values or the +[`aggregate.rate()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/aggregate/rate/) +to calculate the average rate of change per window of time. +If time between points varies, these functions normalize points to a common time interval +making values easily comparable. + +- [Rate of change between subsequent values](#rate-of-change-between-subsequent-values) +- [Average rate of change per window of time](#average-rate-of-change-per-window-of-time) + +## Rate of change between subsequent values +Use the [`derivative()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/derivative/) +to calculate the rate of change per unit of time between subsequent _non-null_ values. + +```js +data + |> derivative(unit: 1s) +``` + +By default, `derivative()` returns only positive derivative values and replaces negative values with _null_. +Cacluated values are returned as [floats](/{{< latest "influxdb" "v2" >}}/reference/flux/language/types/#numeric-types). + + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:00:00Z | 250 | +| 2020-01-01T00:04:00Z | 160 | +| 2020-01-01T00:12:00Z | 150 | +| 2020-01-01T00:19:00Z | 220 | +| 2020-01-01T00:32:00Z | 200 | +| 2020-01-01T00:51:00Z | 290 | +| 2020-01-01T01:00:00Z | 340 | +{{% /flex-content %}} +{{% flex-content %}} +**`derivative(unit: 1m)` returns:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:04:00Z | | +| 2020-01-01T00:12:00Z | | +| 2020-01-01T00:19:00Z | 10.0 | +| 2020-01-01T00:32:00Z | | +| 2020-01-01T00:51:00Z | 4.74 | +| 2020-01-01T01:00:00Z | 5.56 | +{{% /flex-content %}} +{{< /flex >}} + +Results represent the rate of change **per minute** between subsequent values with +negative values set to _null_. + +### Return negative derivative values +To return negative derivative values, set the `nonNegative` parameter to `false`, + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:00:00Z | 250 | +| 2020-01-01T00:04:00Z | 160 | +| 2020-01-01T00:12:00Z | 150 | +| 2020-01-01T00:19:00Z | 220 | +| 2020-01-01T00:32:00Z | 200 | +| 2020-01-01T00:51:00Z | 290 | +| 2020-01-01T01:00:00Z | 340 | +{{% /flex-content %}} +{{% flex-content %}} +**The following returns:** + +```js +|> derivative( + unit: 1m, + nonNegative: false +) +``` + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:04:00Z | -22.5 | +| 2020-01-01T00:12:00Z | -1.25 | +| 2020-01-01T00:19:00Z | 10.0 | +| 2020-01-01T00:32:00Z | -1.54 | +| 2020-01-01T00:51:00Z | 4.74 | +| 2020-01-01T01:00:00Z | 5.56 | +{{% /flex-content %}} +{{< /flex >}} + +Results represent the rate of change **per minute** between subsequent values and +include negative values. + +## Average rate of change per window of time + +Use the [`aggregate.rate()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/experimental/aggregate/rate/) +to calculate the average rate of change per window of time. + +```js +import "experimental/aggregate" + +data + |> aggregate.rate( + every: 1m, + unit: 1s, + groupColumns: ["tag1", "tag2"] + ) +``` + +`aggregate.rate()` returns the average rate of change (as a [float](/{{< latest "influxdb" "v2" >}}/reference/flux/language/types/#numeric-types)) +per `unit` for time intervals defined by `every`. +Negative values are replaced with _null_. + +{{% note %}} +`aggregate.rate()` does not support `nonNegative: false`. +{{% /note %}} + +{{< flex >}} +{{% flex-content %}} +**Given the following input:** + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:00:00Z | 250 | +| 2020-01-01T00:04:00Z | 160 | +| 2020-01-01T00:12:00Z | 150 | +| 2020-01-01T00:19:00Z | 220 | +| 2020-01-01T00:32:00Z | 200 | +| 2020-01-01T00:51:00Z | 290 | +| 2020-01-01T01:00:00Z | 340 | +{{% /flex-content %}} +{{% flex-content %}} +**The following returns:** + +```js +|> aggregate.rate( + every: 20m, + unit: 1m +) +``` + +| _time | _value | +|:----- | ------:| +| 2020-01-01T00:20:00Z | | +| 2020-01-01T00:40:00Z | 10.0 | +| 2020-01-01T01:00:00Z | 4.74 | +| 2020-01-01T01:20:00Z | 5.56 | +{{% /flex-content %}} +{{< /flex >}} + +Results represent the **average change rate per minute** of every **20 minute interval** +with negative values set to _null_. +Timestamps represent the right bound of the time window used to average values. diff --git a/content/enterprise_influxdb/v1.9/flux/guides/regular-expressions.md b/content/enterprise_influxdb/v1.9/flux/guides/regular-expressions.md new file mode 100644 index 000000000..91d1e1865 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/regular-expressions.md @@ -0,0 +1,93 @@ +--- +title: Use regular expressions in Flux +list_title: Regular expressions +description: This guide walks through using regular expressions in evaluation logic in Flux functions. +menu: + enterprise_influxdb_1_9: + name: Regular expressions + parent: Query with Flux +weight: 20 +list_query_example: regular_expressions +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/regular-expressions/ +v2: /influxdb/v2.0/query-data/flux/regular-expressions/ +--- + +Regular expressions (regexes) are incredibly powerful when matching patterns in large collections of data. +With Flux, regular expressions are primarily used for evaluation logic in predicate functions for things +such as filtering rows, dropping and keeping columns, state detection, etc. +This guide shows how to use regular expressions in your Flux scripts. + +If you're just getting started with Flux queries, check out the following: + +- [Get started with Flux](/enterprise_influxdb/v1.9/flux/get-started/) for a conceptual overview of Flux and parts of a Flux query. +- [Execute queries](/enterprise_influxdb/v1.9/flux/guides/execute-queries/) to discover a variety of ways to run your queries. + +## Go regular expression syntax +Flux uses Go's [regexp package](https://golang.org/pkg/regexp/) for regular expression search. +The links [below](#helpful-links) provide information about Go's regular expression syntax. + +## Regular expression operators +Flux provides two comparison operators for use with regular expressions. + +#### `=~` +When the expression on the left **MATCHES** the regular expression on the right, this evaluates to `true`. + +#### `!~` +When the expression on the left **DOES NOT MATCH** the regular expression on the right, this evaluates to `true`. + +## Regular expressions in Flux +When using regex matching in your Flux scripts, enclose your regular expressions with `/`. +The following is the basic regex comparison syntax: + +###### Basic regex comparison syntax +```js +expression =~ /regex/ +expression !~ /regex/ +``` +## Examples + +### Use a regex to filter by tag value +The following example filters records by the `cpu` tag. +It only keeps records for which the `cpu` is either `cpu0`, `cpu1`, or `cpu2`. + +```js +from(bucket: "db/rp") + |> range(start: -15m) + |> filter(fn: (r) => + r._measurement == "cpu" and + r._field == "usage_user" and + r.cpu =~ /cpu[0-2]/ + ) +``` + +### Use a regex to filter by field key +The following example excludes records that do not have `_percent` in a field key. + +```js +from(bucket: "db/rp") + |> range(start: -15m) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field =~ /_percent/ + ) +``` + +### Drop columns matching a regex +The following example drops columns whose names do not being with `_`. + +```js +from(bucket: "db/rp") + |> range(start: -15m) + |> filter(fn: (r) => r._measurement == "mem") + |> drop(fn: (column) => column !~ /_.*/) +``` + +## Helpful links + +##### Syntax documentation +[regexp Syntax GoDoc](https://godoc.org/regexp/syntax) +[RE2 Syntax Overview](https://github.com/google/re2/wiki/Syntax) + +##### Go regex testers +[Regex Tester - Golang](https://regex-golang.appspot.com/assets/html/index.html) +[Regex101](https://regex101.com/) diff --git a/content/enterprise_influxdb/v1.9/flux/guides/scalar-values.md b/content/enterprise_influxdb/v1.9/flux/guides/scalar-values.md new file mode 100644 index 000000000..51002e841 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/scalar-values.md @@ -0,0 +1,262 @@ +--- +title: Extract scalar values in Flux +list_title: Extract scalar values +description: > + Use Flux stream and table functions to extract scalar values from Flux query output. + This lets you, for example, dynamically set variables using query results. +menu: + enterprise_influxdb_1_9: + name: Extract scalar values + parent: Query with Flux +weight: 20 +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/scalar-values/ +v2: /influxdb/v2.0/query-data/flux/scalar-values/ +list_code_example: | + ```js + scalarValue = { + _record = + data + |> tableFind(fn: key => true) + |> getRecord(idx: 0) + return _record._value + } + ``` +--- + +Use Flux [stream and table functions](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/stream-table/) +to extract scalar values from Flux query output. +This lets you, for example, dynamically set variables using query results. + +**To extract scalar values from output:** + +1. [Extract a table](#extract-a-table). +2. [Extract a column from the table](#extract-a-column-from-the-table) + _**or**_ [extract a row from the table](#extract-a-row-from-the-table). + +_The samples on this page use the [sample data provided below](#sample-data)._ + +{{% warn %}} +#### Current limitations +- The InfluxDB user interface (UI) does not currently support raw scalar output. + Use [`map()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/map/) to add + scalar values to output data. +- The [Flux REPL](/enterprise_influxdb/v1.9/flux/guides/execute-queries/#influx-cli) does not currently support + Flux stream and table functions (also known as "dynamic queries"). + See [#15321](https://github.com/influxdata/influxdb/issues/15231). +{{% /warn %}} + +## Extract a table +Flux formats query results as a stream of tables. +To extract a scalar value from a stream of tables, you must first extract a single table. + +to extract a single table from the stream of tables. + +{{% note %}} +If query results include only one table, it is still formatted as a stream of tables. +You still must extract that table from the stream. +{{% /note %}} + +Use [`tableFind()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/stream-table/tablefind/) +to extract the **first** table whose [group key](/enterprise_influxdb/v1.9/flux/get-started/#group-keys) +values match the `fn` **predicate function**. +The predicate function requires a `key` record, which represents the group key of +each table. + +```js +sampleData + |> tableFind(fn: (key) => + key._field == "temp" and + key.location == "sfo" + ) +``` + +The example above returns a single table: + +| _time | location | _field | _value | +|:----- |:--------:|:------:| ------:| +| 2019-11-01T12:00:00Z | sfo | temp | 65.1 | +| 2019-11-01T13:00:00Z | sfo | temp | 66.2 | +| 2019-11-01T14:00:00Z | sfo | temp | 66.3 | +| 2019-11-01T15:00:00Z | sfo | temp | 66.8 | + +{{% note %}} +#### Extract the correct table +Flux functions do not guarantee table order and `tableFind()` returns only the +**first** table that matches the `fn` predicate. +To extract the table that includes the data you actually want, be very specific in +your predicate function or filter and transform your data to minimize the number +of tables piped-forward into `tableFind()`. +{{% /note %}} + +## Extract a column from the table +Use the [`getColumn()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/stream-table/getcolumn/) +to output an array of values from a specific column in the extracted table. + + +```js +sampleData + |> tableFind(fn: (key) => + key._field == "temp" and + key.location == "sfo" + ) + |> getColumn(column: "_value") + +// Returns [65.1, 66.2, 66.3, 66.8] +``` + +### Use extracted column values +Use a variable to store the array of values. +In the example below, `SFOTemps` represents the array of values. +Reference a specific index (integer starting from `0`) in the array to return the +value at that index. + +```js +SFOTemps = sampleData + |> tableFind(fn: (key) => + key._field == "temp" and + key.location == "sfo" + ) + |> getColumn(column: "_value") + +SFOTemps +// Returns [65.1, 66.2, 66.3, 66.8] + +SFOTemps[0] +// Returns 65.1 + +SFOTemps[2] +// Returns 66.3 +``` + +## Extract a row from the table +Use the [`getRecord()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/stream-table/getrecord/) +to output data from a single row in the extracted table. +Specify the index of the row to output using the `idx` parameter. +The function outputs a record with key-value pairs for each column. + +```js +sampleData + |> tableFind(fn: (key) => + key._field == "temp" and + key.location == "sfo" + ) + |> getRecord(idx: 0) + +// Returns { +// _time:2019-11-11T12:00:00Z, +// _field:"temp", +// location:"sfo", +// _value: 65.1 +// } +``` + +### Use an extracted row record +Use a variable to store the extracted row record. +In the example below, `tempInfo` represents the extracted row. +Use [dot notation](/enterprise_influxdb/v1.9/flux/get-started/syntax-basics/#records) to reference +keys in the record. + +```js +tempInfo = sampleData + |> tableFind(fn: (key) => + key._field == "temp" and + key.location == "sfo" + ) + |> getRecord(idx: 0) + +tempInfo +// Returns { +// _time:2019-11-11T12:00:00Z, +// _field:"temp", +// location:"sfo", +// _value: 65.1 +// } + +tempInfo._time +// Returns 2019-11-11T12:00:00Z + +tempInfo.location +// Returns sfo +``` + +## Example helper functions +Create custom helper functions to extract scalar values from query output. + +##### Extract a scalar field value +```js +// Define a helper function to extract field values +getFieldValue = (tables=<-, field) => { + extract = tables + |> tableFind(fn: (key) => key._field == field) + |> getColumn(column: "_value") + return extract[0] +} + +// Use the helper function to define a variable +lastJFKTemp = sampleData + |> filter(fn: (r) => r.location == "kjfk") + |> last() + |> getFieldValue(field: "temp") + +lastJFKTemp +// Returns 71.2 +``` + +##### Extract scalar row data +```js +// Define a helper function to extract a row as a record +getRow = (tables=<-, field, idx=0) => { + extract = tables + |> tableFind(fn: (key) => true) + |> getRecord(idx: idx) + return extract +} + +// Use the helper function to define a variable +lastReported = sampleData + |> last() + |> getRow(idx: 0) + +"The last location to report was ${lastReported.location}. +The temperature was ${string(v: lastReported._value)}°F." + +// Returns: +// The last location to report was kord. +// The temperature was 38.9°F. +``` + +--- + +## Sample data + +The following sample data set represents fictional temperature metrics collected +from three locations. +It's formatted in [annotated CSV](https://v2.docs.influxdata.com/v2.0/reference/syntax/annotated-csv/) and imported +into the Flux query using the [`csv.from()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/csv/from/). + +Place the following at the beginning of your query to use the sample data: + +{{% truncate %}} +```js +import "csv" + +sampleData = csv.from(csv: " +#datatype,string,long,dateTime:RFC3339,string,string,double +#group,false,true,false,true,true,false +#default,,,,,, +,result,table,_time,location,_field,_value +,,0,2019-11-01T12:00:00Z,sfo,temp,65.1 +,,0,2019-11-01T13:00:00Z,sfo,temp,66.2 +,,0,2019-11-01T14:00:00Z,sfo,temp,66.3 +,,0,2019-11-01T15:00:00Z,sfo,temp,66.8 +,,1,2019-11-01T12:00:00Z,kjfk,temp,69.4 +,,1,2019-11-01T13:00:00Z,kjfk,temp,69.9 +,,1,2019-11-01T14:00:00Z,kjfk,temp,71.0 +,,1,2019-11-01T15:00:00Z,kjfk,temp,71.2 +,,2,2019-11-01T12:00:00Z,kord,temp,46.4 +,,2,2019-11-01T13:00:00Z,kord,temp,46.3 +,,2,2019-11-01T14:00:00Z,kord,temp,42.7 +,,2,2019-11-01T15:00:00Z,kord,temp,38.9 +") +``` +{{% /truncate %}} diff --git a/content/enterprise_influxdb/v1.9/flux/guides/sort-limit.md b/content/enterprise_influxdb/v1.9/flux/guides/sort-limit.md new file mode 100644 index 000000000..ba9fa4a1f --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/sort-limit.md @@ -0,0 +1,70 @@ +--- +title: Sort and limit data with Flux +seotitle: Sort and limit data in InfluxDB with Flux +list_title: Sort and limit +description: > + Use the `sort()`function to order records within each table by specific columns and the + `limit()` function to limit the number of records in output tables to a fixed number, `n`. +menu: + enterprise_influxdb_1_9: + name: Sort and limit + parent: Query with Flux +weight: 3 +list_query_example: sort_limit +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/sort-limit/ +v2: /influxdb/v2.0/query-data/flux/sort-limit/ +--- + +Use the [`sort()`function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/sort) +to order records within each table by specific columns and the +[`limit()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/limit) +to limit the number of records in output tables to a fixed number, `n`. + +If you're just getting started with Flux queries, check out the following: + +- [Get started with Flux](/enterprise_influxdb/v1.9/flux/get-started/) for a conceptual overview of Flux and parts of a Flux query. +- [Execute queries](/enterprise_influxdb/v1.9/flux/guides/execute-queries/) to discover a variety of ways to run your queries. + +##### Example sorting system uptime + +The following example orders system uptime first by region, then host, then value. + +```js +from(bucket:"db/rp") + |> range(start:-12h) + |> filter(fn: (r) => + r._measurement == "system" and + r._field == "uptime" + ) + |> sort(columns:["region", "host", "_value"]) +``` + +The [`limit()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/limit) +limits the number of records in output tables to a fixed number, `n`. +The following example shows up to 10 records from the past hour. + +```js +from(bucket:"db/rp") + |> range(start:-1h) + |> limit(n:10) +``` + +You can use `sort()` and `limit()` together to show the top N records. +The example below returns the 10 top system uptime values sorted first by +region, then host, then value. + +```js +from(bucket:"db/rp") + |> range(start:-12h) + |> filter(fn: (r) => + r._measurement == "system" and + r._field == "uptime" + ) + |> sort(columns:["region", "host", "_value"]) + |> limit(n:10) +``` + +You now have created a Flux query that sorts and limits data. +Flux also provides the [`top()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/top) +and [`bottom()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/selectors/bottom) +functions to perform both of these functions at the same time. diff --git a/content/enterprise_influxdb/v1.9/flux/guides/sql.md b/content/enterprise_influxdb/v1.9/flux/guides/sql.md new file mode 100644 index 000000000..f7b510ce0 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/sql.md @@ -0,0 +1,215 @@ +--- +title: Query SQL data sources +seotitle: Query SQL data sources with InfluxDB +list_title: Query SQL data +description: > + The Flux `sql` package provides functions for working with SQL data sources. + Use `sql.from()` to query SQL databases like PostgreSQL and MySQL +menu: + enterprise_influxdb_1_9: + parent: Query with Flux + list_title: SQL data +weight: 20 +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/sql/ +v2: /influxdb/v2.0/query-data/flux/sql/ +list_code_example: | + ```js + import "sql" + + sql.from( + driverName: "postgres", + dataSourceName: "postgresql://user:password@localhost", + query: "SELECT * FROM example_table" + ) + ``` +--- + +The Flux `sql` package provides functions for working with SQL data sources. +[`sql.from()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/sql/from/) lets you query SQL data sources +like [PostgreSQL](https://www.postgresql.org/), [MySQL](https://www.mysql.com/), +and [SQLite](https://www.sqlite.org/index.html), and use the results with InfluxDB +dashboards, tasks, and other operations. + +- [Query a SQL data source](#query-a-sql-data-source) +- [Join SQL data with data in InfluxDB](#join-sql-data-with-data-in-influxdb) +- [Sample sensor data](#sample-sensor-data) + +## Query a SQL data source +To query a SQL data source: + +1. Import the `sql` package in your Flux query +2. Use the `sql.from()` function to specify the driver, data source name (DSN), + and query used to query data from your SQL data source: + +{{< code-tabs-wrapper >}} +{{% code-tabs %}} +[PostgreSQL](#) +[MySQL](#) +[SQLite](#) +{{% /code-tabs %}} + +{{% code-tab-content %}} +```js +import "sql" + +sql.from( + driverName: "postgres", + dataSourceName: "postgresql://user:password@localhost", + query: "SELECT * FROM example_table" +) +``` +{{% /code-tab-content %}} + +{{% code-tab-content %}} +```js +import "sql" + +sql.from( + driverName: "mysql", + dataSourceName: "user:password@tcp(localhost:3306)/db", + query: "SELECT * FROM example_table" +) +``` +{{% /code-tab-content %}} + +{{% code-tab-content %}} +```js +// NOTE: InfluxDB OSS and InfluxDB Cloud do not have access to +// the local filesystem and cannot query SQLite data sources. +// Use the Flux REPL to query an SQLite data source. + +import "sql" +sql.from( + driverName: "sqlite3", + dataSourceName: "file:/path/to/test.db?cache=shared&mode=ro", + query: "SELECT * FROM example_table" +) +``` +{{% /code-tab-content %}} +{{< /code-tabs-wrapper >}} + +_See the [`sql.from()` documentation](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/sql/from/) for +information about required function parameters._ + +## Join SQL data with data in InfluxDB +One of the primary benefits of querying SQL data sources from InfluxDB +is the ability to enrich query results with data stored outside of InfluxDB. + +Using the [air sensor sample data](#sample-sensor-data) below, the following query +joins air sensor metrics stored in InfluxDB with sensor information stored in PostgreSQL. +The joined data lets you query and filter results based on sensor information +that isn't stored in InfluxDB. + +```js +// Import the "sql" package +import "sql" + +// Query data from PostgreSQL +sensorInfo = sql.from( + driverName: "postgres", + dataSourceName: "postgresql://localhost?sslmode=disable", + query: "SELECT * FROM sensors" +) + +// Query data from InfluxDB +sensorMetrics = from(bucket: "telegraf/autogen") + |> range(start: -1h) + |> filter(fn: (r) => r._measurement == "airSensors") + +// Join InfluxDB query results with PostgreSQL query results +join(tables: {metric: sensorMetrics, info: sensorInfo}, on: ["sensor_id"]) +``` + +--- + +## Sample sensor data +The [sample data generator](#download-and-run-the-sample-data-generator) and +[sample sensor information](#import-the-sample-sensor-information) simulate a +group of sensors that measure temperature, humidity, and carbon monoxide +in rooms throughout a building. +Each collected data point is stored in InfluxDB with a `sensor_id` tag that identifies +the specific sensor it came from. +Sample sensor information is stored in PostgreSQL. + +**Sample data includes:** + +- Simulated data collected from each sensor and stored in the `airSensors` measurement in **InfluxDB**: + - temperature + - humidity + - co + +- Information about each sensor stored in the `sensors` table in **PostgreSQL**: + - sensor_id + - location + - model_number + - last_inspected + +### Import and generate sample sensor data + +#### Download and run the sample data generator +`air-sensor-data.rb` is a script that generates air sensor data and stores the data in InfluxDB. +To use `air-sensor-data.rb`: + +1. [Create a database](/enterprise_influxdb/v1.9/introduction/get-started/#creating-a-database) to store the data. +2. Download the sample data generator. _This tool requires [Ruby](https://www.ruby-lang.org/en/)._ + + Download Air Sensor Generator + +3. Give `air-sensor-data.rb` executable permissions: + + ``` + chmod +x air-sensor-data.rb + ``` + +4. Start the generator. Specify your database. + + ``` + ./air-sensor-data.rb -d database-name + ``` + + The generator begins to write data to InfluxDB and will continue until stopped. + Use `ctrl-c` to stop the generator. + + _**Note:** Use the `--help` flag to view other configuration options._ + + +5. Query your target database to ensure the generated data is writing successfully. + The generator doesn't catch errors from write requests, so it will continue running + even if data is not writing to InfluxDB successfully. + + ``` + from(bucket: "database-name/autogen") + |> range(start: -1m) + |> filter(fn: (r) => r._measurement == "airSensors") + ``` + +#### Import the sample sensor information +1. [Download and install PostgreSQL](https://www.postgresql.org/download/). +2. Download the sample sensor information CSV. + + Download Sample Data + +3. Use a PostgreSQL client (`psql` or a GUI) to create the `sensors` table: + + ``` + CREATE TABLE sensors ( + sensor_id character varying(50), + location character varying(50), + model_number character varying(50), + last_inspected date + ); + ``` + +4. Import the downloaded CSV sample data. + _Update the `FROM` file path to the path of the downloaded CSV sample data._ + + ``` + COPY sensors(sensor_id,location,model_number,last_inspected) + FROM '/path/to/sample-sensor-info.csv' DELIMITER ',' CSV HEADER; + ``` + +5. Query the table to ensure the data was imported correctly: + + ``` + SELECT * FROM sensors; + ``` diff --git a/content/enterprise_influxdb/v1.9/flux/guides/window-aggregate.md b/content/enterprise_influxdb/v1.9/flux/guides/window-aggregate.md new file mode 100644 index 000000000..b6cf91d83 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/guides/window-aggregate.md @@ -0,0 +1,354 @@ +--- +title: Window and aggregate data with Flux +seotitle: Window and aggregate data in InfluxDB with Flux +list_title: Window & aggregate +description: > + This guide walks through windowing and aggregating data with Flux and outlines + how it shapes your data in the process. +menu: + enterprise_influxdb_1_9: + name: Window & aggregate + parent: Query with Flux +weight: 4 +list_query_example: aggregate_window +canonical: /{{< latest "influxdb" "v2" >}}/query-data/flux/window-aggregate/ +v2: /influxdb/v2.0/query-data/flux/window-aggregate/ +--- + +A common operation performed with time series data is grouping data into windows of time, +or "windowing" data, then aggregating windowed values into a new value. +This guide walks through windowing and aggregating data with Flux and demonstrates +how data is shaped in the process. + +If you're just getting started with Flux queries, check out the following: + +- [Get started with Flux](/enterprise_influxdb/v1.9/flux/get-started/) for a conceptual overview of Flux and parts of a Flux query. +- [Execute queries](/enterprise_influxdb/v1.9/flux/guides/execute-queries/) to discover a variety of ways to run your queries. + +{{% note %}} +The following example is an in-depth walk-through of the steps required to window and aggregate data. +The [`aggregateWindow()` function](#summing-up) performs these operations for you, but understanding +how data is shaped in the process helps to successfully create your desired output. +{{% /note %}} + +## Data set +For the purposes of this guide, define a variable that represents your base data set. +The following example queries the memory usage of the host machine. + +```js +dataSet = from(bucket: "db/rp") + |> range(start: -5m) + |> filter(fn: (r) => + r._measurement == "mem" and + r._field == "used_percent" + ) + |> drop(columns: ["host"]) +``` + +{{% note %}} +This example drops the `host` column from the returned data since the memory data +is only tracked for a single host and it simplifies the output tables. +Dropping the `host` column is optional and not recommended if monitoring memory +on multiple hosts. +{{% /note %}} + +`dataSet` can now be used to represent your base data, which will look similar to the following: + +{{% truncate %}} +``` +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:00.000000000Z 71.11611366271973 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:10.000000000Z 67.39630699157715 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:20.000000000Z 64.16666507720947 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:30.000000000Z 64.19951915740967 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:40.000000000Z 64.2122745513916 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:50:50.000000000Z 64.22209739685059 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:00.000000000Z 64.6336555480957 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:10.000000000Z 64.16516304016113 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:20.000000000Z 64.18349742889404 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:30.000000000Z 64.20474052429199 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:40.000000000Z 68.65062713623047 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:50.000000000Z 67.20139980316162 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:00.000000000Z 70.9143877029419 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:10.000000000Z 64.14549350738525 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:20.000000000Z 64.15379047393799 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:30.000000000Z 64.1592264175415 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:40.000000000Z 64.18190002441406 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:50.000000000Z 64.28837776184082 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:00.000000000Z 64.29731845855713 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:10.000000000Z 64.36963081359863 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:20.000000000Z 64.37397003173828 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:30.000000000Z 64.44413661956787 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:40.000000000Z 64.42906856536865 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:50.000000000Z 64.44573402404785 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:00.000000000Z 64.48912620544434 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:10.000000000Z 64.49522972106934 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:20.000000000Z 64.48652744293213 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:30.000000000Z 64.49949741363525 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:40.000000000Z 64.4949197769165 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:50.000000000Z 64.49787616729736 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49816226959229 +``` +{{% /truncate %}} + +## Windowing data +Use the [`window()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/window) +to group your data based on time bounds. +The most common parameter passed with the `window()` is `every` which +defines the duration of time between windows. +Other parameters are available, but for this example, window the base data +set into one minute windows. + +```js +dataSet + |> window(every: 1m) +``` + +{{% note %}} +The `every` parameter supports all [valid duration units](/{{< latest "influxdb" "v2" >}}/reference/flux/language/types/#duration-types), +including **calendar months (`1mo`)** and **years (`1y`)**. +{{% /note %}} + +Each window of time is output in its own table containing all records that fall within the window. + +{{% truncate %}} +###### window() output tables +``` +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:00.000000000Z 71.11611366271973 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:10.000000000Z 67.39630699157715 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:20.000000000Z 64.16666507720947 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:30.000000000Z 64.19951915740967 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:40.000000000Z 64.2122745513916 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:50:50.000000000Z 64.22209739685059 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:00.000000000Z 64.6336555480957 +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:10.000000000Z 64.16516304016113 +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:20.000000000Z 64.18349742889404 +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:30.000000000Z 64.20474052429199 +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:40.000000000Z 68.65062713623047 +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:51:50.000000000Z 67.20139980316162 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:00.000000000Z 70.9143877029419 +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:10.000000000Z 64.14549350738525 +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:20.000000000Z 64.15379047393799 +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:30.000000000Z 64.1592264175415 +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:40.000000000Z 64.18190002441406 +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:52:50.000000000Z 64.28837776184082 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:00.000000000Z 64.29731845855713 +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:10.000000000Z 64.36963081359863 +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:20.000000000Z 64.37397003173828 +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:30.000000000Z 64.44413661956787 +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:40.000000000Z 64.42906856536865 +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:53:50.000000000Z 64.44573402404785 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:00.000000000Z 64.48912620544434 +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:10.000000000Z 64.49522972106934 +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:20.000000000Z 64.48652744293213 +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:30.000000000Z 64.49949741363525 +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:40.000000000Z 64.4949197769165 +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:50.000000000Z 64.49787616729736 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:55:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49816226959229 +``` +{{% /truncate %}} + +When visualized in the InfluxDB UI, each window table is displayed in a different color. + +![Windowed data](/img/flux/simple-windowed-data.png) + +## Aggregate data +[Aggregate functions](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates) take the values +of all rows in a table and use them to perform an aggregate operation. +The result is output as a new value in a single-row table. + +Since windowed data is split into separate tables, aggregate operations run against +each table separately and output new tables containing only the aggregated value. + +For this example, use the [`mean()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/mean) +to output the average of each window: + +```js +dataSet + |> window(every: 1m) + |> mean() +``` + +{{% truncate %}} +###### mean() output tables +``` +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 65.88549613952637 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 65.50651391347249 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 65.30719598134358 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 64.39330975214641 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 64.49386278788249 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ---------------------------- +2018-11-03T17:55:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 64.49816226959229 +``` +{{% /truncate %}} + +Because each data point is contained in its own table, when visualized, +they appear as single, unconnected points. + +![Aggregated windowed data](/img/flux/simple-windowed-aggregate-data.png) + +### Recreate the time column +**Notice the `_time` column is not in the [aggregated output tables](#mean-output-tables).** +Because records in each table are aggregated together, their timestamps no longer +apply and the column is removed from the group key and table. + +Also notice the `_start` and `_stop` columns still exist. +These represent the lower and upper bounds of the time window. + +Many Flux functions rely on the `_time` column. +To further process your data after an aggregate function, you need to re-add `_time`. +Use the [`duplicate()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/duplicate) to +duplicate either the `_start` or `_stop` column as a new `_time` column. + +```js +dataSet + |> window(every: 1m) + |> mean() + |> duplicate(column: "_stop", as: "_time") +``` + +{{% truncate %}} +###### duplicate() output tables +``` +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:50:00.000000000Z 2018-11-03T17:51:00.000000000Z used_percent mem 2018-11-03T17:51:00.000000000Z 65.88549613952637 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:51:00.000000000Z 2018-11-03T17:52:00.000000000Z used_percent mem 2018-11-03T17:52:00.000000000Z 65.50651391347249 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:52:00.000000000Z 2018-11-03T17:53:00.000000000Z used_percent mem 2018-11-03T17:53:00.000000000Z 65.30719598134358 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:53:00.000000000Z 2018-11-03T17:54:00.000000000Z used_percent mem 2018-11-03T17:54:00.000000000Z 64.39330975214641 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:54:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49386278788249 + + +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:55:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49816226959229 +``` +{{% /truncate %}} + +## "Unwindow" aggregate tables +Keeping aggregate values in separate tables generally isn't the format in which you want your data. +Use the `window()` function to "unwindow" your data into a single infinite (`inf`) window. + +```js +dataSet + |> window(every: 1m) + |> mean() + |> duplicate(column: "_stop", as: "_time") + |> window(every: inf) +``` + +{{% note %}} +Windowing requires a `_time` column which is why it's necessary to +[recreate the `_time` column](#recreate-the-time-column) after an aggregation. +{{% /note %}} + +###### Unwindowed output table +``` +Table: keys: [_start, _stop, _field, _measurement] + _start:time _stop:time _field:string _measurement:string _time:time _value:float +------------------------------ ------------------------------ ---------------------- ---------------------- ------------------------------ ---------------------------- +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:51:00.000000000Z 65.88549613952637 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:52:00.000000000Z 65.50651391347249 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:53:00.000000000Z 65.30719598134358 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:54:00.000000000Z 64.39330975214641 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49386278788249 +2018-11-03T17:50:00.000000000Z 2018-11-03T17:55:00.000000000Z used_percent mem 2018-11-03T17:55:00.000000000Z 64.49816226959229 +``` + +With the aggregate values in a single table, data points in the visualization are connected. + +![Unwindowed aggregate data](/img/flux/simple-unwindowed-data.png) + +## Summing up +You have now created a Flux query that windows and aggregates data. +The data transformation process outlined in this guide should be used for all aggregation operations. + +Flux also provides the [`aggregateWindow()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/aggregatewindow) +which performs all these separate functions for you. + +The following Flux query will return the same results: + +###### aggregateWindow function +```js +dataSet + |> aggregateWindow(every: 1m, fn: mean) +``` diff --git a/content/enterprise_influxdb/v1.9/flux/installation.md b/content/enterprise_influxdb/v1.9/flux/installation.md new file mode 100644 index 000000000..bbfb12102 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/installation.md @@ -0,0 +1,33 @@ +--- +title: Enable Flux +description: Instructions for enabling Flux in your InfluxDB configuration. +menu: + enterprise_influxdb_1_9: + name: Enable Flux + parent: Flux + weight: 1 +--- + +Flux is packaged with **InfluxDB v1.8+** and does not require any additional installation, +however it is **disabled by default and needs to be enabled**. + +## Enable Flux +Enable Flux by setting the `flux-enabled` option to `true` under the `[http]` section of your `influxdb.conf`: + +###### influxdb.conf +```toml +# ... + +[http] + + # ... + + flux-enabled = true + + # ... +``` + +> The default location of your `influxdb.conf` depends on your operating system. +> More information is available in the [Configuring InfluxDB](/enterprise_influxdb/v1.9/administration/config/#using-the-configuration-file) guide. + +When InfluxDB starts, the Flux daemon starts as well and data can be queried using Flux. diff --git a/content/enterprise_influxdb/v1.9/flux/optimize-queries.md b/content/enterprise_influxdb/v1.9/flux/optimize-queries.md new file mode 100644 index 000000000..f4c25504a --- /dev/null +++ b/content/enterprise_influxdb/v1.9/flux/optimize-queries.md @@ -0,0 +1,180 @@ +--- +title: Optimize Flux queries +description: > + Optimize your Flux queries to reduce their memory and compute (CPU) requirements. +weight: 4 +menu: + enterprise_influxdb_1_9: + parent: Flux +canonical: /influxdb/cloud/query-data/optimize-queries/ +aliases: + - /enterprise_influxdb/v1.9/flux/guides/optimize-queries +--- + +Optimize your Flux queries to reduce their memory and compute (CPU) requirements. + +- [Start queries with pushdowns](#start-queries-with-pushdowns) +- [Avoid processing filters inline](#avoid-processing-filters-inline) +- [Avoid short window durations](#avoid-short-window-durations) +- [Use "heavy" functions sparingly](#use-heavy-functions-sparingly) +- [Use set() instead of map() when possible](#use-set-instead-of-map-when-possible) +- [Balance time range and data precision](#balance-time-range-and-data-precision) +- [Measure query performance with Flux profilers](#measure-query-performance-with-flux-profilers) + +## Start queries with pushdowns +**Pushdowns** are functions or function combinations that push data operations to the underlying data source rather than operating on data in memory. Start queries with pushdowns to improve query performance. Once a non-pushdown function runs, Flux pulls data into memory and runs all subsequent operations there. + +#### Pushdown functions and function combinations +The following pushdowns are supported in InfluxDB Enterprise 1.9+. + +| Functions | Supported | +| :----------------------------- | :------------------: | +| **count()** | {{< icon "check" >}} | +| **drop()** | {{< icon "check" >}} | +| **duplicate()** | {{< icon "check" >}} | +| **filter()** {{% req " \*" %}} | {{< icon "check" >}} | +| **fill()** | {{< icon "check" >}} | +| **first()** | {{< icon "check" >}} | +| **group()** | {{< icon "check" >}} | +| **keep()** | {{< icon "check" >}} | +| **last()** | {{< icon "check" >}} | +| **max()** | {{< icon "check" >}} | +| **mean()** | {{< icon "check" >}} | +| **min()** | {{< icon "check" >}} | +| **range()** | {{< icon "check" >}} | +| **rename()** | {{< icon "check" >}} | +| **sum()** | {{< icon "check" >}} | +| **window()** | {{< icon "check" >}} | +| _Function combinations_ | | +| **window()** \|> **count()** | {{< icon "check" >}} | +| **window()** \|> **first()** | {{< icon "check" >}} | +| **window()** \|> **last()** | {{< icon "check" >}} | +| **window()** \|> **max()** | {{< icon "check" >}} | +| **window()** \|> **min()** | {{< icon "check" >}} | +| **window()** \|> **sum()** | {{< icon "check" >}} | + +{{% caption %}} +{{< req "\*" >}} **filter()** only pushes down when all parameter values are static. +See [Avoid processing filters inline](#avoid-processing-filters-inline). +{{% /caption %}} + +Use pushdown functions and function combinations at the beginning of your query. +Once a non-pushdown function runs, Flux pulls data into memory and runs all +subsequent operations there. + +##### Pushdown functions in use +```js +from(bucket: "db/rp") + |> range(start: -1h) // + |> filter(fn: (r) => r.sensor == "abc123") // + |> group(columns: ["_field", "host"]) // Pushed to the data source + |> aggregateWindow(every: 5m, fn: max) // + |> filter(fn: (r) => r._value >= 90.0) // + + |> top(n: 10) // Run in memory +``` + +### Avoid processing filters inline +Avoid using mathematic operations or string manipulation inline to define data filters. +Processing filter values inline prevents `filter()` from pushing its operation down +to the underlying data source, so data returned by the +previous function loads into memory. +This often results in a significant performance hit. + +For example, the following query uses [Chronograf dashboard template variables](/{{< latest "chronograf" >}}/guides/dashboard-template-variables/) +and string concatenation to define a region to filter by. +Because `filter()` uses string concatenation inline, it can't push its operation +to the underlying data source and loads all data returned from `range()` into memory. + +```js +from(bucket: "db/rp") + |> range(start: -1h) + |> filter(fn: (r) => r.region == v.provider + v.region) +``` + +To dynamically set filters and maintain the pushdown ability of the `filter()` function, +use variables to define filter values outside of `filter()`: + +```js +region = v.provider + v.region + +from(bucket: "db/rp") + |> range(start: -1h) + |> filter(fn: (r) => r.region == region) +``` + +## Avoid short window durations +Windowing (grouping data based on time intervals) is commonly used to aggregate and downsample data. +Increase performance by avoiding short window durations. +More windows require more compute power to evaluate which window each row should be assigned to. +Reasonable window durations depend on the total time range queried. + +## Use "heavy" functions sparingly +The following functions use more memory or CPU than others. +Consider their necessity in your data processing before using them: + +- [map()](/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/map/) +- [reduce()](/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/aggregates/reduce/) +- [join()](/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/join/) +- [union()](/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/union/) +- [pivot()](/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/pivot/) + +## Use set() instead of map() when possible +[`set()`](/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/set/), +[`experimental.set()`](/influxdb/v2.0/reference/flux/stdlib/experimental/set/), +and [`map`](/influxdb/v2.0/reference/flux/stdlib/built-in/transformations/map/) +can each set columns value in data, however **set** functions have performance +advantages over `map()`. + +Use the following guidelines to determine which to use: + +- If setting a column value to a predefined, static value, use `set()` or `experimental.set()`. +- If dynamically setting a column value using **existing row data**, use `map()`. + +#### Set a column value to a static value +The following queries are functionally the same, but using `set()` is more performant than using `map()`. + +```js +data + |> map(fn: (r) => ({ r with foo: "bar" })) + +// Recommended +data + |> set(key: "foo", value: "bar") +``` + +#### Dynamically set a column value using existing row data +```js +data + |> map(fn: (r) => ({ r with foo: r.bar })) +``` + +## Balance time range and data precision +To ensure queries are performant, balance the time range and the precision of your data. +For example, if you query data stored every second and request six months worth of data, +results would include ≈15.5 million points per series. +Depending on the number of series returned after `filter()`([cardinality](/enterprise_influxdb/v1.9/concepts/glossary/#series-cardinality)), +this can quickly become many billions of points. +Flux must store these points in memory to generate a response. +Use [pushdowns](#pushdown-functions-and-function-combinations) to optimize how +many points are stored in memory. + +## Measure query performance with Flux profilers +Use the [Flux Profiler package](/influxdb/v2.0/reference/flux/stdlib/profiler/) +to measure query performance and append performance metrics to your query output. +The following Flux profilers are available: + +- **query**: provides statistics about the execution of an entire Flux script. +- **operator**: provides statistics about each operation in a query. + +Import the `profiler` package and enable profilers with the `profile.enabledProfilers` option. + +```js +import "profiler" + +option profiler.enabledProfilers = ["query", "operator"] + +// Query to profile +``` + +For more information about Flux profilers, see the [Flux Profiler package](/influxdb/v2.0/reference/flux/stdlib/profiler/). diff --git a/content/enterprise_influxdb/v1.9/guides/_index.md b/content/enterprise_influxdb/v1.9/guides/_index.md new file mode 100644 index 000000000..a9c56fc89 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/guides/_index.md @@ -0,0 +1,12 @@ +--- +title: InfluxDB Enterprise guides +description: Step-by-step guides for using InfluxDB Enterprise. +aliases: + - /enterprise/v1.8/guides/ +menu: + enterprise_influxdb_1_9: + name: Guides + weight: 60 +--- + +{{< children hlevel="h2" >}} diff --git a/content/enterprise_influxdb/v1.9/guides/calculate_percentages.md b/content/enterprise_influxdb/v1.9/guides/calculate_percentages.md new file mode 100644 index 000000000..2acdfa402 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/guides/calculate_percentages.md @@ -0,0 +1,267 @@ +--- +title: Calculate percentages in a query +description: > + Calculate percentages using basic math operators available in InfluxQL or Flux. + This guide walks through use-cases and examples of calculating percentages from two values in a single query. +menu: + enterprise_influxdb_1_9: + weight: 50 + parent: Guides + name: Calculate percentages +aliases: + - /enterprise_influxdb/v1.9/guides/calculating_percentages/ +v2: /influxdb/v2.0/query-data/flux/calculate-percentages/ +--- + +Use Flux or InfluxQL to calculate percentages in a query. + +{{< tabs-wrapper >}} +{{% tabs %}} +[Flux](#) +[InfluxQL](#) +{{% /tabs %}} + +{{% tab-content %}} + +[Flux](/flux/latest/) lets you perform simple math equations, for example, calculating a percentage. + +## Calculate a percentage + +Learn how to calculate a percentage using the following examples: + +- [Basic calculations within a query](#basic-calculations-within-a-query) +- [Calculate a percentage from two fields](#calculate-a-percentage-from-two-fields) +- [Calculate a percentage using aggregate functions](#calculate-a-percentage-using-aggregate-functions) +- [Calculate the percentage of total weight per apple variety](#calculate-the-percentage-of-total-weight-per-apple-variety) +- [Calculate the aggregate percentage per variety](#calculate-the-percentage-of-total-weight-per-apple-variety) + +## Basic calculations within a query + +When performing any math operation in a Flux query, you must complete the following steps: + +1. Specify the [bucket](/{{< latest "influxdb" "v2" >}}/query-data/get-started/#buckets) to query from and the time range to query. +2. Filter your data by measurements, fields, and other applicable criteria. +3. Align values in one row (required to perform math in Flux) by using one of the following functions: + - To query **from multiple** data sources, use the [`join()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/join/). + - To query **from the same** data source, use the [`pivot()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/pivot/). + +For examples using the `join()` function to calculate percentages and more examples of calculating percentages, see [Calculate percentages with Flux](/{{< latest "influxdb" "v2" >}}/query-data/flux/calculate-percentages/). + +#### Data variable + +To shorten examples, we'll store a basic Flux query in a `data` variable for reuse. + +Here's how that looks in Flux: + +```js +// Query data from the past 15 minutes pivot fields into columns so each row +// contains values for each field +data = from(bucket:"your_db/your_retention_policy") + |> range(start: -15m) + |> filter(fn: (r) => r._measurement == "measurement_name" and r._field =~ /field[1-2]/) + |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value") +``` + +Each row now contains the values necessary to perform a math operation. For example, to add two field keys, start with the `data` variable created above, and then use `map()` to re-map values in each row. + +```js +data + |> map(fn: (r) => ({ r with _value: r.field1 + r.field2})) +``` + +> **Note:** Flux supports basic math operators such as `+`,`-`,`/`, `*`, and `()`. For example, to subtract `field2` from `field1`, change `+` to `-`. + +## Calculate a percentage from two fields + +Use the `data` variable created above, and then use the [`map()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/map/) to divide one field by another, multiply by 100, and add a new `percent` field to store the percentage values in. + +```js +data + |> map(fn: (r) => ({ + _time: r._time, + _measurement: r._measurement, + _field: "percent", + _value: field1 / field2 * 100.0 + })) +``` + +>**Note:** In this example, `field1` and `field2` are float values, hence multiplied by 100.0. For integer values, multiply by 100 or use the `float()` function to cast integers to floats. + +## Calculate a percentage using aggregate functions + +Use [`aggregateWindow()`](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/aggregates/aggregatewindow) to window data by time and perform an aggregate function on each window. + +```js +from(bucket:"/") + |> range(start: -15m) + |> filter(fn: (r) => r._measurement == "measurement_name" and r._field =~ /fieldkey[1-2]/) + |> aggregateWindow(every: 1m, fn:sum) + |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value") + |> map(fn: (r) => ({ r with _value: r.field1 / r.field2 * 100.0 })) +``` + +## Calculate the percentage of total weight per apple variety + +Use simulated apple stand data to track the weight of apples (by type) throughout a day. + +1. [Download the sample data](https://gist.githubusercontent.com/sanderson/8f8aec94a60b2c31a61f44a37737bfea/raw/c29b239547fa2b8ee1690f7d456d31f5bd461386/apple_stand.txt) +2. Import the sample data: + +```bash +influx -import -path=path/to/apple_stand.txt -precision=ns -database=apple_stand +``` + +Use the following query to calculate the percentage of the total weight each variety +accounts for at each given point in time. + +```js +from(bucket:"apple_stand/autogen") + |> range(start: 2018-06-18T12:00:00Z, stop: 2018-06-19T04:35:00Z) + |> filter(fn: (r) => r._measurement == "variety") + |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value") + |> map(fn: (r) => ({ r with + granny_smith: r.granny_smith / r.total_weight * 100.0 , + golden_delicious: r.golden_delicious / r.total_weight * 100.0 , + fuji: r.fuji / r.total_weight * 100.0 , + gala: r.gala / r.total_weight * 100.0 , + braeburn: r.braeburn / r.total_weight * 100.0 ,})) +``` + +## Calculate the average percentage of total weight per variety each hour + +With the apple stand data from the prior example, use the following query to calculate the average percentage of the total weight each variety accounts for per hour. + +```js +from(bucket:"apple_stand/autogen") + |> range(start: 2018-06-18T00:00:00.00Z, stop: 2018-06-19T16:35:00.00Z) + |> filter(fn: (r) => r._measurement == "variety") + |> aggregateWindow(every:1h, fn: mean) + |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value") + |> map(fn: (r) => ({ r with + granny_smith: r.granny_smith / r.total_weight * 100.0, + golden_delicious: r.golden_delicious / r.total_weight * 100.0, + fuji: r.fuji / r.total_weight * 100.0, + gala: r.gala / r.total_weight * 100.0, + braeburn: r.braeburn / r.total_weight * 100.0 + })) +``` + +{{% /tab-content %}} + +{{% tab-content %}} + +[InfluxQL](/enterprise_influxdb/v1.9/query_language/) lets you perform simple math equations +which makes calculating percentages using two fields in a measurement pretty simple. +However there are some caveats of which you need to be aware. + +## Basic calculations within a query + +`SELECT` statements support the use of basic math operators such as `+`,`-`,`/`, `*`, `()`, etc. + +```sql +-- Add two field keys +SELECT field_key1 + field_key2 AS "field_key_sum" FROM "measurement_name" WHERE time < now() - 15m + +-- Subtract one field from another +SELECT field_key1 - field_key2 AS "field_key_difference" FROM "measurement_name" WHERE time < now() - 15m + +-- Grouping and chaining mathematical calculations +SELECT (field_key1 + field_key2) - (field_key3 + field_key4) AS "some_calculation" FROM "measurement_name" WHERE time < now() - 15m +``` + +## Calculating a percentage in a query + +Using basic math functions, you can calculate a percentage by dividing one field value +by another and multiplying the result by 100: + +```sql +SELECT (field_key1 / field_key2) * 100 AS "calculated_percentage" FROM "measurement_name" WHERE time < now() - 15m +``` + +## Calculating a percentage using aggregate functions + +If using aggregate functions in your percentage calculation, all data must be referenced +using aggregate functions. +_**You can't mix aggregate and non-aggregate data.**_ + +All Aggregate functions need a `GROUP BY time()` clause defining the time intervals +in which data points are grouped and aggregated. + +```sql +SELECT (sum(field_key1) / sum(field_key2)) * 100 AS "calculated_percentage" FROM "measurement_name" WHERE time < now() - 15m GROUP BY time(1m) +``` + +## Examples + +#### Sample data + +The following example uses simulated Apple Stand data that tracks the weight of +baskets containing different varieties of apples throughout a day of business. + +1. [Download the sample data](https://gist.githubusercontent.com/sanderson/8f8aec94a60b2c31a61f44a37737bfea/raw/c29b239547fa2b8ee1690f7d456d31f5bd461386/apple_stand.txt) +2. Import the sample data: + +```bash +influx -import -path=path/to/apple_stand.txt -precision=ns -database=apple_stand +``` + +### Calculating percentage of total weight per apple variety + +The following query calculates the percentage of the total weight each variety +accounts for at each given point in time. + +```sql +SELECT + ("braeburn"/total_weight)*100, + ("granny_smith"/total_weight)*100, + ("golden_delicious"/total_weight)*100, + ("fuji"/total_weight)*100, + ("gala"/total_weight)*100 +FROM "apple_stand"."autogen"."variety" +``` +
        +\*
        + +If visualized as a [stacked graph](/chronograf/v1.8/guides/visualization-types/#stacked-graph) +in Chronograf, it would look like: + +![Percentage of total per apple variety](/img/influxdb/1-5-calc-percentage-apple-variety.png) + +### Calculating aggregate percentage per variety + +The following query calculates the average percentage of the total weight each variety +accounts for per hour. + +```sql +SELECT + (mean("braeburn")/mean(total_weight))*100, + (mean("granny_smith")/mean(total_weight))*100, + (mean("golden_delicious")/mean(total_weight))*100, + (mean("fuji")/mean(total_weight))*100, + (mean("gala")/mean(total_weight))*100 +FROM "apple_stand"."autogen"."variety" +WHERE time >= '2018-06-18T12:00:00Z' AND time <= '2018-06-19T04:35:00Z' +GROUP BY time(1h) +``` +
        + +_**Note the following about this query:**_ + +- It uses aggregate functions (`mean()`) for pulling all data. +- It includes a `GROUP BY time()` clause which aggregates data into 1 hour blocks. +- It includes an explicitly limited time window. Without it, aggregate functions + are very resource-intensive. + +If visualized as a [stacked graph](/chronograf/v1.8/guides/visualization-types/#stacked-graph) +in Chronograf, it would look like: + +![Hourly average percentage of total per apple variety](/img/influxdb/1-5-calc-percentage-hourly-apple-variety.png) + +{{% /tab-content %}} +{{< /tabs-wrapper >}} diff --git a/content/enterprise_influxdb/v1.9/guides/downsample_and_retain.md b/content/enterprise_influxdb/v1.9/guides/downsample_and_retain.md new file mode 100644 index 000000000..a945c452e --- /dev/null +++ b/content/enterprise_influxdb/v1.9/guides/downsample_and_retain.md @@ -0,0 +1,224 @@ +--- +title: Downsample and retain data +description: Downsample data to keep high precision while preserving storage. +menu: + enterprise_influxdb_1_9: + weight: 30 + parent: Guides +aliases: + - /enterprise_influxdb/v1.9/guides/downsampling_and_retention/ +v2: /influxdb/v2.0/process-data/common-tasks/downsample-data/ +--- + +InfluxDB can handle hundreds of thousands of data points per second. Working with that much data over a long period of time can create storage concerns. +A natural solution is to downsample the data; keep the high precision raw data for only a limited time, and store the lower precision, summarized data longer. +This guide describes how to automate the process of downsampling data and expiring old data using InfluxQL. To downsample and retain data using Flux and InfluxDB 2.0, +see [Process Data with InfluxDB tasks](/influxdb/v2.0/process-data/). + +### Definitions + +- **Continuous query** (CQ) is an InfluxQL query that runs automatically and periodically within a database. +CQs require a function in the `SELECT` clause and must include a `GROUP BY time()` clause. + +- **Retention policy** (RP) is the part of InfluxDB data structure that describes for how long InfluxDB keeps data. +InfluxDB compares your local server's timestamp to the timestamps on your data and deletes data older than the RP's `DURATION`. +A single database can have several RPs and RPs are unique per database. + +This guide doesn't go into detail about the syntax for creating and managing CQs and RPs or tasks. +If you're new to these concepts, we recommend reviewing the following: + +- [CQ documentation](/enterprise_influxdb/v1.9/query_language/continuous_queries/) and +- [RP documentation](/enterprise_influxdb/v1.9/query_language/manage-database/#retention-policy-management). + +### Sample data + +This section uses fictional real-time data to track the number of food orders +to a restaurant via phone and via website at ten second intervals. +We store this data in a [database](/enterprise_influxdb/v1.9/concepts/glossary/#database) or [bucket]() called `food_data`, in +the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) `orders`, and +in the [fields](/enterprise_influxdb/v1.9/concepts/glossary/#field) `phone` and `website`. + +Sample: + +```bash +name: orders +------------ +time phone website +2016-05-10T23:18:00Z 10 30 +2016-05-10T23:18:10Z 12 39 +2016-05-10T23:18:20Z 11 56 +``` + +### Goal + +Assume that, in the long run, we're only interested in the average number of orders by phone +and by website at 30 minute intervals. +In the next steps, we use RPs and CQs to: + +* Automatically aggregate the ten-second resolution data to 30-minute resolution data +* Automatically delete the raw, ten-second resolution data that are older than two hours +* Automatically delete the 30-minute resolution data that are older than 52 weeks + +### Database preparation + +We perform the following steps before writing the data to the database +`food_data`. +We do this **before** inserting any data because CQs only run against recent +data; that is, data with timestamps that are no older than `now()` minus +the `FOR` clause of the CQ, or `now()` minus the `GROUP BY time()` interval if +the CQ has no `FOR` clause. + +#### 1. Create the database + +```sql +> CREATE DATABASE "food_data" +``` + +#### 2. Create a two-hour `DEFAULT` retention policy + +InfluxDB writes to the `DEFAULT` retention policy if we do not supply an explicit RP when +writing a point to the database. +We make the `DEFAULT` RP keep data for two hours, because we want InfluxDB to +automatically write the incoming ten-second resolution data to that RP. + +Use the +[`CREATE RETENTION POLICY`](/enterprise_influxdb/v1.9/query_language/manage-database/#create-retention-policies-with-create-retention-policy) +statement to create a `DEFAULT` RP: + +```sql +> CREATE RETENTION POLICY "two_hours" ON "food_data" DURATION 2h REPLICATION 1 DEFAULT +``` + +That query creates an RP called `two_hours` that exists in the database +`food_data`. +`two_hours` keeps data for a `DURATION` of two hours (`2h`) and it's the `DEFAULT` +RP for the database `food_data`. + +{{% warn %}} +The replication factor (`REPLICATION 1`) is a required parameter but must always +be set to 1 for single node instances. +{{% /warn %}} + +> **Note:** When we created the `food_data` database in step 1, InfluxDB +automatically generated an RP named `autogen` and set it as the `DEFAULT` +RP for the database. +The `autogen` RP has an infinite retention period. +With the query above, the RP `two_hours` replaces `autogen` as the `DEFAULT` RP +for the `food_data` database. + +#### 3. Create a 52-week retention policy + +Next we want to create another retention policy that keeps data for 52 weeks and is not the +`DEFAULT` retention policy (RP) for the database. +Ultimately, the 30-minute rollup data will be stored in this RP. + +Use the +[`CREATE RETENTION POLICY`](/enterprise_influxdb/v1.9/query_language/manage-database/#create-retention-policies-with-create-retention-policy) +statement to create a non-`DEFAULT` retention policy: + +```sql +> CREATE RETENTION POLICY "a_year" ON "food_data" DURATION 52w REPLICATION 1 +``` + +That query creates a retention policy (RP) called `a_year` that exists in the database +`food_data`. +The `a_year` setting keeps data for a `DURATION` of 52 weeks (`52w`). +Leaving out the `DEFAULT` argument ensures that `a_year` is not the `DEFAULT` +RP for the database `food_data`. +That is, write and read operations against `food_data` that do not specify an +RP will still go to the `two_hours` RP (the `DEFAULT` RP). + +#### 4. Create the continuous query + +Now that we've set up our RPs, we want to create a continuous query (CQ) that will automatically +and periodically downsample the ten-second resolution data to the 30-minute +resolution, and then store those results in a different measurement with a different +retention policy. + +Use the +[`CREATE CONTINUOUS QUERY`](/enterprise_influxdb/v1.9/query_language/continuous_queries/) +statement to generate a CQ: + +```sql +> CREATE CONTINUOUS QUERY "cq_30m" ON "food_data" BEGIN + SELECT mean("website") AS "mean_website",mean("phone") AS "mean_phone" + INTO "a_year"."downsampled_orders" + FROM "orders" + GROUP BY time(30m) +END +``` + +That query creates a CQ called `cq_30m` in the database `food_data`. +`cq_30m` tells InfluxDB to calculate the 30-minute average of the two fields +`website` and `phone` in the measurement `orders` and in the `DEFAULT` RP +`two_hours`. +It also tells InfluxDB to write those results to the measurement +`downsampled_orders` in the retention policy `a_year` with the field keys +`mean_website` and `mean_phone`. +InfluxDB will run this query every 30 minutes for the previous 30 minutes. + +> **Note:** Notice that we fully qualify (that is, we use the syntax +`"".""`) the measurement in the `INTO` +clause. +InfluxDB requires that syntax to write data to an RP other than the `DEFAULT` +RP. + +### Results + +With the new CQ and two new RPs, `food_data` is ready to start receiving data. +After writing data to our database and letting things run for a bit, we see +two measurements: `orders` and `downsampled_orders`. + +```sql +> SELECT * FROM "orders" LIMIT 5 +name: orders +--------- +time phone website +2016-05-13T23:00:00Z 10 30 +2016-05-13T23:00:10Z 12 39 +2016-05-13T23:00:20Z 11 56 +2016-05-13T23:00:30Z 8 34 +2016-05-13T23:00:40Z 17 32 + +> SELECT * FROM "a_year"."downsampled_orders" LIMIT 5 +name: downsampled_orders +--------------------- +time mean_phone mean_website +2016-05-13T15:00:00Z 12 23 +2016-05-13T15:30:00Z 13 32 +2016-05-13T16:00:00Z 19 21 +2016-05-13T16:30:00Z 3 26 +2016-05-13T17:00:00Z 4 23 +``` + +The data in `orders` are the raw, ten-second resolution data that reside in the +two-hour RP. +The data in `downsampled_orders` are the aggregated, 30-minute resolution data +that are subject to the 52-week RP. + +Notice that the first timestamps in `downsampled_orders` are older than the first +timestamps in `orders`. +This is because InfluxDB has already deleted data from `orders` with timestamps +that are older than our local server's timestamp minus two hours (assume we + executed the `SELECT` queries at `2016-05-14T00:59:59Z`). +InfluxDB will only start dropping data from `downsampled_orders` after 52 weeks. + +> **Notes:** +> +* Notice that we fully qualify (that is, we use the syntax +`"".""`) `downsampled_orders` in +the second `SELECT` statement. We must specify the RP in that query to `SELECT` +data that reside in an RP other than the `DEFAULT` RP. +> +* By default, InfluxDB checks to enforce an RP every 30 minutes. +Between checks, `orders` may have data that are older than two hours. +The rate at which InfluxDB checks to enforce an RP is a configurable setting, +see +[Database Configuration](/enterprise_influxdb/v1.9/administration/config#check-interval-30m0s). + +Using a combination of RPs and CQs, we've successfully set up our database to +automatically keep the high precision raw data for a limited time, create lower +precision data, and store that lower precision data for a longer period of time. +Now that you have a general understanding of how these features can work +together, check out the detailed documentation on [CQs](/enterprise_influxdb/v1.9/query_language/continuous_queries/) and [RPs](/enterprise_influxdb/v1.9/query_language/manage-database/#retention-policy-management) +to see all that they can do for you. diff --git a/content/enterprise_influxdb/v1.9/guides/fine-grained-authorization.md b/content/enterprise_influxdb/v1.9/guides/fine-grained-authorization.md new file mode 100644 index 000000000..35c4032e3 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/guides/fine-grained-authorization.md @@ -0,0 +1,625 @@ +--- +title: Use fine-grained authorization in InfluxDB Enterprise +description: > + Fine-grained authorization (FGA) in InfluxDB Enterprise controls user access at the database, measurement, and series levels. +alias: + -/docs/v1.5/administration/fga +menu: + enterprise_influxdb_1_9: + name: Use fine-grained authorization + weight: 10 + parent: Guides +--- + +Use fine-grained authorization (FGA) in InfluxDB Enterprise to control user access at the database, measurement, and series levels. + +> **Note:** InfluxDB OSS controls access at the database level only. + +You must have [admin permissions](/enterprise_influxdb/v1.9/administration/authentication_and_authorization/#admin-user-management) to set up FGA. + +## Set up fine-grained authorization + +1. [Enable authentication](/enterprise_influxdb/v1.9/administration/authentication_and_authorization/#set-up-authentication) in your InfluxDB configuration file. + +2. Create users through the InfluxDB query API. + + ```sql + CREATE USER username WITH PASSWORD 'password' + ``` + + For more information, see [User management commands](/enterprise_influxdb/v1.9/administration/authentication_and_authorization/#user-management-commands). + +3. Ensure that you can access the **meta node** API (port 8091 by default). + + > In a typical cluster configuration, the HTTP ports for data nodes + > (8086 by default) are exposed to clients but the meta node HTTP ports are not. + > You may need to work with your network administrator to gain access to the meta node HTTP ports. + +4. _(Optional)_ [Create roles](#manage-roles). + Roles let you grant permissions to groups of users assigned to each role. + + > For an overview of how users and roles work in InfluxDB Enterprise, see [InfluxDB Enterprise users](/enterprise_influxdb/v1.9/features/users/). + +5. [Set up restrictions](#manage-restrictions). + Restrictions apply to all non-admin users. + + > Permissions (currently "read" and "write") may be restricted independently depending on the scenario. + +7. [Set up grants](#manage-grants) to remove restrictions for specified users and roles. + +--- + +{{% note %}} +#### Notes about examples +The examples below use `curl`, a command line tool for transferring data, to send +HTTP requests to the Meta API, and [`jq`](https://stedolan.github.io/jq/), a command line JSON processor, +to make the JSON output easier to read. +Alternatives for each are available, but are not covered in this documentation. + +All examples assume authentication is enabled in InfluxDB. +Admin credentials must be sent with each request. +Use the `curl -u` flag to pass authentication credentials: + +```sh +curl -u `username:password` #... +``` +{{% /note %}} + +--- + +## Matching methods +The following matching methods are available when managing restrictions and grants to databases, measurements, or series: + +- `exact` (matches only exact string matches) +- `prefix` (matches strings the begin with a specified prefix) + +```sh +# Match a database name exactly +"database": {"match": "exact", "value": "my_database"} + +# Match any databases that begin with "my_" +"database": {"match": "prefix", "value": "my_"} +``` + +{{% note %}} +#### Wildcard matching +Neither `exact` nor `prefix` matching methods allow for wildcard matching. +{{% /note %}} + +## Manage roles +Roles allow you to assign permissions to groups of users. +The following examples assume the `user1`, `user2` and `ops` users already exist in InfluxDB. + +### Create a role +To create a new role, use the InfluxDB Meta API `/role` endpoint with the `action` +field set to `create` in the request body. + +The following examples create two new roles: + +- east +- west + +```sh +# Create east role +curl -s -L -XPOST "http://localhost:8091/role" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "action": "create", + "role": { + "name": "east" + } + }' + +# Create west role +curl -s -L -XPOST "http://localhost:8091/role" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "action": "create", + "role": { + "name": "west" + } + }' +``` + +### Specify role permissions +To specify permissions for a role, +use the InfluxDB Meta API `/role` endpoint with the `action` field set to `add-permissions`. +Specify the [permissions](/chronograf/v1.8/administration/managing-influxdb-users/#permissions) to add for each database. + +The following example sets read and write permissions on `db1` for both `east` and `west` roles. + +```sh +curl -s -L -XPOST "http://localhost:8091/role" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "action": "add-permissions", + "role": { + "name": "east", + "permissions": { + "db1": ["ReadData", "WriteData"] + } + } + }' + +curl -s -L -XPOST "http://localhost:8091/role" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "action": "add-permissions", + "role": { + "name": "west", + "permissions": { + "db1": ["ReadData", "WriteData"] + } + } + }' +``` + +### Remove role permissions +To remove permissions from a role, use the InfluxDB Meta API `/role` endpoint with the `action` field +set to `remove-permissions`. +Specify the [permissions](/{{< latest "chronograf" >}}/administration/managing-influxdb-users/#permissions) to remove from each database. + +The following example removes read and write permissions from `db1` for the `east` role. + +```sh +curl -s -L -XPOST "http://localhost:8091/role" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "action": "remove-permissions", + "role": { + "name": "east", + "permissions": { + "db1": ["ReadData", "WriteData"] + } + } + }' +``` + +### Assign users to a role +To assign users to role, set the `action` field to `add-users` and include a list +of users in the `role` field. + +The following examples add user1, user2 and the ops user to the `east` and `west` roles. + +```sh +# Add user1 and ops to the east role +curl -s -L -XPOST "http://localhost:8091/role" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "action": "add-users", + "role": { + "name": "east", + "users": ["user1", "ops"] + } + }' + +# Add user1 and ops to the west role +curl -s -L -XPOST "http://localhost:8091/role" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "action": "add-users", + "role": { + "name": "west", + "users": ["user2", "ops"] + } + }' +``` + +### View existing roles +To view existing roles with their assigned permissions and users, use the `GET` +request method with the InfluxDB Meta API `/role` endpoint. + +```sh +curl -L -XGET http://localhost:8091/role | jq +``` + +### Delete a role +To delete a role, the InfluxDB Meta API `/role` endpoint and set the `action` +field to `delete` and include the name of the role to delete. + +```sh +curl -s -L -XPOST "http://localhost:8091/role" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "action": "delete", + "role": { + "name": "west" + } + }' +``` + +{{% note %}} +Deleting a role does not delete users assigned to the role. +{{% /note %}} + +## Manage restrictions +Restrictions restrict either or both read and write permissions on InfluxDB assets. +Restrictions apply to all non-admin users. +[Grants](#manage-grants) override restrictions. + +> In order to run meta queries (such as `SHOW MEASUREMENTS` or `SHOW TAGS` ), +> users must have read permissions for the database and retention policy they are querying. + +Manage restrictions using the InfluxDB Meta API `acl/restrictions` endpoint. + +```sh +curl -L -XGET "http://localhost:8091/influxdb/v2/acl/restrictions" +``` + +- [Restrict by database](#restrict-by-database) +- [Restrict by measurement in a database](#restrict-by-measurement-in-a-database) +- [Restrict by series in a database](#restrict-by-series-in-a-database) +- [View existing restrictions](#view-existing-restrictions) +- [Update a restriction](#update-a-restriction) +- [Remove a restriction](#remove-a-restriction) + +> **Note:** For the best performance, set up minimal restrictions. + +### Restrict by database +In most cases, restricting the database is the simplest option, and has minimal impact on performance. +The following example restricts reads and writes on the `my_database` database. + +```sh +curl -L -XPOST "http://localhost:8091/influxdb/v2/acl/restrictions" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "database": {"match": "exact", "value": "my_database"}, + "permissions": ["read", "write"] + }' +``` + +### Restrict by measurement in a database +The following example restricts read and write permissions on the `network` +measurement in the `my_database` database. +_This restriction does not apply to other measurements in the `my_database` database._ + +```sh +curl -L -XPOST "http://localhost:8091/influxdb/v2/acl/restrictions" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "database": {"match": "exact", "value": "my_database"}, + "measurement": {"match": "exact", "value": "network"}, + "permissions": ["read", "write"] + }' +``` + +### Restrict by series in a database +The most fine-grained restriction option is to restrict specific tags in a measurement and database. +The following example restricts read and write permissions on the `datacenter=east` tag in the +`network` measurement in the `my_database` database. +_This restriction does not apply to other tags or tag values in the `network` measurement._ + +```sh +curl -L -XPOST "http://localhost:8091/influxdb/v2/acl/restrictions" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "database": {"match": "exact", "value": "my_database"}, + "measurement": {"match": "exact", "value": "network"}, + "tags": [{"match": "exact", "key": "datacenter", "value": "east"}], + "permissions": ["read", "write"] + }' +``` + +_Consider this option carefully, as it allows writes to `network` without tags or +writes to `network` with a tag key of `datacenter` and a tag value of anything but `east`._ + +##### Apply restrictions to a series defined by multiple tags +```sh +curl -L -XPOST "http://localhost:8091/influxdb/v2/acl/restrictions" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "database": {"match": "exact", "value": "my_database"}, + "measurement": {"match": "exact", "value": "network"}, + "tags": [ + {"match": "exact", "key": "tag1", "value": "value1"}, + {"match": "exact", "key": "tag2", "value": "value2"} + ], + "permissions": ["read", "write"] + }' +``` + +{{% note %}} +#### Create multiple restrictions at a time +There may be times where you need to create restrictions using unique values for each. +To create multiple restrictions for a list of values, use a bash `for` loop: + +```sh +for value in val1 val2 val3 val4; do + curl -L -XPOST "http://localhost:8091/influxdb/v2/acl/restrictions" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "database": {"match": "exact", "value": "my_database"}, + "measurement": {"match": "exact", "value": "network"}, + "tags": [{"match": "exact", "key": "datacenter", "value": "'$value'"}], + "permissions": ["read", "write"] + }' +done +``` +{{% /note %}} + +### View existing restrictions +To view existing restrictions, use the `GET` request method with the `acl/restrictions` endpoint. + +```sh +curl -L -u "admin-username:admin-password" -XGET "http://localhost:8091/influxdb/v2/acl/restrictions" | jq +``` + +### Update a restriction +_You can not directly modify a restriction. +Delete the existing restriction and create a new one with updated parameters._ + +### Remove a restriction +To remove a restriction, obtain the restriction ID using the `GET` request method +with the `acl/restrictions` endpoint. +Use the `DELETE` request method to delete a restriction by ID. + +```sh +# Obtain the restriction ID from the list of restrictions +curl -L -u "admin-username:admin-password" \ + -XGET "http://localhost:8091/influxdb/v2/acl/restrictions" | jq + +# Delete the restriction using the restriction ID +curl -L -u "admin-username:admin-password" \ + -XDELETE "http://localhost:8091/influxdb/v2/acl/restrictions/" +``` + +## Manage grants +Grants remove restrictions and grant users or roles either or both read and write +permissions on InfluxDB assets. + +Manage grants using the InfluxDB Meta API `acl/grants` endpoint. + +```sh +curl -L -u "admin-username:admin-password" \ + -XGET "http://localhost:8091/influxdb/v2/acl/grants" +``` + +- [Grant permissions by database](#grant-permissions-by-database) +- [Grant permissions by measurement in a database](#grant-permissions-by-measurement-in-a-database) +- [Grant permissions by series in a database](#grant-permissions-by-series-in-a-database) +- [View existing grants](#view-existing-grants) +- [Update a grant](#update-a-grant) +- [Remove a grant](#remove-a-grant) + +### Grant permissions by database +The following examples grant read and write permissions on the `my_database` database. + +> **Note:** This offers no guarantee that the users will write to the correct measurement or use the correct tags. + +##### Grant database-level permissions to users +```sh +curl -s -L -XPOST "http://localhost:8091/influxdb/v2/acl/grants" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "database": {"match": "exact", "value": "my_database"}, + "permissions": ["read", "write"], + "users": [ + {"name": "user1"}, + {"name": "user2"} + ] + }' +``` + +##### Grant database-level permissions to roles +```sh +curl -s -L -XPOST "http://localhost:8091/influxdb/v2/acl/grants" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "database": {"match": "exact", "value": "my_database"}, + "permissions": ["read", "write"], + "roles": [ + {"name": "role1"}, + {"name": "role2"} + ] + }' +``` + +### Grant permissions by measurement in a database +The following examples grant permissions to the `network` measurement in the `my_database` database. +These grants do not apply to other measurements in the `my_database` database nor +guarantee that users will use the correct tags. + +##### Grant measurement-level permissions to users +```sh +curl -s -L -XPOST "http://localhost:8091/influxdb/v2/acl/grants" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "database": {"match": "exact", "value": "my_database"}, + "measurement": {"match": "exact", "value": "network"}, + "permissions": ["read", "write"], + "users": [ + {"name": "user1"}, + {"name": "user2"} + ] + }' +``` + +To grant access for roles, run: + +```sh +curl -s -L -XPOST "http://localhost:8091/influxdb/v2/acl/grants" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "database": {"match": "exact", "value": "my_database"}, + "measurement": {"match": "exact", "value": "network"}, + "permissions": ["read", "write"], + "roles": [ + {"name": "role1"}, + {"name": "role2"} + ] + }' +``` + +### Grant permissions by series in a database + +The following examples grant access only to data with the corresponding `datacenter` tag. +_Neither guarantees the users will use the `network` measurement._ + +##### Grant series-level permissions to users +```sh +# Grant user1 read/write permissions on data with the 'datacenter=east' tag set. +curl -s -L -XPOST "http://localhost:8091/influxdb/v2/acl/grants" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "database": {"match": "exact", "value": "my_database"}, + "tags": [{"match": "exact", "key": "datacenter", "value": "east"}], + "permissions": ["read", "write"], + "users": [{"name": "user1"}] + }' + +# Grant user2 read/write permissions on data with the 'datacenter=west' tag set. +curl -s -L -XPOST "http://localhost:8091/influxdb/v2/acl/grants" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "database": {"match": "exact", "value": "my_database"}, + "tags": [{"match": "exact", "key": "datacenter", "value": "west"}], + "permissions": ["read", "write"], + "users": [{"name": "user2"}] + }' +``` + +##### Grant series-level permissions to roles +```sh +# Grant role1 read/write permissions on data with the 'datacenter=east' tag set. +curl -s -L -XPOST "http://localhost:8091/influxdb/v2/acl/grants" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "database": {"match": "exact", "value": "my_database"}, + "tags": [{"match": "exact", "key": "datacenter", "value": "east"}], + "permissions": ["read", "write"], + "roles": [{"name": "role1"}] + }' + +# Grant role2 read/write permissions on data with the 'datacenter=west' tag set. +curl -s -L -XPOST "http://localhost:8091/influxdb/v2/acl/grants" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "database": {"match": "exact", "value": "my_database"}, + "tags": [{"match": "exact", "key": "datacenter", "value": "west"}], + "permissions": ["read", "write"], + "roles": [{"name": "role2"}] + }' +``` + +### Grant access to specific series in a measurement +The following examples grant read and write permissions to corresponding `datacenter` +tags in the `network` measurement. +_They each specify the measurement in the request body._ + +##### Grant series-level permissions in a measurement to users +```sh +# Grant user1 read/write permissions on data with the 'datacenter=west' tag set +# inside the 'network' measurement. +curl -s -L -XPOST "http://localhost:8091/influxdb/v2/acl/grants" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "database": {"match": "exact", "value": "my_database"}, + "measurement": {"match": "exact", "value": "network"}, + "tags": [{"match": "exact", "key": "datacenter", "value": "east"}], + "permissions": ["read", "write"], + "users": [{"name": "user1"}] + }' + +# Grant user2 read/write permissions on data with the 'datacenter=west' tag set +# inside the 'network' measurement. +curl -s -L -XPOST "http://localhost:8091/influxdb/v2/acl/grants" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "database": {"match": "exact", "value": "my_database"}, + "measurement": {"match": "exact", "value": "network"}, + "tags": [{"match": "exact", "key": "datacenter", "value": "west"}], + "permissions": ["read", "write"], + "users": [{"name": "user2"}] + }' +``` + +##### Grant series-level permissions in a measurement to roles +```sh +# Grant role1 read/write permissions on data with the 'datacenter=west' tag set +# inside the 'network' measurement. +curl -s -L -XPOST "http://localhost:8091/influxdb/v2/acl/grants" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "database": {"match": "exact", "value": "my_database"}, + "measurement": {"match": "exact", "value": "network"}, + "tags": [{"match": "exact", "key": "datacenter", "value": "east"}], + "permissions": ["read", "write"], + "roles": [{"name": "role1"}] + }' + +# Grant role2 read/write permissions on data with the 'datacenter=west' tag set +# inside the 'network' measurement. +curl -s -L -XPOST "http://localhost:8091/influxdb/v2/acl/grants" \ + -u "admin-username:admin-password" \ + -H "Content-Type: application/json" \ + --data-binary '{ + "database": {"match": "exact", "value": "my_database"}, + "measurement": {"match": "exact", "value": "network"}, + "tags": [{"match": "exact", "key": "datacenter", "value": "west"}], + "permissions": ["read", "write"], + "roles": [{"name": "role2"}] + }' +``` + +Grants for specific series also apply to [meta queries](/enterprise_influxdb/v1.9/query_language/schema_exploration). +Results from meta queries are restricted based on series-level permissions. +For example, `SHOW TAG VALUES` only returns tag values that the user is authorized to see. + +With these grants in place, a user or role can only read or write data from or to +the `network` measurement if the data includes the appropriate `datacenter` tag set. + +{{% note %}} +Note that this is only the requirement of the presence of that tag; +`datacenter=east,foo=bar` will also be accepted. +{{% /note %}} + +### View existing grants +To view existing grants, use the `GET` request method with the `acl/grants` endpoint. + +```sh +curl -L -u "admin-username:admin-password" \ + -XGET "http://localhost:8091/influxdb/v2/acl/grants" | jq +``` + +### Update a grant +_You can not directly modify a grant. +Delete the existing grant and create a new one with updated parameters._ + +### Remove a grant +To delete a grant, obtain the grant ID using the `GET` request method with the +`acl/grants` endpoint. +Use the `DELETE` request method to delete a grant by ID. + +```sh +# Obtain the grant ID from the list of grants +curl -L -u "admin-username:admin-password" \ + -XGET "http://localhost:8091/influxdb/v2/acl/grants" | jq + +# Delete the grant using the grant ID +curl -L -u "admin-username:admin-password" \ + -XDELETE "http://localhost:8091/influxdb/v2/acl/grants/" +``` diff --git a/content/enterprise_influxdb/v1.9/guides/https_setup.md b/content/enterprise_influxdb/v1.9/guides/https_setup.md new file mode 100644 index 000000000..b6773937a --- /dev/null +++ b/content/enterprise_influxdb/v1.9/guides/https_setup.md @@ -0,0 +1,296 @@ +--- +title: Enable HTTPS for InfluxDB Enterprise +description: > + Enabling HTTPS encrypts the communication between clients and the InfluxDB Enterprise server, and between nodes in the cluster. +menu: + enterprise_influxdb_1_9: + name: Enable HTTPS + weight: 100 + parent: Guides +--- + +Enabling HTTPS encrypts the communication between clients and the InfluxDB Enterprise server, and between nodes in the cluster. +When configured with a signed certificate, HTTPS can also verify the authenticity of the InfluxDB Enterprise server to connecting clients. + +This pages outlines how to set up HTTPS with InfluxDB Enterprise using either a signed or self-signed certificate. + +{{% warn %}} +InfluxData **strongly recommends** enabling HTTPS, especially if you plan on sending requests to InfluxDB Enterprise over a network. +{{% /warn %}} + +{{% note %}} +These steps have been tested on Debian-based Linux distributions. +Specific steps may vary on other operating systems. +{{% /note %}} + +## Requirements + +To enable HTTPS with InfluxDB Enterprise, you need a Transport Layer Security (TLS) certificate, also known as a Secured Sockets Layer (SSL) certificate. +InfluxDB supports three types of TLS certificates: + +* **Single domain certificates signed by a [Certificate Authority](https://en.wikipedia.org/wiki/Certificate_authority)** + + Single domain certificates provide cryptographic security to HTTPS requests and allow clients to verify the identity of the InfluxDB server. + These certificates are signed and issued by a trusted, third-party Certificate Authority (CA). + With this certificate option, every InfluxDB instance requires a unique single domain certificate. + +* **Wildcard certificates signed by a Certificate Authority** + + These certificates provide cryptographic security to HTTPS requests and allow clients to verify the identity of the InfluxDB server. + Wildcard certificates can be used across multiple InfluxDB Enterprise instances on different servers. + +* **Self-signed certificates** + + Self-signed certificates are _not_ signed by a trusted, third-party CA. + Unlike CA-signed certificates, self-signed certificates only provide cryptographic security to HTTPS requests. + They do not allow clients to verify the identity of the InfluxDB server. + With this certificate option, every InfluxDB Enterprise instance requires a unique self-signed certificate. + You can generate a self-signed certificate on your own machine. + +Regardless of your certificate's type, InfluxDB Enterprise supports certificates composed of +a private key file (`.key`) and a signed certificate file (`.crt`) file pair, as well as certificates +that combine the private key file and the signed certificate file into a single bundled file (`.pem`). + +## Set up HTTPS in an InfluxDB Enterprise cluster + +1. **Download or generate certificate files** + + If using a certificate provided by a CA, follow their instructions to download the certificate files. + + {{% note %}} +If using one or more self-signed certificates, use the `openssl` utility to create a certificate. +The following command generates a private key file (`.key`) and a self-signed +certificate file (`.crt`) which remain valid for the specified `NUMBER_OF_DAYS`. + +```sh +sudo openssl req -x509 -nodes -newkey rsa:2048 \ + -keyout influxdb-selfsigned.key \ + -out influxdb-selfsigned.crt \ + -days +``` + +The command will prompt you for more information. +You can choose to fill out these fields or leave them blank; both actions generate valid certificate files. + +In subsequent steps, you will need to copy the certificate and key (or `.pem` file) to each node in the cluster. + {{% /note %}} + +2. **Install the SSL/TLS certificate in each Node** + + Place the private key file (`.key`) and the signed certificate file (`.crt`) + or the single bundled file (`.pem`) + in the `/etc/ssl/` directory of each meta node and data node. + + {{% note %}} +Some Certificate Authorities provide certificate files with other extensions. +Consult your CA if you are unsure about how to use these files. + {{% /note %}} + +3. **Ensure file permissions for each Node** + + Certificate files require read and write access by the `influxdb` user. + Ensure that you have the correct file permissions in each meta node and data node by running the following commands: + + ```sh + sudo chown influxdb:influxdb /etc/ssl/ + sudo chmod 644 /etc/ssl/ + sudo chmod 600 /etc/ssl/ + ``` + +4. **Enable HTTPS within the configuration file for each meta node** + + Enable HTTPS for each meta node within the `[meta]` section of the meta node configuration file (`influxdb-meta.conf`) by setting: + + ```toml + [meta] + + [...] + + # Determines whether HTTPS is enabled. + https-enabled = true + + # The SSL certificate to use when HTTPS is enabled. + https-certificate = "influxdb-meta.crt" + + # Use a separate private key location. + https-private-key = "influxdb-meta.key" + + # If using a self-signed certificate: + https-insecure-tls = true + + # Use TLS when communicating with data notes + data-use-tls = true + data-insecure-tls = true + + ``` + +5. **Enable HTTPS within the configuration file for each data node** + + Make the following sets of changes in the configuration file (`influxdb.conf`) on each data node: + + 1. Enable HTTPS for each data node within the `[http]` section of the configuration file by setting: + + ```toml + [http] + + [...] + + # Determines whether HTTPS is enabled. + https-enabled = true + + [...] + + # The SSL certificate to use when HTTPS is enabled. + https-certificate = "influxdb-data.crt" + + # Use a separate private key location. + https-private-key = "influxdb-data.key" + ``` + + 2. Configure the data nodes to use HTTPS when communicating with other data nodes. + In the `[cluster]` section of the configuration file, set the following: + + ```toml + [cluster] + + [...] + + # Determines whether data nodes use HTTPS to communicate with each other. + https-enabled = true + + # The SSL certificate to use when HTTPS is enabled. + https-certificate = "influxdb-data.crt" + + # Use a separate private key location. + https-private-key = "influxdb-data.key" + + # If using a self-signed certificate: + https-insecure-tls = true + ``` + + 3. Configure the data nodes to use HTTPS when communicating with the meta nodes. + In the `[meta]` section of the configuration file, set the following: + + ```toml + [meta] + + [...] + meta-tls-enabled = true + + # If using a self-signed certificate: + meta-insecure-tls = true + ``` + +6. **Restart InfluxDB Enterprise** + + Restart the InfluxDB Enterprise processes for the configuration changes to take effect: + + ```sh + sudo systemctl restart influxdb-meta + ``` + + Restart the InfluxDB Enterprise data node processes for the configuration changes to take effect: + + ```sh + sudo systemctl restart influxdb + ``` + +7. **Verify the HTTPS Setup** + + Verify that HTTPS is working on the meta nodes by using `influxd-ctl`. + + ```sh + influxd-ctl -bind-tls show + ``` + + If using a self-signed certificate, use: + + ```sh + influxd-ctl -bind-tls -k show + ``` + + {{% warn %}} +Once you have enabled HTTPS, you must use `-bind-tls` in order for `influxd-ctl` to connect to the meta node. +With a self-signed certificate, you must also use the `-k` option to skip certificate verification. + {{% /warn %}} + + A successful connection returns output which should resemble the following: + + ``` + Data Nodes + ========== + ID TCP Address Version + 4 enterprise-data-01:8088 1.x.y-c1.x.y + 5 enterprise-data-02:8088 1.x.y-c1.x.y + + Meta Nodes + ========== + TCP Address Version + enterprise-meta-01:8091 1.x.y-c1.x.z + enterprise-meta-02:8091 1.x.y-c1.x.z + enterprise-meta-03:8091 1.x.y-c1.x.z + ``` + + Next, verify that HTTPS is working by connecting to InfluxDB Enterprise with the [`influx` command line interface](/enterprise_influxdb/v1.9/tools/use-influx/): + + ```sh + influx -ssl -host .com + ``` + + If using a self-signed certificate, use + + ```sh + influx -ssl -unsafeSsl -host .com + ``` + + A successful connection returns the following: + + ```sh + Connected to https://.com:8086 version 1.x.y + InfluxDB shell version: 1.x.y + > + ``` + + That's it! You've successfully set up HTTPS with InfluxDB Enterprise. + +## Connect Telegraf to a secured InfluxDB Enterprise instance + +Connecting [Telegraf](/{{< latest "telegraf" >}}/) +to an HTTPS-enabled InfluxDB Enterprise instance requires some additional steps. + +In Telegraf's configuration file (`/etc/telegraf/telegraf.conf`), under the OUTPUT PLUGINS section, +edit the `urls` setting to indicate `https` instead of `http`. +Also change `localhost` to the relevant domain name. + +The best practice in terms of security is to transfer the certificate to the client and make it trusted +(either by putting in the operating system's trusted certificate system or using the `ssl_ca` option). +The alternative is to sign the certificate using an internal CA and then trust the CA certificate. + +If you're using a self-signed certificate, +uncomment the `insecure_skip_verify` setting and set it to `true`. + +```toml +############################################################################### +# OUTPUT PLUGINS # +############################################################################### + +# Configuration for influxdb server to send metrics to +[[outputs.influxdb]] + ## The full HTTP or UDP endpoint URL for your InfluxDB Enterprise instance. + ## Multiple urls can be specified as part of the same cluster, + ## this means that only ONE of the urls will be written to each interval. + # urls = ["udp://localhost:8089"] # UDP endpoint example + urls = ["https://.com:8086"] + +[...] + + ## Optional SSL Config + [...] + insecure_skip_verify = true # <-- Update only if you're using a self-signed certificate +``` + +Next, restart Telegraf and you're all set! + +```sh +sudo systemctl restart telegraf +``` diff --git a/content/enterprise_influxdb/v1.9/guides/query_data.md b/content/enterprise_influxdb/v1.9/guides/query_data.md new file mode 100644 index 000000000..e22befc45 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/guides/query_data.md @@ -0,0 +1,111 @@ +--- +title: Query data with the InfluxDB API +description: Query data with Flux and InfluxQL in the InfluxDB API. +alias: + -/docs/v1.8/query_language/querying_data/ +menu: + enterprise_influxdb_1_9: + weight: 20 + parent: Guides +aliases: + - /enterprise_influxdb/v1.9/guides/querying_data/ +v2: /influxdb/v2.0/query-data/ +--- + + +The InfluxDB API is the primary means for querying data in InfluxDB (see the [command line interface](/enterprise_influxdb/v1.9/tools/use-influx/) and [client libraries](/enterprise_influxdb/v1.9/tools/api_client_libraries/) for alternative ways to query the database). + +Query data with the InfluxDB API using [Flux](#query-data-with-flux) or [InfluxQL](#query-data-with-influxql). + +> **Note**: The following examples use `curl`, a command line tool that transfers data using URLs. Learn the basics of `curl` with the [HTTP Scripting Guide](https://curl.haxx.se/docs/httpscripting.html). + +## Query data with Flux + +For Flux queries, the `/api/v2/query` endpoint accepts `POST` HTTP requests. Use the following HTTP headers: +- `Accept: application/csv` +- `Content-type: application/vnd.flux` + +If you have authentication enabled, provide your InfluxDB username and password with the `Authorization` header and `Token` schema. For example: `Authorization: Token username:password`. + + +The following example queries Telegraf data using Flux: +: + +```bash +$ curl -XPOST localhost:8086/api/v2/query -sS \ + -H 'Accept:application/csv' \ + -H 'Content-type:application/vnd.flux' \ + -d 'from(bucket:"telegraf") + |> range(start:-5m) + |> filter(fn:(r) => r._measurement == "cpu")' +``` +Flux returns [annotated CSV](/influxdb/v2.0/reference/syntax/annotated-csv/): + +``` +{,result,table,_start,_stop,_time,_value,_field,_measurement,cpu,host +,_result,0,2020-04-07T18:02:54.924273Z,2020-04-07T19:02:54.924273Z,2020-04-07T18:08:19Z,4.152553004641827,usage_user,cpu,cpu-total,host1 +,_result,0,2020-04-07T18:02:54.924273Z,2020-04-07T19:02:54.924273Z,2020-04-07T18:08:29Z,7.608695652173913,usage_user,cpu,cpu-total,host1 +,_result,0,2020-04-07T18:02:54.924273Z,2020-04-07T19:02:54.924273Z,2020-04-07T18:08:39Z,2.9363988504310883,usage_user,cpu,cpu-total,host1 +,_result,0,2020-04-07T18:02:54.924273Z,2020-04-07T19:02:54.924273Z,2020-04-07T18:08:49Z,6.915093159934975,usage_user,cpu,cpu-total,host1} +``` + +The header row defines column labels for the table. The `cpu` [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) has four points, each represented by one of the record rows. For example the first point has a [timestamp](/enterprise_influxdb/v1.9/concepts/glossary/#timestamp) of `2020-04-07T18:08:19`. + +### Flux + +Check out the [Get started with Flux](/influxdb/v2.0/query-data/get-started/) to learn more about building queries with Flux. +For more information about querying data with the InfluxDB API using Flux, see the [API reference documentation](/enterprise_influxdb/v1.9/tools/api/#influxdb-2-0-api-compatibility-endpoints). + +## Query data with InfluxQL + +To perform an InfluxQL query, send a `GET` request to the `/query` endpoint, set the URL parameter `db` as the target database, and set the URL parameter `q` as your query. +You can also use a `POST` request by sending the same parameters either as URL parameters or as part of the body with `application/x-www-form-urlencoded`. +The example below uses the InfluxDB API to query the same database that you encountered in [Writing Data](/enterprise_influxdb/v1.9/guides/writing_data/). + +```bash +curl -G 'http://localhost:8086/query?pretty=true' --data-urlencode "db=mydb" --data-urlencode "q=SELECT \"value\" FROM \"cpu_load_short\" WHERE \"region\"='us-west'" +``` + +InfluxDB returns JSON: + + +```json +{ + "results": [ + { + "statement_id": 0, + "series": [ + { + "name": "cpu_load_short", + "columns": [ + "time", + "value" + ], + "values": [ + [ + "2015-01-29T21:55:43.702900257Z", + 2 + ], + [ + "2015-01-29T21:55:43.702900257Z", + 0.55 + ], + [ + "2015-06-11T20:46:02Z", + 0.64 + ] + ] + } + ] + } + ] +} +``` + +> **Note:** Appending `pretty=true` to the URL enables pretty-printed JSON output. +While this is useful for debugging or when querying directly with tools like `curl`, it is not recommended for production use as it consumes unnecessary network bandwidth. + +### InfluxQL + +Check out the [Data Exploration page](/enterprise_influxdb/v1.9/query_language/explore-data/) to get acquainted with InfluxQL. +For more information about querying data with the InfluxDB API using InfluxQL, see the [API reference documentation](/enterprise_influxdb/v1.9/tools/api/#influxdb-1-x-http-endpoints). diff --git a/content/enterprise_influxdb/v1.9/guides/rebalance.md b/content/enterprise_influxdb/v1.9/guides/rebalance.md new file mode 100644 index 000000000..dba32ed93 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/guides/rebalance.md @@ -0,0 +1,419 @@ +--- +title: Rebalance InfluxDB Enterprise clusters +description: Manually rebalance an InfluxDB Enterprise cluster. +aliases: + - /enterprise/v1.8/guides/rebalance/ +menu: + enterprise_influxdb_1_9: + name: Rebalance clusters + weight: 19 + parent: Guides +--- + +## Introduction + +This guide describes how to manually rebalance an InfluxDB Enterprise cluster. +Rebalancing a cluster involves two primary goals: + +* Evenly distribute +[shards](/enterprise_influxdb/v1.9/concepts/glossary/#shard) across all data nodes in the +cluster +* Ensure that every +shard is on *n* number of nodes, where *n* is determined by the retention policy's +[replication factor](/enterprise_influxdb/v1.9/concepts/glossary/#replication-factor) + +Rebalancing a cluster is essential for cluster health. +Perform a rebalance if you add a new data node to your cluster. +The proper rebalance path depends on the purpose of the new data node. +If you added a data node to expand the disk size of the cluster or increase +write throughput, follow the steps in +[Rebalance Procedure 1](#rebalance-procedure-1-rebalance-a-cluster-to-create-space). +If you added a data node to increase data availability for queries and query +throughput, follow the steps in +[Rebalance Procedure 2](#rebalance-procedure-2-rebalance-a-cluster-to-increase-availability). + +### Requirements + +The following sections assume that you already added a new data node to the +cluster, and they use the +[`influxd-ctl` tool](/enterprise_influxdb/v1.9/tools/influxd-ctl/) available on +all meta nodes. + +{{% warn %}} +Before you begin, stop writing historical data to InfluxDB. +Historical data have timestamps that occur at anytime in the past. +Performing a rebalance while writing historical data can lead to data loss. +{{% /warn %}} + +## Rebalance Procedure 1: Rebalance a cluster to create space + +For demonstration purposes, the next steps assume that you added a third +data node to a previously two-data-node cluster that has a +[replication factor](/enterprise_influxdb/v1.9/concepts/glossary/#replication-factor) of +two. +This rebalance procedure is applicable for different cluster sizes and +replication factors, but some of the specific, user-provided values will depend +on that cluster size. + +Rebalance Procedure 1 focuses on how to rebalance a cluster after adding a +data node to expand the total disk capacity of the cluster. +In the next steps, you will safely move shards from one of the two original data +nodes to the new data node. + +### Step 1: Truncate Hot Shards + +Hot shards are shards that are currently receiving writes. +Performing any action on a hot shard can lead to data inconsistency within the +cluster which requires manual intervention from the user. + +To prevent data inconsistency, truncate hot shards before moving any shards +across data nodes. +The command below creates a new hot shard which is automatically distributed +across all data nodes in the cluster, and the system writes all new points to +that shard. +All previous writes are now stored in cold shards. + +``` +influxd-ctl truncate-shards +``` + +The expected ouput of this command is: + +``` +Truncated shards. +``` + +Once you truncate the shards, you can work on redistributing the cold shards +without the threat of data inconsistency in the cluster. +Any hot or new shards are now evenly distributed across the cluster and require +no further intervention. + +### Step 2: Identify Cold Shards + +In this step, you identify the cold shards that you will copy to the new data node +and remove from one of the original two data nodes. + +The following command lists every shard in our cluster: + +``` +influxd-ctl show-shards +``` + +The expected output is similar to the items in the codeblock below: + +``` +Shards +========== +ID Database Retention Policy Desired Replicas [...] End Owners +21 telegraf autogen 2 [...] 2017-01-26T18:00:00Z [{4 enterprise-data-01:8088} {5 enterprise-data-02:8088}] +22 telegraf autogen 2 [...] 2017-01-26T18:05:36.418734949Z* [{4 enterprise-data-01:8088} {5 enterprise-data-02:8088}] +24 telegraf autogen 2 [...] 2017-01-26T19:00:00Z [{5 enterprise-data-02:8088} {6 enterprise-data-03:8088}] +``` + +The sample output includes three shards. +The first two shards are cold shards. +The timestamp in the `End` column occurs in the past (assume that the current +time is just after `2017-01-26T18:05:36.418734949Z`), and the shards' owners +are the two original data nodes: `enterprise-data-01:8088` and +`enterprise-data-02:8088`. +The second shard is the truncated shard; truncated shards have an asterix (`*`) +on the timestamp in the `End` column. + +The third shard is the newly-created hot shard; the timestamp in the `End` +column is in the future (again, assume that the current time is just after +`2017-01-26T18:05:36.418734949Z`), and the shard's owners include one of the +original data nodes (`enterprise-data-02:8088`) and the new data node +(`enterprise-data-03:8088`). +That hot shard and any subsequent shards require no attention during +the rebalance process. + +Identify the cold shards that you'd like to move from one of the original two +data nodes to the new data node. +Take note of the cold shard's `ID` (for example: `22`) and the TCP address of +one of its owners in the `Owners` column (for example: +`enterprise-data-01:8088`). + +> **Note:** +> +Use the following command string to determine the size of the shards in +your cluster: +> + find /var/lib/influxdb/data/ -mindepth 3 -type d -exec du -h {} \; +> +In general, we recommend moving larger shards to the new data node to increase the +available disk space on the original data nodes. +Users should note that moving shards will impact network traffic. + +### Step 3: Copy Cold Shards + +Next, copy the relevant cold shards to the new data node with the syntax below. +Repeat this command for every cold shard that you'd like to move to the +new data node. + +``` +influxd-ctl copy-shard +``` + +Where `source_TCP_address` is the address that you noted in step 2, +`destination_TCP_address` is the TCP address of the new data node, and `shard_ID` +is the ID of the shard that you noted in step 2. + +The expected output of the command is: +``` +Copied shard from to +``` + +### Step 4: Confirm the Copied Shards + +Confirm that the TCP address of the new data node appears in the `Owners` column +for every copied shard: + +``` +influxd-ctl show-shards +``` + +The expected output shows that the copied shard now has three owners: +``` +Shards +========== +ID Database Retention Policy Desired Replicas [...] End Owners +22 telegraf autogen 2 [...] 2017-01-26T18:05:36.418734949Z* [{4 enterprise-data-01:8088} {5 enterprise-data-02:8088} {6 enterprise-data-03:8088}] +``` + +In addition, verify that the copied shards appear in the new data node's shard +directory and match the shards in the source data node's shard directory. +Shards are located in +`/var/lib/influxdb/data///`. + +Here's an example of the correct output for shard `22`: +``` +# On the source data node (enterprise-data-01) + +~# ls /var/lib/influxdb/data/telegraf/autogen/22 +000000001-000000001.tsm # 👍 + +# On the new data node (enterprise-data-03) + +~# ls /var/lib/influxdb/data/telegraf/autogen/22 +000000001-000000001.tsm # 👍 +``` + +It is essential that every copied shard appears on the new data node both +in the `influxd-ctl show-shards` output and in the shard directory. +If a shard does not pass both of the tests above, please repeat step 3. + +### Step 5: Remove Unnecessary Cold Shards + +Next, remove the copied shard from the original data node with the command below. +Repeat this command for every cold shard that you'd like to remove from one of +the original data nodes. +**Removing a shard is an irrecoverable, destructive action; please be +cautious with this command.** + +``` +influxd-ctl remove-shard +``` + +Where `source_TCP_address` is the TCP address of the original data node and +`shard_ID` is the ID of the shard that you noted in step 2. + +The expected output of the command is: +``` +Removed shard from +``` + +### Step 6: Confirm the Rebalance + +For every relevant shard, confirm that the TCP address of the new data node and +only one of the original data nodes appears in the `Owners` column: + +``` +influxd-ctl show-shards +``` + +The expected output shows that the copied shard now has only two owners: +``` +Shards +========== +ID Database Retention Policy Desired Replicas [...] End Owners +22 telegraf autogen 2 [...] 2017-01-26T18:05:36.418734949Z* [{5 enterprise-data-02:8088} {6 enterprise-data-03:8088}] +``` + +That's it. +You've successfully rebalanced your cluster; you expanded the available disk +size on the original data nodes and increased the cluster's write throughput. + +## Rebalance Procedure 2: Rebalance a cluster to increase availability + +For demonstration purposes, the next steps assume that you added a third +data node to a previously two-data-node cluster that has a +[replication factor](/enterprise_influxdb/v1.9/concepts/glossary/#replication-factor) of +two. +This rebalance procedure is applicable for different cluster sizes and +replication factors, but some of the specific, user-provided values will depend +on that cluster size. + +Rebalance Procedure 2 focuses on how to rebalance a cluster to improve availability +and query throughput. +In the next steps, you will increase the retention policy's replication factor and +safely copy shards from one of the two original data nodes to the new data node. + +### Step 1: Update the Retention Policy + +[Update](/enterprise_influxdb/v1.9/query_language/manage-database/#modify-retention-policies-with-alter-retention-policy) +every retention policy to have a replication factor of three. +This step ensures that the system automatically distributes all newly-created +shards across the three data nodes in the cluster. + +The following query increases the replication factor to three. +Run the query on any data node for each retention policy and database. +Here, we use InfluxDB's [CLI](/enterprise_influxdb/v1.9/tools/use-influx/) to execute the query: + +``` +> ALTER RETENTION POLICY "" ON "" REPLICATION 3 +> +``` + +A successful `ALTER RETENTION POLICY` query returns no results. +Use the +[`SHOW RETENTION POLICIES` query](/enterprise_influxdb/v1.9/query_language/explore-schema/#show-retention-policies) +to verify the new replication factor. + +Example: +``` +> SHOW RETENTION POLICIES ON "telegraf" + +name duration shardGroupDuration replicaN default +---- -------- ------------------ -------- ------- +autogen 0s 1h0m0s 3 #👍 true +``` + +### Step 2: Truncate Hot Shards + +Hot shards are shards that are currently receiving writes. +Performing any action on a hot shard can lead to data inconsistency within the +cluster which requires manual intervention from the user. + +To prevent data inconsistency, truncate hot shards before copying any shards +to the new data node. +The command below creates a new hot shard which is automatically distributed +across the three data nodes in the cluster, and the system writes all new points +to that shard. +All previous writes are now stored in cold shards. + +``` +influxd-ctl truncate-shards +``` + +The expected ouput of this command is: + +``` +Truncated shards. +``` + +Once you truncate the shards, you can work on distributing the cold shards +without the threat of data inconsistency in the cluster. +Any hot or new shards are now automatically distributed across the cluster and +require no further intervention. + +### Step 3: Identify Cold Shards + +In this step, you identify the cold shards that you will copy to the new data node. + +The following command lists every shard in your cluster: + +``` +influxd-ctl show-shards +``` + +The expected output is similar to the items in the codeblock below: + +``` +Shards +========== +ID Database Retention Policy Desired Replicas [...] End Owners +21 telegraf autogen 3 [...] 2017-01-26T18:00:00Z [{4 enterprise-data-01:8088} {5 enterprise-data-02:8088}] +22 telegraf autogen 3 [...] 2017-01-26T18:05:36.418734949Z* [{4 enterprise-data-01:8088} {5 enterprise-data-02:8088}] +24 telegraf autogen 3 [...] 2017-01-26T19:00:00Z [{4 enterprise-data-01:8088} {5 enterprise-data-02:8088} {6 enterprise-data-03:8088}] +``` + +The sample output includes three shards. +The first two shards are cold shards. +The timestamp in the `End` column occurs in the past (assume that the current +time is just after `2017-01-26T18:05:36.418734949Z`), and the shards' owners +are the two original data nodes: `enterprise-data-01:8088` and +`enterprise-data-02:8088`. +The second shard is the truncated shard; truncated shards have an asterix (`*`) +on the timestamp in the `End` column. + +The third shard is the newly-created hot shard; the timestamp in the `End` +column is in the future (again, assume that the current time is just after +`2017-01-26T18:05:36.418734949Z`), and the shard's owners include all three +data nodes: `enterprise-data-01:8088`, `enterprise-data-02:8088`, and +`enterprise-data-03:8088`. +That hot shard and any subsequent shards require no attention during +the rebalance process. + +Identify the cold shards that you'd like to copy from one of the original two +data nodes to the new data node. +Take note of the cold shard's `ID` (for example: `22`) and the TCP address of +one of its owners in the `Owners` column (for example: +`enterprise-data-01:8088`). + +### Step 4: Copy Cold Shards + +Next, copy the relevant cold shards to the new data node with the syntax below. +Repeat this command for every cold shard that you'd like to move to the +new data node. + +``` +influxd-ctl copy-shard +``` + +Where `source_TCP_address` is the address that you noted in step 3, +`destination_TCP_address` is the TCP address of the new data node, and `shard_ID` +is the ID of the shard that you noted in step 3. + +The expected output of the command is: +``` +Copied shard from to +``` + +### Step 5: Confirm the Rebalance + +Confirm that the TCP address of the new data node appears in the `Owners` column +for every copied shard: + +``` +influxd-ctl show-shards +``` + +The expected output shows that the copied shard now has three owners: +``` +Shards +========== +ID Database Retention Policy Desired Replicas [...] End Owners +22 telegraf autogen 3 [...] 2017-01-26T18:05:36.418734949Z* [{4 enterprise-data-01:8088} {5 enterprise-data-02:8088} {6 enterprise-data-03:8088}] +``` + +In addition, verify that the copied shards appear in the new data node's shard +directory and match the shards in the source data node's shard directory. +Shards are located in +`/var/lib/influxdb/data///`. + +Here's an example of the correct output for shard `22`: +``` +# On the source data node (enterprise-data-01) + +~# ls /var/lib/influxdb/data/telegraf/autogen/22 +000000001-000000001.tsm # 👍 + +# On the new data node (enterprise-data-03) + +~# ls /var/lib/influxdb/data/telegraf/autogen/22 +000000001-000000001.tsm # 👍 +``` + +That's it. +You've successfully rebalanced your cluster and increased data availability for +queries and query throughput. diff --git a/content/enterprise_influxdb/v1.9/guides/replacing-nodes.md b/content/enterprise_influxdb/v1.9/guides/replacing-nodes.md new file mode 100644 index 000000000..b15aa4366 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/guides/replacing-nodes.md @@ -0,0 +1,426 @@ +--- +title: Replace InfluxDB Enterprise cluster meta nodes and data nodes +description: Replace meta and data nodes in an InfluxDB Enterprise cluster. +menu: + enterprise_influxdb_1_9: + name: Replace cluster nodes + weight: 10 + parent: Guides +--- + +## Introduction + +Nodes in an InfluxDB Enterprise cluster may need to be replaced at some point due to hardware needs, hardware issues, or something else entirely. +This guide outlines processes for replacing both meta nodes and data nodes in an InfluxDB Enterprise cluster. + +## Concepts +Meta nodes manage and monitor both the uptime of nodes in the cluster as well as distribution of [shards](/enterprise_influxdb/v1.9/concepts/glossary/#shard) among nodes in the cluster. +They hold information about which data nodes own which shards; information on which the +[anti-entropy](/enterprise_influxdb/v1.9/administration/anti-entropy/) (AE) process depends. + +Data nodes hold raw time-series data and metadata. Data shards are both distributed and replicated across data nodes in the cluster. The AE process runs on data nodes and references the shard information stored in the meta nodes to ensure each data node has the shards they need. + +`influxd-ctl` is a CLI included in each meta node and is used to manage your InfluxDB Enterprise cluster. + +## Scenarios + +### Replace nodes in clusters with security enabled +Many InfluxDB Enterprise clusters are configured with security enabled, forcing secure TLS encryption between all nodes in the cluster. +Both `influxd-ctl` and `curl`, the command line tools used when replacing nodes, have options that facilitate the use of TLS. + +#### `influxd-ctl -bind-tls` +In order to manage your cluster over TLS, pass the `-bind-tls` flag with any `influxd-ctl` commmand. + +> If using a self-signed certificate, pass the `-k` flag to skip certificate verification. + +```bash +# Pattern +influxd-ctl -bind-tls [-k] + +# Example +influxd-ctl -bind-tls remove-meta enterprise-meta-02:8091 +``` + +#### `curl -k` + +`curl` natively supports TLS/SSL connections, but if using a self-signed certificate, pass the `-k`/`--insecure` flag to allow for "insecure" SSL connections. + +> Self-signed certificates are considered "insecure" due to their lack of a valid chain of authority. However, data is still encrypted when using self-signed certificates. + +```bash +# Pattern +curl [-k, --insecure] + +# Example +curl -k https://localhost:8091/status +``` + +### Replace meta nodes in a functional cluster + +If all meta nodes in the cluster are fully functional, simply follow the steps for [replacing meta nodes](#replace-meta-nodes-in-an-influxdb-enterprise-cluster). + +### Replace an unresponsive meta node + +If replacing a meta node that is either unreachable or unrecoverable, you need to forcefully remove it from the meta cluster. Instructions for forcefully removing meta nodes are provided in the [step 2.2](#2-2-remove-the-non-leader-meta-node) of the [replacing meta nodes](#replace-meta-nodes-in-an-influxdb-enterprise-cluster) process. + +### Replace responsive and unresponsive data nodes in a cluster + +The process of replacing both responsive and unresponsive data nodes is the same. Simply follow the instructions for [replacing data nodes](#replace-data-nodes-in-an-influxdb-enterprise-cluster). + +### Reconnect a data node with a failed disk + +A disk drive failing is never a good thing, but it does happen, and when it does, +all shards on that node are lost. + +Often in this scenario, rather than replacing the entire host, you just need to replace the disk. +Host information remains the same, but once started again, the `influxd` process doesn't know +to communicate with the meta nodes so the AE process can't start the shard-sync process. + +To resolve this, log in to a meta node and use the [`influxd-ctl update-data`](/enterprise_influxdb/v1.9/tools/influxd-ctl/#update-data) command +to [update the failed data node to itself](#2-replace-the-old-data-node-with-the-new-data-node). + +```bash +# Pattern +influxd-ctl update-data + +# Example +influxd-ctl update-data enterprise-data-01:8088 enterprise-data-01:8088 +``` + +This will connect the `influxd` process running on the newly replaced disk to the cluster. +The AE process will detect the missing shards and begin to sync data from other +shards in the same shard group. + + +## Replace meta nodes in an InfluxDB Enterprise cluster + +[Meta nodes](/enterprise_influxdb/v1.9/concepts/clustering/#meta-nodes) together form a [Raft](https://raft.github.io/) cluster in which nodes elect a leader through consensus vote. +The leader oversees the management of the meta cluster, so it is important to replace non-leader nodes before the leader node. +The process for replacing meta nodes is as follows: + +1. [Identify the leader node](#1-identify-the-leader-node) +2. [Replace all non-leader nodes](#2-replace-all-non-leader-nodes) + 2.1. [Provision a new meta node](#2-1-provision-a-new-meta-node) + 2.2. [Remove the non-leader meta node](#2-2-remove-the-non-leader-meta-node) + 2.3. [Add the new meta node](#2-3-add-the-new-meta-node) + 2.4. [Confirm the meta node was added](#2-4-confirm-the-meta-node-was-added) + 2.5. [Remove and replace all other non-leader meta nodes](#2-5-remove-and-replace-all-other-non-leader-meta-nodes) +3. [Replace the leader node](#3-replace-the-leader-node) + 3.1. [Kill the meta process on the leader node](#3-1-kill-the-meta-process-on-the-leader-node) + 3.2. [Remove and replace the old leader node](#3-2-remove-and-replace-the-old-leader-node) + +### 1. Identify the leader node + +Log into any of your meta nodes and run the following: + +```bash +curl -s localhost:8091/status | jq +``` + +> Piping the command into `jq` is optional, but does make the JSON output easier to read. + +The output will include information about the current meta node, the leader of the meta cluster, and a list of "peers" in the meta cluster. + +```json +{ + "nodeType": "meta", + "leader": "enterprise-meta-01:8089", + "httpAddr": "enterprise-meta-01:8091", + "raftAddr": "enterprise-meta-01:8089", + "peers": [ + "enterprise-meta-01:8089", + "enterprise-meta-02:8089", + "enterprise-meta-03:8089" + ] +} +``` + +Identify the `leader` of the cluster. When replacing nodes in a cluster, non-leader nodes should be replaced _before_ the leader node. + +### 2. Replace all non-leader nodes + +#### 2.1. Provision a new meta node + +[Provision and start a new meta node](/enterprise_influxdb/v1.9/installation/meta_node_installation/), but **do not** add it to the cluster yet. +For this guide, the new meta node's hostname will be `enterprise-meta-04`. + +#### 2.2. Remove the non-leader meta node + +Now remove the non-leader node you are replacing by using the `influxd-ctl remove-meta` command and the TCP address of the meta node (ex. `enterprise-meta-02:8091`): + +```bash +# Pattern +influxd-ctl remove-meta + +# Example +influxd-ctl remove-meta enterprise-meta-02:8091 +``` + +> Only use `remove-meta` if you want to permanently remove a meta node from a cluster. + + + +> **For unresponsive or unrecoverable meta nodes:** + +>If the meta process is not running on the node you are trying to remove or the node is neither reachable nor recoverable, use the `-force` flag. +When forcefully removing a meta node, you must also pass the `-tcpAddr` flag with the TCP and HTTP bind addresses of the node you are removing. + +```bash +# Pattern +influxd-ctl remove-meta -force -tcpAddr + +# Example +influxd-ctl remove-meta -force -tcpAddr enterprise-meta-02:8089 enterprise-meta-02:8091 +``` + +#### 2.3. Add the new meta node + +Once the non-leader meta node has been removed, use `influxd-ctl add-meta` to replace it with the new meta node: + +```bash +# Pattern +influxd-ctl add-meta + +# Example +influxd-ctl add-meta enterprise-meta-04:8091 +``` + +You can also add a meta node remotely through another meta node: + +```bash +# Pattern +influxd-ctl -bind add-meta + +# Example +influxd-ctl -bind enterprise-meta-node-01:8091 add-meta enterprise-meta-node-04:8091 +``` + +>This command contacts the meta node running at `cluster-meta-node-01:8091` and adds a meta node to that meta node’s cluster. +The added meta node has the hostname `cluster-meta-node-04` and runs on port `8091`. + +#### 2.4. Confirm the meta node was added + +Confirm the new meta-node has been added by running: + +```bash +influxd-ctl show +``` + +The new meta node should appear in the output: + +```bash +Data Nodes +========== +ID TCP Address Version +4 enterprise-data-01:8088 1.8.x-c1.8.x +5 enterprise-data-02:8088 1.8.x-c1.8.x + +Meta Nodes +========== +TCP Address Version +enterprise-meta-01:8091 1.8.x-c1.8.x +enterprise-meta-03:8091 1.8.x-c1.8.x +enterprise-meta-04:8091 1.8.x-c1.8.x # <-- The newly added meta node +``` + +#### 2.5. Remove and replace all other non-leader meta nodes + +**If replacing only one meta node, no further action is required.** +If replacing others, repeat steps [2.1-2.4](#2-1-provision-a-new-meta-node) for all non-leader meta nodes one at a time. + +### 3. Replace the leader node + +As non-leader meta nodes are removed and replaced, the leader node oversees the replication of data to each of the new meta nodes. +Leave the leader up and running until at least two of the new meta nodes are up, running and healthy. + +#### 3.1 - Kill the meta process on the leader node + +Log into the leader meta node and kill the meta process. + +```bash +# List the running processes and get the +# PID of the 'influx-meta' process +ps aux + +# Kill the 'influx-meta' process +kill +``` + +Once killed, the meta cluster will elect a new leader using the [raft consensus algorithm](https://raft.github.io/). +Confirm the new leader by running: + +```bash +curl localhost:8091/status | jq +``` + +#### 3.2 - Remove and replace the old leader node + +Remove the old leader node and replace it by following steps [2.1-2.4](#2-1-provision-a-new-meta-node). +The minimum number of meta nodes you should have in your cluster is 3. + +## Replace data nodes in an InfluxDB Enterprise cluster + +[Data nodes](/enterprise_influxdb/v1.9/concepts/clustering/#data-nodes) house all raw time series data and metadata. +The process of replacing data nodes is as follows: + +1. [Provision a new data node](#1-provision-a-new-data-node) +2. [Replace the old data node with the new data node](#2-replace-the-old-data-node-with-the-new-data-node) +3. [Confirm the data node was added](#3-confirm-the-data-node-was-added) +4. [Check the copy-shard-status](#4-check-the-copy-shard-status) + +### 1. Provision a new data node + +[Provision and start a new data node](/enterprise_influxdb/v1.9/installation/data_node_installation/), but **do not** add it to your cluster yet. + +### 2. Replace the old data node with the new data node + +Log into any of your cluster's meta nodes and use `influxd-ctl update-data` to replace the old data node with the new data node: + +```bash +# Pattern +influxd-ctl update-data + +# Example +influxd-ctl update-data enterprise-data-01:8088 enterprise-data-03:8088 +``` + +### 3. Confirm the data node was added + +Confirm the new data node has been added by running: + +```bash +influxd-ctl show +``` + +The new data node should appear in the output: + +```bash +Data Nodes +========== +ID TCP Address Version +4 enterprise-data-03:8088 1.8.x-c1.8.x # <-- The newly added data node +5 enterprise-data-02:8088 1.8.x-c1.8.x + +Meta Nodes +========== +TCP Address Version +enterprise-meta-01:8091 1.8.x-c1.8.x +enterprise-meta-02:8091 1.8.x-c1.8.x +enterprise-meta-03:8091 1.8.x-c1.8.x +``` + +Inspect your cluster's shard distribution with `influxd-ctl show-shards`. +Shards will immediately reflect the new address of the node. + +```bash +influxd-ctl show-shards + +Shards +========== +ID Database Retention Policy Desired Replicas Shard Group Start End Expires Owners +3 telegraf autogen 2 2 2018-03-19T00:00:00Z 2018-03-26T00:00:00Z [{5 enterprise-data-02:8088} {4 enterprise-data-03:8088}] +1 _internal monitor 2 1 2018-03-22T00:00:00Z 2018-03-23T00:00:00Z 2018-03-30T00:00:00Z [{5 enterprise-data-02:8088}] +2 _internal monitor 2 1 2018-03-22T00:00:00Z 2018-03-23T00:00:00Z 2018-03-30T00:00:00Z [{4 enterprise-data-03:8088}] +4 _internal monitor 2 3 2018-03-23T00:00:00Z 2018-03-24T00:00:00Z 2018-03-01T00:00:00Z [{5 enterprise-data-02:8088}] +5 _internal monitor 2 3 2018-03-23T00:00:00Z 2018-03-24T00:00:00Z 2018-03-01T00:00:00Z [{4 enterprise-data-03:8088}] +6 foo autogen 2 4 2018-03-19T00:00:00Z 2018-03-26T00:00:00Z [{5 enterprise-data-02:8088} {4 enterprise-data-03:8088}] +``` + +Within the duration defined by [`anti-entropy.check-interval`](/enterprise_influxdb/v1.9/administration/config-data-nodes#check-interval-10m), +the AE service will begin copying shards from other shard owners to the new node. +The time it takes for copying to complete is determined by the number of shards copied and how much data is stored in each. + +### 4. Check the `copy-shard-status` + +Check on the status of the copy-shard process with: + +```bash +influxd-ctl copy-shard-status +``` + +The output will show all currently running copy-shard processes. + +```bash +Source Dest Database Policy ShardID TotalSize CurrentSize StartedAt +enterprise-data-02:8088 enterprise-data-03:8088 telegraf autogen 3 119624324 119624324 2018-04-17 23:45:09.470696179 +0000 UTC +``` + +> **Important:** If replacing other data nodes in the cluster, make sure shards are completely copied from nodes in the same shard group before replacing the other nodes. +View the [Anti-entropy](/enterprise_influxdb/v1.9/administration/anti-entropy/#concepts) documentation for important information regarding anti-entropy and your database's replication factor. + +## Troubleshoot + +### Cluster commands result in timeout without error + +In some cases, commands used to add or remove nodes from your cluster +timeout, but don't return an error. + +```bash +add-data: operation timed out with error: +``` + +#### Check your InfluxDB user permissions + +In order to add or remove nodes to or from a cluster, your user must have `AddRemoveNode` permissions. +Attempting to manage cluster nodes without the appropriate permissions results +in a timeout with no accompanying error. + +To check user permissions, log in to one of your meta nodes and `curl` the `/user` API endpoint: + +```bash +curl localhost:8091/user +``` + +You can also check the permissions of a specific user by passing the username with the `name` parameter: + +```bash +# Pattern +curl localhost:8091/user?name= + +# Example +curl localhost:8091/user?name=bob +``` + +The JSON output will include user information and permissions: + +```json +"users": [ + { + "name": "bob", + "hash": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx", + "permissions": { + "": [ + "ViewAdmin", + "ViewChronograf", + "CreateDatabase", + "CreateUserAndRole", + "DropDatabase", + "DropData", + "ReadData", + "WriteData", + "ManageShard", + "ManageContinuousQuery", + "ManageQuery", + "ManageSubscription", + "Monitor" + ] + } + } +] +``` + +_In the output above, `bob` does not have the required `AddRemoveNode` permissions +and would not be able to add or remove nodes from the cluster._ + +#### Check the network connection between nodes + +Something may be interrupting the network connection between nodes. +To check, `ping` the server or node you're trying to add or remove. +If the ping is unsuccessful, something in the network is preventing communication. + +```bash +ping enterprise-data-03:8088 +``` + +_If pings are unsuccessful, be sure to ping from other meta nodes as well to determine +if the communication issues are unique to specific nodes._ diff --git a/content/enterprise_influxdb/v1.9/guides/write_data.md b/content/enterprise_influxdb/v1.9/guides/write_data.md new file mode 100644 index 000000000..474980ab7 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/guides/write_data.md @@ -0,0 +1,190 @@ +--- +title: Write data with the InfluxDB API +description: > + Use the command line interface (CLI) to write data into InfluxDB with the API. +menu: + enterprise_influxdb_1_9: + weight: 10 + parent: Guides +aliases: + - /enterprise_influxdb/v1.9/guides/writing_data/ +v2: /influxdb/v2.0/write-data/ +--- + +Write data into InfluxDB using the [command line interface](/enterprise_influxdb/v1.9/tools/use-influx/), [client libraries](/enterprise_influxdb/v1.9/clients/api/), and plugins for common data formats such as [Graphite](/enterprise_influxdb/v1.9/write_protocols/graphite/). + +> **Note**: The following examples use `curl`, a command line tool that transfers data using URLs. Learn the basics of `curl` with the [HTTP Scripting Guide](https://curl.haxx.se/docs/httpscripting.html). + +### Create a database using the InfluxDB API + +To create a database send a `POST` request to the `/query` endpoint and set the URL parameter `q` to `CREATE DATABASE `. +The example below sends a request to InfluxDB running on `localhost` and creates the `mydb` database: + +```bash +curl -i -XPOST http://localhost:8086/query --data-urlencode "q=CREATE DATABASE mydb" +``` + +### Write data using the InfluxDB API + +The InfluxDB API is the primary means of writing data into InfluxDB. + +- To **write to a database using the InfluxDB 1.8 API**, send `POST` requests to the `/write` endpoint. For example, to write a single point to the `mydb` database. +The data consists of the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) `cpu_load_short`, the [tag keys](/enterprise_influxdb/v1.9/concepts/glossary/#tag-key) `host` and `region` with the [tag values](/enterprise_influxdb/v1.9/concepts/glossary/#tag-value) `server01` and `us-west`, the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) `value` with a [field value](/enterprise_influxdb/v1.9/concepts/glossary/#field-value) of `0.64`, and the [timestamp](/enterprise_influxdb/v1.9/concepts/glossary/#timestamp) `1434055562000000000`. + +```bash +curl -i -XPOST 'http://localhost:8086/write?db=mydb' +--data-binary 'cpu_load_short,host=server01,region=us-west value=0.64 1434055562000000000' +``` + +- To **write to a database using the InfluxDB 2.0 API (compatible with InfluxDB 1.8+)**, send `POST` requests to the [`/api/v2/write` endpoint](/enterprise_influxdb/v1.9/tools/api/#api-v2-write-http-endpoint): + +```bash +curl -i -XPOST 'http://localhost:8086/api/v2/write?bucket=db/rp&precision=ns' \ + --header 'Authorization: Token username:password' \ + --data-raw 'cpu_load_short,host=server01,region=us-west value=0.64 1434055562000000000' +``` + +When writing points, you must specify an existing database in the `db` query parameter. +Points will be written to `db`'s default retention policy if you do not supply a retention policy via the `rp` query parameter. +See the [InfluxDB API Reference](/enterprise_influxdb/v1.9/tools/api/#write-http-endpoint) documentation for a complete list of the available query parameters. + +The body of the POST or [InfluxDB line protocol](/enterprise_influxdb/v1.9/concepts/glossary/#influxdb-line-protocol) contains the time series data that you want to store. Data includes: + +- **Measurement (required)** +- **Tags**: Strictly speaking, tags are optional but most series include tags to differentiate data sources and to make querying both easy and efficient. +Both tag keys and tag values are strings. +- **Fields (required)**: Field keys are required and are always strings, and, [by default](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types), field values are floats. +- **Timestamp**: Supplied at the end of the line in Unix time in nanoseconds since January 1, 1970 UTC - is optional. If you do not specify a timestamp, InfluxDB uses the server's local nanosecond timestamp in Unix epoch. +Time in InfluxDB is in UTC format by default. + +> **Note:** Avoid using the following reserved keys: `_field`, `_measurement`, and `time`. If reserved keys are included as a tag or field key, the associated point is discarded. + +### Configure gzip compression + +InfluxDB supports gzip compression. To reduce network traffic, consider the following options: + +* To accept compressed data from InfluxDB, add the `Accept-Encoding: gzip` header to InfluxDB API requests. + +* To compress data before sending it to InfluxDB, add the `Content-Encoding: gzip` header to InfluxDB API requests. + +For details about enabling gzip for client libraries, see your client library documentation. + +#### Enable gzip compression in the Telegraf InfluxDB output plugin + +* In the Telegraf configuration file (telegraf.conf), under [[outputs.influxdb]], change + `content_encoding = "identity"` (default) to `content_encoding = "gzip"` + +>**Note** +Writes to InfluxDB 2.x [[outputs.influxdb_v2]] are configured to compress content in gzip format by default. + +### Writing multiple points + +Post multiple points to multiple series at the same time by separating each point with a new line. +Batching points in this manner results in much higher performance. + +The following example writes three points to the database `mydb`. +The first point belongs to the series with the measurement `cpu_load_short` and tag set `host=server02` and has the server's local timestamp. +The second point belongs to the series with the measurement `cpu_load_short` and tag set `host=server02,region=us-west` and has the specified timestamp `1422568543702900257`. +The third point has the same specified timestamp as the second point, but it is written to the series with the measurement `cpu_load_short` and tag set `direction=in,host=server01,region=us-west`. + +```bash +curl -i -XPOST 'http://localhost:8086/write?db=mydb' --data-binary 'cpu_load_short,host=server02 value=0.67 +cpu_load_short,host=server02,region=us-west value=0.55 1422568543702900257 +cpu_load_short,direction=in,host=server01,region=us-west value=2.0 1422568543702900257' +``` + +### Writing points from a file + +Write points from a file by passing `@filename` to `curl`. +The data in the file should follow the [InfluxDB line protocol syntax](/enterprise_influxdb/v1.9/write_protocols/write_syntax/). + +Example of a properly-formatted file (`cpu_data.txt`): + +```txt +cpu_load_short,host=server02 value=0.67 +cpu_load_short,host=server02,region=us-west value=0.55 1422568543702900257 +cpu_load_short,direction=in,host=server01,region=us-west value=2.0 1422568543702900257 +``` + +Write the data in `cpu_data.txt` to the `mydb` database with: + +```bash +curl -i -XPOST 'http://localhost:8086/write?db=mydb' --data-binary @cpu_data.txt +``` + +> **Note:** If your data file has more than 5,000 points, it may be necessary to split that file into several files in order to write your data in batches to InfluxDB. +By default, the HTTP request times out after five seconds. +InfluxDB will still attempt to write the points after that time out but there will be no confirmation that they were successfully written. + +### Schemaless Design + +InfluxDB is a schemaless database. +You can add new measurements, tags, and fields at any time. +Note that if you attempt to write data with a different type than previously used (for example, writing a string to a field that previously accepted integers), InfluxDB will reject those data. + +### A note on REST + +InfluxDB uses HTTP solely as a convenient and widely supported data transfer protocol. + +Modern web APIs have settled on REST because it addresses a common need. +As the number of endpoints grows the need for an organizing system becomes pressing. +REST is the industry agreed style for organizing large numbers of endpoints. +This consistency is good for those developing and consuming the API: everyone involved knows what to expect. + +REST, however, is a convention. +InfluxDB makes do with three API endpoints. +This simple, easy to understand system uses HTTP as a transfer method for [InfluxQL](/enterprise_influxdb/v1.9/query_language/spec/). +The InfluxDB API makes no attempt to be RESTful. + +### HTTP response summary + +* 2xx: If your write request received `HTTP 204 No Content`, it was a success! +* 4xx: InfluxDB could not understand the request. +* 5xx: The system is overloaded or significantly impaired. + +#### Examples + +##### Writing a float to a field that previously accepted booleans + +```bash +curl -i -XPOST 'http://localhost:8086/write?db=hamlet' --data-binary 'tobeornottobe booleanonly=true' + +curl -i -XPOST 'http://localhost:8086/write?db=hamlet' --data-binary 'tobeornottobe booleanonly=5' +``` + +returns: + +```bash +HTTP/1.1 400 Bad Request +Content-Type: application/json +Request-Id: [...] +X-Influxdb-Version: 1.4.x +Date: Wed, 01 Mar 2017 19:38:01 GMT +Content-Length: 150 + +{"error":"field type conflict: input field \"booleanonly\" on measurement \"tobeornottobe\" is type float, already exists as type boolean dropped=1"} +``` + +##### Writing a point to a database that doesn't exist + +```bash +curl -i -XPOST 'http://localhost:8086/write?db=atlantis' --data-binary 'liters value=10' +``` + +returns: + +```bash +HTTP/1.1 404 Not Found +Content-Type: application/json +Request-Id: [...] +X-Influxdb-Version: 1.4.x +Date: Wed, 01 Mar 2017 19:38:35 GMT +Content-Length: 45 + +{"error":"database not found: \"atlantis\""} +``` + +### Next steps + +Now that you know how to write data with the InfluxDB API, discover how to query them with the [Querying data](/enterprise_influxdb/v1.9/guides/querying_data/) guide! +For more information about writing data with the InfluxDB API, please see the [InfluxDB API reference](/enterprise_influxdb/v1.9/tools/api/#write-http-endpoint). diff --git a/content/enterprise_influxdb/v1.9/introduction/_index.md b/content/enterprise_influxdb/v1.9/introduction/_index.md new file mode 100644 index 000000000..566364853 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/introduction/_index.md @@ -0,0 +1,14 @@ +--- +title: Introducing InfluxDB Enterprise +description: Tasks required to get up and running with InfluxDB Enterprise. +aliases: + - /enterprise/v1.8/introduction/ +weight: 2 + +menu: + enterprise_influxdb_1_9: + name: Introduction + +--- + +{{< children >}} diff --git a/content/enterprise_influxdb/v1.9/introduction/download.md b/content/enterprise_influxdb/v1.9/introduction/download.md new file mode 100644 index 000000000..e95d38028 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/introduction/download.md @@ -0,0 +1,22 @@ +--- +title: InfluxDB Enterprise downloads +description: Download InfluxDB Enterprise. +aliases: + - /enterprise/v1.8/introduction/download/ + +menu: + enterprise_influxdb_1_9: + name: Downloads + weight: 101 + parent: Introduction +--- + +You must have a valid license to run InfluxDB Enterprise. +You may obtain a 14-day demo license via the [InfluxDB Enterprise portal](https://portal.influxdata.com/users/new). + +If you have purchased a license or already obtained a demo license, +log in to the [InfluxDB Enterprise portal](https://portal.influxdata.com/users/sign_in) +to get your license key and download URLs. + +See the [installation documentation](/enterprise_influxdb/v1.9/install-and-deploy/) +for more information about getting started. diff --git a/content/enterprise_influxdb/v1.9/introduction/getting-started.md b/content/enterprise_influxdb/v1.9/introduction/getting-started.md new file mode 100644 index 000000000..3af569497 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/introduction/getting-started.md @@ -0,0 +1,22 @@ +--- +title: Getting started with InfluxDB Enterprise +description: Set up your cluster as a data source in Chronograf. +aliases: + - /enterprise_influxdb/v1.9/introduction/getting_started/ + - /enterprise/v1.8/introduction/getting_started/ +menu: + enterprise_influxdb_1_9: + name: Getting started + weight: 104 + parent: Introduction +--- + +After you successfully [install and set up](/enterprise_influxdb/v1.9/install-and-deploy/installation/) InfluxDB Enterprise, learn how to [monitor your InfluxDB Enterprise clusters](/{{< latest "chronograf" >}}/guides/monitoring-influxenterprise-clusters) with Chronograf, InfluxDB, and Telegraf. + +### Where to from here? + +- [Review key concepts](/enterprise_influxdb/v1.9/concepts/) and check out the [Enterprise features](/enterprise_influxdb/v1.9/features/). +- Learn how to [administer your cluster](/enterprise_influxdb/v1.9/administration/), including how to backup and restore clusters, configure clusters, log and trace clusters, manage security, and manage subscriptions. +- Find [Enterprise guides](/enterprise_influxdb/v1.9/guides/) on a variety of topics, such as how to downsample and retain data, rebalance InfluxDB Enterprise clusters, use fine-grained authorization, and more! +- Explore the [InfluxQL](/enterprise_influxdb/v1.9/query_language/) and [Flux](/enterprise_influxdb/v1.9/flux/) languages. +- Learn about [InfluxDB line protocol](/enterprise_influxdb/v1.9/write_protocols/) and other [supported protocols](/enterprise_influxdb/v1.9/supported_protocols/). diff --git a/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/_index.md b/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/_index.md new file mode 100644 index 000000000..6aec41dd3 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/_index.md @@ -0,0 +1,29 @@ +--- +title: Install and deploy InfluxDB Enterprise +description: Install InfluxDB Enterprise to on-premise or cloud providers, including Google Cloud Platform, Amazon Web Services, and Azure. +aliases: +- /enterprise_influxdb/v1.9/install-and-deploy/deploying/ +- /enterprise_influxdb/v1.9/install-and-deploy/ +menu: + enterprise_influxdb_1_9: + name: Install and deploy + weight: 30 + parent: Introduction +--- + +Install or deploy your InfluxDB Enterprise cluster in the environment of your choice: + +- Your own environment +- Your cloud provider + +## Your own environment + +Learn how to [install a cluster in your own environment](/enterprise_influxdb/v1.9/install-and-deploy/installation/). + +## Your cloud provider + +Learn how to deploy a cluster on the cloud provider of your choice: + + - [GCP](/enterprise_influxdb/v1.9/install-and-deploy/deploying/google-cloud-platform/) + - [AWS](/enterprise_influxdb/v1.9/install-and-deploy/deploying/aws/) + - [Azure](/enterprise_influxdb/v1.9/install-and-deploy/deploying/azure/) diff --git a/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/deploying/_index.md b/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/deploying/_index.md new file mode 100644 index 000000000..eee062c65 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/deploying/_index.md @@ -0,0 +1,20 @@ +--- +title: Deploy InfluxDB Enterprise clusters +description: > + Install InfluxDB Enterprise to a cloud provider of your choice, including Google Cloud Platform, Amazon Web Services, and Azure. +aliases: + - /enterprise_influxdb/v1.9/other-options/ + - /enterprise_influxdb/v1.9/install-and-deploy/deploying/index/ +menu: + enterprise_influxdb_1_9: + name: Deploy in cloud + identifier: deploy-in-cloud-enterprise + weight: 30 + parent: Install and deploy +--- + +Deploy InfluxDB Enterprise clusters on the cloud provider of your choice. + +> **Note:** To install in your own environment, see [Install an InfluxDB Enterprise cluster in your own environment](/enterprise_influxdb/v1.9/install-and-deploy/installation/). + +{{< children hlevel="h2" >}} diff --git a/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/deploying/aws/_index.md b/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/deploying/aws/_index.md new file mode 100644 index 000000000..3031ff526 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/deploying/aws/_index.md @@ -0,0 +1,19 @@ +--- +title: Deploy InfluxDB Enterprise clusters on Amazon Web Services +description: Deploy InfluxDB Enterprise clusters on Amazon Web Services (AWS). +aliases: + - /enterprise_influxdb/v1.9/other-options/ + - /enterprise_influxdb/v1.9/install-and-deploy/aws/ + - /enterprise_influxdb/v1.9/install-and-deploy/deploying/aws/ +menu: + enterprise_influxdb_1_9: + name: AWS + identifier: deploy-on-aws + weight: 30 + parent: deploy-in-cloud-enterprise +--- + +The following articles detail how to deploy InfluxDB clusters in AWS: + +- [Deploy an InfluxDB Enterprise cluster on Amazon Web Services](/enterprise_influxdb/v1.9/install-and-deploy/deploying/aws/setting-up-template) +- [AWS configuration options](/enterprise_influxdb/v1.9/install-and-deploy/deploying/aws/config-options) diff --git a/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/deploying/aws/config-options.md b/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/deploying/aws/config-options.md new file mode 100644 index 000000000..088faadce --- /dev/null +++ b/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/deploying/aws/config-options.md @@ -0,0 +1,32 @@ +--- +title: AWS configuration options +description: > + Configuration options when deploying InfluxDB Enterprise on Amazon Web Services (AWS). +aliases: + - /enterprise_influxdb/v1.9/install-and-deploy/aws/config-options/ + - /enterprise_influxdb/v1.9/install-and-deploy/aws/config-options/ + - /enterprise_influxdb/v1.9/install-and-deploy/deploying/aws/config-options/ +menu: + enterprise_influxdb_1_9: + name: AWS configuration options + weight: 30 + parent: deploy-on-aws +--- +When deploying InfluxDB Enterprise on AWS using the template described in [Deploy an InfluxDB Enterprise cluster on Amazon Web Services](/enterprise_influxdb/v1.9/install-and-deploy/aws/setting-up-template), the following configuration options are available: + +- **VPC ID**: The VPC ID of your existing Virtual Private Cloud (VPC). +- **Subnets**: A list of SubnetIds in your Virtual Private Cloud (VPC) where nodes will be created. The subnets must be in the same order as the availability zones they reside in. For a list of which availability zones correspond to which subnets, see the [Subnets section of your VPC dashboard](https://console.aws.amazon.com/vpc/home?region=us-east-1#subnets:sort=SubnetId). +- **Availability Zones**: Availability zones to correspond with your subnets above. The availability zones must be in the same order as their related subnets. For a list of which availability zones correspond to which subnets, see the [Subnets section of your VPC dashboard](https://console.aws.amazon.com/vpc/home?region=us-east-1#subnets:sort=SubnetId). +- **SSH Key Name**: An existing key pair to enable SSH access for the instances. For details on how to create a key pair, see [Creating a Key Pair Using Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#having-ec2-create-your-key-pair). +- **InfluxDB ingress CIDR**: The IP address range that can be used to connect to the InfluxDB API endpoint. To allow all traffic, enter 0.0.0.0/0. +- **SSH Access CIDR**: The IP address range that can be used to SSH into the EC2 instances. To allow all traffic, enter 0.0.0.0/0. +- **InfluxDB Enterprise License Key**: Your InfluxDB Enterprise license key. Applies only to BYOL. +- **InfluxDB Administrator Username**: Your InfluxDB administrator username. Applies only to BYOL. +- **InfluxDB Administrator Password**: Your InfluxDB administrator password. Applies only to BYOL. +- **InfluxDB Enterprise Version**: The version of InfluxDB. Defaults to current version. +- **Telegraf Version**: The version of Telegraf. Defaults to current version. +- **InfluxDB Data Node Disk Size**: The size in GB of the EBS io1 volume each data node. Defaults to 250. +- **InfluxDB Data Node Disk IOPS**: The IOPS of the EBS io1 volume on each data node. Defaults to 1000. +- **DataNodeInstanceType**: The instance type of the data node. Defaults to m5.large. +- **MetaNodeInstanceType**: The instance type of the meta node. Defaults to t3.small. +- **MonitorInstanceType**: The instance type of the monitor node. Defaults to t3.large. diff --git a/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/deploying/aws/setting-up-template.md b/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/deploying/aws/setting-up-template.md new file mode 100644 index 000000000..1bc1a49f5 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/deploying/aws/setting-up-template.md @@ -0,0 +1,58 @@ +--- +title: Deploy an InfluxDB Enterprise cluster on Amazon Web Services +description: Deploy an InfluxDB Enterprise cluster on Amazon Web Services (AWS). +aliases: + - /enterprise_influxdb/v1.9/install-and-deploy/aws/setting-up-template/ + - /enterprise_influxdb/v1.9/install-and-deploy/deploying/aws/setting-up-template/ +menu: + enterprise_influxdb_1_9: + name: Deploy on Amazon Web Services + weight: 20 + parent: deploy-on-aws +--- + +Follow these steps to deploy an InfluxDB Enterprise cluster on AWS. + +## Step 1: Specify template + +After you complete the marketplace flow, you'll be directed to the Cloud Formation Template. + +1. In the Prepare template section, select **Template is ready**. +2. In the Specify template section, the **Amazon S3 URL** field is automatically populated with either the BYOL or integrated billing template, depending on the option you selected in the marketplace. +3. Click **Next**. + +## Step 2: Specify stack details + +1. In the Stack name section, enter a name for your stack. +2. Complete the Network Configuration section: + - **VPC ID**: Click the dropdown menu to fill in your VPC. + - **Subnets**: Select three subnets. + - **Availability Zones**: Select three availability zones to correspond with your subnets above. The availability zones must be in the same order as their related subnets. For a list of which availability zones correspond to which subnets, see the [Subnets section of your VPC dashboard](https://console.aws.amazon.com/vpc/home?region=us-east-1#subnets:sort=SubnetId). + - **SSH Key Name**: Select an existing key pair to enable SSH access for the instances. For details on how to create a key pair, see [Creating a Key Pair Using Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#having-ec2-create-your-key-pair). + - **InfluxDB ingress CIDR**: Enter the IP address range that can be used to connect to the InfluxDB API endpoint. To allow all traffic, enter 0.0.0.0/0. + - **SSH Access CIDR**: Enter the IP address range that can be used to SSH into the EC2 instances. To allow all traffic, enter 0.0.0.0/0. +3. Complete the **InfluxDB Configuration** section: + - **InfluxDB Enterprise License Key**: Applies only to BYOL. Enter your InfluxDB Enterprise license key. + - **InfluxDB Administrator Username**: Applies only to BYOL. Enter your InfluxDB administrator username. + - **InfluxDB Administrator Password**: Applies only to BYOL. Enter your InfluxDB administrator password. + - **InfluxDB Enterprise Version**: Defaults to current version. + - **Telegraf Version**: Defaults to current version. + - **InfluxDB Data Node Disk Size**: The size in GB of the EBS io1 volume each data node. Defaults to 250. + - **InfluxDB Data Node Disk IOPS**: The IOPS of the EBS io1 volume on each data node. Defaults to 1000. +4. Review the **Other Parameters** section and modify if needed. The fields in this section are all automatically populated and shouldn't require changes. + - **DataNodeInstanceType**: Defaults to m5.large. + - **MetaNodeInstanceType**: Defaults to t3.small. + - **MonitorInstanceType**: Defaults to t3.large. +5. Click **Next**. + +## Step 3: Configure stack options + +1. In the **Tags** section, enter any key-value pairs you want to apply to resources in the stack. +2. Review the **Permissions** and **Advanced options** sections. In most cases, there's no need to modify anything in these sections. +3. Click **Next**. + +## Step 4: Review + +1. Review the configuration options for all of the above sections. +2. In the **Capabilities** section, check the box acknowledging that AWS CloudFormation might create IAM resources. +3. Click **Create stack**. diff --git a/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/deploying/azure.md b/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/deploying/azure.md new file mode 100644 index 000000000..f76d9a489 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/deploying/azure.md @@ -0,0 +1,76 @@ +--- +title: Deploy an InfluxDB Enterprise cluster on Azure Cloud Platform +description: > + Deploy an InfluxDB Enterprise cluster on Microsoft Azure cloud computing service. +aliases: + - /enterprise_influxdb/v1.9/install-and-deploy/azure/ + - /enterprise_influxdb/v1.9/install-and-deploy/deploying/azure/ +menu: + enterprise_influxdb_1_9: + name: Azure + weight: 20 + parent: deploy-in-cloud-enterprise +--- + +For deploying InfluxDB Enterprise clusters on Microsoft Azure cloud computing service, InfluxData provides an [InfluxDB Enterprise application](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/influxdata.influxdb-enterprise-cluster) on the [Azure Marketplace](https://azuremarketplace.microsoft.com/) that makes the installation and setup process easy and straightforward. Clusters are deployed through an Azure Marketplace subscription and are ready for production. Billing occurs through your Azure subscription. + +> Please submit issues and feature requests for the Azure Marketplace deployment [through the related GitHub repository](https://github.com/influxdata/azure-resource-manager-influxdb-enterprise/issues/new) (requires a GitHub account) or by contacting [InfluxData Support](mailto:support@influxdata.com). + +## Prerequisites + +This guide requires the following: + +- Microsoft Azure account with access to the [Azure Marketplace](https://azuremarketplace.microsoft.com/). +- SSH access to cluster instances. + +To deploy InfluxDB Enterprise clusters on platforms other than Azure, see [Deploy InfluxDB Enterprise](/enterprise_influxdb/v1.9/install-and-deploy/). + +## Deploy a cluster + +1. Log in to your Azure Cloud Platform account and navigate to [InfluxData's InfluxDB Enterprise (Official Version) application](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/influxdata.influxdb-enterprise-cluster) on Azure Marketplace. + +2. Click **Get It Now**, read and agree to the terms of use, and then click **Continue**. Once in the Azure Portal, click **Create**. + +3. Select the subscription to use for your InfluxDB Enterprise cluster. Then select a resource group and region where the cluster resources will be deployed. + + > **Tip:** If you do not know which resource group to use, we recommend creating a new one for the InfluxDB Enterprise cluster. + +4. In the Instance Details section, set the OS username and SSH authentication type you will use to access the cluster VMs. For password authentication, enter a username and password. For SSH public key authentication, copy an SSH public key. The cluster VMs are built from an Ubuntu base image. + +5. Click **Next: Cluster Configuration**, and then enter details including the InfluxDB admin username and password, the number of meta and data nodes, and the VM size for both meta and data nodes. We recommend using the default VM sizes and increasing the data node VM size if you anticipate needing more resources for your workload. + + > **Note:** Make sure to save the InfluxDB admin credentials. They will be required to access InfluxDB. + +6. Click **Next: External Access & Chronograf**, and then do the following: + + - To create a separate instance to monitor the cluster and run [Chronograf](https://www.influxdata.com/time-series-platform/chronograf/), select **Yes**. Otherwise, select **No**. + + > **Note:** Adding a Chronograf instance will also configure that instance as an SSH bastion. All cluster instances will only be accessible through the Chronograf instance. + + - Select the appropriate access for the InfluxDB load balancer: **External** to allow external Internet access; otherwise, select **Internal**. + + {{% warn %}}The cluster uses HTTP by default. You must configure HTTPS after the cluster has been deployed.{{% /warn %}} + +7. Click **Next: Review + create** to validate your cluster configuration details. If validation passes, your InfluxDB Enterprise cluster is deployed. + + > **Note:** Some Azure accounts may have vCPU quotas limited to 10 vCPUs available in certain regions. Selecting VM sizes larger than the default can cause a validation error for exceeding the vCPU limit for the region. + +## Access InfluxDB + +Once the cluster is created, access the InfluxDB API at the IP address associated with the load balancer resource (`lb-influxdb`). If external access was configured during setup, the load balancer is publicly accessible. Otherwise, the load balancer is only accessible to the cluster's virtual network. + +Use the load balancer IP address and the InfluxDB admin credentials entered during the cluster creation to interact with InfluxDB Enterprise via the [`influx` CLI](/enterprise_influxdb/v1.9/tools/use-influx/) or use the InfluxDB's [query](/enterprise_influxdb/v1.9/guides/query_data/) and [write](/enterprise_influxdb/v1.9/guides/write_data/) HTTP APIs. + +## Access the cluster + +The InfluxDB Enterprise cluster's VMs are only reachable within the virtual network using the SSH credentails provided during setup. + +If a Chronograf instance has been added to the cluster, the Chronograf instance is publically accessible via SSH. The other VMs can then be reached from the Chronograf VM. + +## Testing + +Azure Resource Manager (ARM) templates used in the InfluxDB Enterprise offering on Azure Marketplace are [available for testing purposes](https://github.com/influxdata/azure-resource-manager-influxdb-enterprise). **Please note, these templates are under active development and not recommended for production.** + +### Next steps + +For an introduction to the InfluxDB database and the InfluxData Platform, see [Getting started with InfluxDB](/platform/introduction/getting-started). diff --git a/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/deploying/google-cloud-platform.md b/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/deploying/google-cloud-platform.md new file mode 100644 index 000000000..c7a7cdcaa --- /dev/null +++ b/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/deploying/google-cloud-platform.md @@ -0,0 +1,100 @@ +--- +title: Deploy an InfluxDB Enterprise cluster on Google Cloud Platform +description: > + Deploy an InfluxDB Enterprise cluster on Google Cloud Platform (GCP). +aliases: + - /enterprise_influxdb/v1.9/other-options/google-cloud/ + - /enterprise_influxdb/v1.9/install-and-deploy/google-cloud-platform/ + - /enterprise_influxdb/v1.9/install-and-deploy/deploying/google-cloud-platform/ +menu: + enterprise_influxdb_1_9: + name: GCP + weight: 30 + parent: deploy-in-cloud-enterprise +--- + +Complete the following steps to deploy an InfluxDB Enterprise cluster on Google Cloud Platform (GCP): + +1. [Verify prerequistes](#verify-prerequisites). +2. [Deploy a cluster](#deploy-a-cluster). +3. [Access the cluster](#access-the-cluster). + +After deploying your cluster, see [Getting started with InfluxDB](/platform/introduction/getting-started) for an introduction to InfluxDB database and the InfluxData platform. + +>**Note:** InfluxDB Enterprise on GCP is a self-managed product. For a fully managed InfluxDB experience, check out [InfluxDB Cloud](/influxdb/cloud/get-started/). + +## Verify prerequisites + +Before deploying an InfluxDB Enterprise cluster on GCP, verify you have the following prerequisites: + +- A [Google Cloud Platform (GCP)](https://cloud.google.com/) account with access to the [GCP Marketplace](https://cloud.google.com/marketplace/). +- Access to [GCP Cloud Shell](https://cloud.google.com/shell/) or the [`gcloud` SDK and command line tools](https://cloud.google.com/sdk/). + +## Deploy a cluster + +1. Log in to your Google Cloud Platform account and go to [InfluxDB Enterprise](https://console.cloud.google.com/marketplace/details/influxdata-public/influxdb-enterprise-vm). + +2. Click **Launch** to create or select a project to open up your cluster's configuration page. + +3. Adjust cluster fields as needed, including: + + - Deployment name: Enter a name for the InfluxDB Enterprise cluster. + - InfluxDB Enterprise admin username: Enter the username of your cluster administrator. + - Zone: Select a region for your cluster. + - Network: Select a network for your cluster. + - Subnetwork: Select a subnetwork for your cluster, if applicable. + + > **Note:** The cluster is only accessible within the network (or subnetwork, if specified) where it's deployed. + +4. Adjust data node fields as needed, including: + + - Data node instance count: Enter the number of data nodes to include in your cluster (we recommend starting with the default, 2). + - Data node machine type: Select the virtual machine type to use for data nodes (by default, 4 vCPUs). Use the down arrow to scroll through list. Notice the amount of memory available for the selected machine. To alter the number of cores and memory for your selected machine type, click the **Customize** link. For detail, see our recommended [hardware sizing guidelines](/enterprise_influxdb/v1.9/reference/hardware_sizing/). + - (Optional) By default, the data node disk type is SSD Persistent Disk and the disk size is 250 GB. To alter these defaults, click More and update if needed. + + > **Note:** Typically, fields in collapsed sections don't need to be altered. + +5. Adjust meta node fields as needed, including: + + - Meta node instance count: Enter the number of meta nodes to include in your cluster (we recommend using the default, 3). + - Meta node machine type: Select the virtual machine type to use for meta nodes (by default, 1 vCPUs). Use the down arrow to scroll through list. Notice the amount of memory available for the selected machine. To alter the number of cores and memory for your selected machine type, click the **Customize** link. + - By default, the meta node disk type is SSD Persistent Disk and the disk size is 10 GB. Alter these defaults if needed. + +6. (Optional) Adjust boot disk options fields is needed. By default the boot disk type is Standard Persistent disk and boot disk is 10 GB . + +7. Accept terms and conditions by selecting both check boxes, and then click **Deploy** to launch the InfluxDB Enterprise cluster. + +The cluster may take a few minutes to fully deploy. If the deployment does not complete or reports an error, read through the list of [common deployment errors](https://cloud.google.com/marketplace/docs/troubleshooting). + +> **Important:** Make sure you save the "Admin username", "Admin password", and "Connection internal IP" values displayed on the screen. They are required to access the cluster. + +## Access the cluster + +Access the cluster's IP address from the GCP network (or subnetwork) specified when you deployed the cluster. A cluster can only be reached from instances or services in the same GCP network or subnetwork. + +1. In the GCP Cloud Shell or `gcloud` CLI, create a new instance to access the InfluxDB Enterprise cluster. + + ``` + gcloud compute instances create influxdb-access --image-family ubuntu-1804-lts --image-project ubuntu-os-cloud + ``` + +2. SSH into the instance. + + ``` + gcloud compute ssh influxdb-access + ``` + +3. On the instance, install the `influx` command line tool via the InfluxDB open source package. + + ``` + wget https://dl.influxdata.com/influxdb/releases/influxdb_{{< latest-patch >}}_amd64.deb + sudo dpkg -i influxdb_{{< latest-patch >}}_amd64.deb + ``` + +4. Access the InfluxDB Enterprise cluster using the following command with "Admin username", "Admin password", and "Connection internal IP" values from the deployment screen substituted for ``. + + ``` + influx -username -password -host -execute "CREATE DATABASE test" + + influx -username -password -host -execute "SHOW DATABASES" + ``` diff --git a/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/installation/_index.md b/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/installation/_index.md new file mode 100644 index 000000000..7fa21d69a --- /dev/null +++ b/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/installation/_index.md @@ -0,0 +1,20 @@ +--- +title: Install an InfluxDB Enterprise cluster in your own environment +description: Install InfluxDB Enterprise in your own on-premise environment. +aliases: + - /enterprise_influxdb/v1.9/installation/ + - /enterprise_influxdb/v1.9/install-and-deploy/installation/ +menu: + enterprise_influxdb_1_9: + name: Install in your environment + weight: 10 + parent: Install and deploy +--- + +Complete the following steps to install an InfluxDB Enterprise cluster in your own environment: + +1. [Install InfluxDB Enterprise meta nodes](/enterprise_influxdb/v1.9/install-and-deploy/installation/meta_node_installation/) +2. [Install InfluxDB data nodes](/enterprise_influxdb/v1.9/install-and-deploy/installation/data_node_installation/) +3. [Install Chronograf](/enterprise_influxdb/v1.9/install-and-deploy/installation/chrono_install/) + +> **Note:** If you're looking for cloud infrastructure and services, check out how to deploy InfluxDB Enterprise (production-ready) on a cloud provider of your choice: [Azure](/enterprise_influxdb/v1.9/install-and-deploy/deploying/azure/), [GCP](/enterprise_influxdb/v1.9/install-and-deploy/deploying/google-cloud-platform/), or [AWS](/enterprise_influxdb/v1.9/install-and-deploy/deploying/aws/). diff --git a/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/installation/chrono_install.md b/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/installation/chrono_install.md new file mode 100644 index 000000000..1c9eb2e0d --- /dev/null +++ b/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/installation/chrono_install.md @@ -0,0 +1,17 @@ +--- +title: Install Chronograf +aliases: + - /enterprise_influxdb/v1.9/installation/chrono_install/ + - /enterprise_influxdb/v1.9/install-and-deploy/installation/chrono_install/ +menu: + enterprise_influxdb_1_9: + name: Install Chronograf + weight: 30 + parent: Install in your environment + identifier: chrono_install +--- + +Now that you've installed the meta nodes and data nodes, you are ready to install Chronograf +to provide you with a user interface to access the InfluxDB Enterprise instance. + +[Installation instruction for Chronograf](/{{< latest "chronograf" >}}/introduction/installation/) diff --git a/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/installation/data_node_installation.md b/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/installation/data_node_installation.md new file mode 100644 index 000000000..a5d4c04b0 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/installation/data_node_installation.md @@ -0,0 +1,288 @@ +--- +title: Install InfluxDB Enterprise data nodes +aliases: + - /enterprise_influxdb/v1.9/installation/data_node_installation/ + - /enterprise_influxdb/v1.9/install-and-deploy/installation/data_node_installation/ +menu: + enterprise_influxdb_1_9: + name: Install data nodes + weight: 20 + parent: Install in your environment +--- + +InfluxDB Enterprise offers highly scalable clusters on your infrastructure +and a management UI for working with clusters. +The next steps will get you up and running with the second essential component of +your InfluxDB Enterprise cluster: the data nodes. + +{{% warn %}} +If you have not set up your meta nodes, please visit +[Installing meta nodes](/enterprise_influxdb/v1.9/install-and-deploy/installation/meta_node_installation/). +Bad things can happen if you complete the following steps without meta nodes. +{{% /warn %}} + +# Data node setup description and requirements + +The installation process sets up two [data nodes](/enterprise_influxdb/v1.9/concepts/glossary#data-node) +and each data node runs on its own server. +You **must** have a minimum of two data nodes in a cluster. +InfluxDB Enterprise clusters require at least two data nodes for high availability and redundancy. +
        +Note: that there is no requirement for each data node to run on its own +server. However, best practices are to deploy each data node on a dedicated server. + +See the +[Clustering guide](/enterprise_influxdb/v1.9/concepts/clustering/#optimal-server-counts) +for more on cluster architecture. + +### Other requirements + +#### License key or file + +InfluxDB Enterprise requires a license key **OR** a license file to run. +Your license key is available at [InfluxPortal](https://portal.influxdata.com/licenses). +Contact support at the email we provided at signup to receive a license file. +License files are required only if the nodes in your cluster cannot reach +`portal.influxdata.com` on port `80` or `443`. + +#### Networking + +Data nodes communicate over ports `8088`, `8089`, and `8091`. + +For licensing purposes, data nodes must also be able to reach `portal.influxdata.com` +on port `80` or `443`. +If the data nodes cannot reach `portal.influxdata.com` on port `80` or `443`, +you'll need to set the `license-path` setting instead of the `license-key` +setting in the data node configuration file. + +#### Load balancer + +InfluxDB Enterprise does not function as a load balancer. +You will need to configure your own load balancer to send client traffic to the +data nodes on port `8086` (the default port for the [HTTP API](/enterprise_influxdb/v1.9/tools/api/)). + +#### User account + +The installation package creates user `influxdb` that is used to run the influxdb data service. `influxdb` user also owns certain files that are needed for the service to start successfully. In some cases, local policies may prevent the local user account from being created and the service fails to start. Contact your systems administrator for assistance with this requirement. + +# Data node setup +## Step 1: Add appropriate DNS entries for each of your servers + +Ensure that your servers' hostnames and IP addresses are added to your network's DNS environment. +The addition of DNS entries and IP assignment is usually site and policy specific; contact your DNS administrator for assistance as necessary. +Ultimately, use entries similar to the following (hostnames and domain IP addresses are representative). + +| Record Type | Hostname | IP | +|:------------|:-------------------------------------:|------------------:| +| A | ```enterprise-data-01.mydomain.com``` | `````` | +| A | ```enterprise-data-02.mydomain.com``` | `````` | + +> **Verification steps:** +> +Before proceeding with the installation, verify on each meta and data server that the other +servers are resolvable. Here is an example set of shell commands using `ping`: +> + ping -qc 1 enterprise-meta-01 + ping -qc 1 enterprise-meta-02 + ping -qc 1 enterprise-meta-03 + ping -qc 1 enterprise-data-01 + ping -qc 1 enterprise-data-02 + +We highly recommend that each server be able to resolve the IP from the hostname alone as shown here. +Resolve any connectivity issues before proceeding with the installation. +A healthy cluster requires that every meta node and data node in a cluster be able to communicate. + +## Step 2: Set up, configure, and start the data node services + +Perform the following steps on each data node. + +### I. Download and install the data service + +#### Ubuntu and Debian (64-bit) + +```bash +wget https://dl.influxdata.com/enterprise/releases/influxdb-data_{{< latest-patch >}}-c{{< latest-patch >}}_amd64.deb +sudo dpkg -i influxdb-data_{{< latest-patch >}}-c{{< latest-patch >}}_amd64.deb +``` + +#### RedHat and CentOS (64-bit) + +```bash +wget https://dl.influxdata.com/enterprise/releases/influxdb-data-{{< latest-patch >}}_c{{< latest-patch >}}.x86_64.rpm +sudo yum localinstall influxdb-data-{{< latest-patch >}}_c{{< latest-patch >}}.x86_64.rpm +``` + +#### Verify the authenticity of release download (recommended) + +For added security, follow these steps to verify the signature of your InfluxDB download with `gpg`. + +1. Download and import InfluxData's public key: + + ``` + curl -s https://repos.influxdata.com/influxdb.key | gpg --import + ``` + +2. Download the signature file for the release by adding `.asc` to the download URL. + For example: + + ``` + wget https://dl.influxdata.com/enterprise/releases/influxdb-data-{{< latest-patch >}}_c{{< latest-patch >}}.x86_64.rpm.asc + ``` + +3. Verify the signature with `gpg --verify`: + + ``` + gpg --verify influxdb-data-{{< latest-patch >}}-c{{< latest-patch >}}.x86_64.rpm.asc influxdb-data-{{< latest-patch >}}_c{{< latest-patch >}}.x86_64.rpm + ``` + + The output from this command should include the following: + + ``` + gpg: Good signature from "InfluxDB Packaging Service " [unknown] + ``` + +### II. Edit the data node configuration files + +First, in `/etc/influxdb/influxdb.conf`: + +* Uncomment `hostname` at the top of the file and set it to the full hostname of the data node. +* Uncomment `auth-enabled` in the `[http]` section and set it to `true`. +* Uncomment `meta-auth-enabled` in the `[meta]` section and set it to `true`. +* Uncomment `meta-internal-shared-secret` in the `[meta]` section and set it to a long pass phrase. The internal shared secret is used in JWT authentication for intra-node communication. This value must be same for all of your data nodes and match the `[meta] internal-shared-secret` value in the configuration files of your meta nodes. + +Second, in `/etc/influxdb/influxdb.conf`, set: + +`license-key` in the `[enterprise]` section to the license key you received on InfluxPortal **OR** `license-path` in the `[enterprise]` section to the local path to the JSON license file you received from InfluxData. + +{{% warn %}} +The `license-key` and `license-path` settings are mutually exclusive and one must remain set to the empty string. +{{% /warn %}} + +```toml +# Change this option to true to disable reporting. +# reporting-disabled = false +# bind-address = ":8088" +hostname="" + +[enterprise] + # license-key and license-path are mutually exclusive, use only one and leave the other blank + license-key = "" # Mutually exclusive with license-path + + # The path to a valid license file. license-key and license-path are mutually exclusive, + # use only one and leave the other blank. + license-path = "/path/to/readable/JSON.license.file" # Mutually exclusive with license-key + +[meta] + # Where the cluster metadata is stored + dir = "/var/lib/influxdb/meta" # data nodes do require a local meta directory +... + # This setting must have the same value as the meta nodes' meta.auth-enabled configuration. + meta-auth-enabled = true + +[...] + +[http] + # Determines whether HTTP endpoint is enabled. + # enabled = true + + # The bind address used by the HTTP service. + # bind-address = ":8086" + + # Determines whether HTTP authentication is enabled. + auth-enabled = true # Recommended, but not required + +[...] + + # The JWT auth shared secret to validate requests using JSON web tokens. + shared-secret = "long pass phrase used for signing tokens" +``` + +### III. Start the data service + +On sysvinit systems, enter: + +```bash +service influxdb start +``` + +On systemd systems, enter: + +```bash +sudo systemctl start influxdb +``` + +> **Verification steps:** +> +Check to see that the process is running by entering: +> + ps aux | grep -v grep | grep influxdb +> +You should see output similar to: +> + influxdb 2706 0.2 7.0 571008 35376 ? Sl 15:37 0:16 /usr/bin/influxd -config /etc/influxdb/influxdb.conf + + +If you do not see the expected output, the process is either not launching or is exiting prematurely. Check the [logs](/enterprise_influxdb/v1.9/administration/logs/) for error messages and verify the previous setup steps are complete. + +If you see the expected output, repeat for the remaining data nodes. +Once all data nodes have been installed, configured, and launched, move on to the next section to join the data nodes to the cluster. + +## Join the data nodes to the cluster + +{{% warn %}}You should join your data nodes to the cluster only when you are adding a brand new node, +either during the initial creation of your cluster or when growing the number of data nodes. +If you are replacing an existing data node with `influxd-ctl update-data`, skip the rest of this guide. +{{% /warn %}} + +On one and only one of the meta nodes that you set up in the +[previous document](/enterprise_influxdb/v1.9/introduction/meta_node_installation/), run: + +```bash +influxd-ctl add-data enterprise-data-01:8088 + +influxd-ctl add-data enterprise-data-02:8088 +``` + +The expected output is: + +```bash +Added data node y at enterprise-data-0x:8088 +``` + +Run the `add-data` command once and only once for each data node you are joining +to the cluster. + +> **Verification steps:** +> +Issue the following command on any meta node: +> + influxd-ctl show +> +The expected output is: +> + Data Nodes + ========== + ID TCP Address Version + 4 enterprise-data-01:8088 {{< latest-patch >}}-c{{< latest-patch >}} + 5 enterprise-data-02:8088 {{< latest-patch >}}-c{{< latest-patch >}} + +> + Meta Nodes + ========== + TCP Address Version + enterprise-meta-01:8091 {{< latest-patch >}}-c{{< latest-patch >}} + enterprise-meta-02:8091 {{< latest-patch >}}-c{{< latest-patch >}} + enterprise-meta-03:8091 {{< latest-patch >}}-c{{< latest-patch >}} + + +The output should include every data node that was added to the cluster. +The first data node added should have `ID=N`, where `N` is equal to one plus the number of meta nodes. +In a standard three meta node cluster, the first data node should have `ID=4` +Subsequently added data nodes should have monotonically increasing IDs. +If not, there may be artifacts of a previous cluster in the metastore. + +If you do not see your data nodes in the output, please retry adding them +to the cluster. + +Once your data nodes are part of your cluster move on to [the final step +to set up Chronograf](/enterprise_influxdb/v1.9/install-and-deploy/installation/chrono_install). diff --git a/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/installation/meta_node_installation.md b/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/installation/meta_node_installation.md new file mode 100644 index 000000000..299ac860b --- /dev/null +++ b/content/enterprise_influxdb/v1.9/introduction/install-and-deploy/installation/meta_node_installation.md @@ -0,0 +1,263 @@ +--- +title: Install InfluxDB Enterprise meta nodes +aliases: + - /enterprise_influxdb/v1.9/installation/meta_node_installation/ + - /enterprise_influxdb/v1.9/install-and-deploy/installation/meta_node_installation/ +menu: + enterprise_influxdb_1_9: + name: Install meta nodes + weight: 10 + parent: Install in your environment +--- + +InfluxDB Enterprise offers highly scalable clusters on your infrastructure +and a management UI ([via Chronograf](/{{< latest "chronograf" >}})) for working with clusters. +The installation process is designed for users looking to +deploy InfluxDB Enterprise in a production environment. +The following steps will get you up and running with the first essential component of +your InfluxDB Enterprise cluster: the meta nodes. + + +To install InfluxDB Enterprise meta nodes, do the following: + +1. Review [meta node setup and requirements](#meta-node-setup-description-and-requirements) +2. [Set up meta nodes](#set-up-meta-nodes): + 1. [Add DNS entries](#add-dns-entries) + 2. [Set up, configure, and start the meta services](#set-up-configure-and-start-the-meta-services) + 3. [Join meta nodes to the cluster](#join-meta-nodes-to-the-cluster) + +## Meta node setup and requirements + +The installation process sets up three [meta nodes](/enterprise_influxdb/v1.9/concepts/glossary/#meta-node), with each meta node running on its own server. + +InfluxDB Enterprise clusters require an *odd number* of *at least three* meta nodes +for high availability and redundancy. +We typically recommend three meta nodes. +If your servers have chronic communication or reliability issues, you can try adding nodes. + +> **Note**: Deploying multiple meta nodes on the same server is strongly discouraged +> since it creates a larger point of potential failure if that particular server is unresponsive. +> InfluxData recommends deploying meta nodes on relatively small footprint servers. + +See the +[Clustering in InfluxDB Enterprise](/enterprise_influxdb/v1.9/concepts/clustering/) +for more on cluster architecture. + +### Other requirements + +#### License key or file + +InfluxDB Enterprise requires a license key *or* a license file to run. +Your license key is available at [InfluxPortal](https://portal.influxdata.com/licenses). +Contact support at the email we provided at signup to receive a license file. +License files are required only if the nodes in your cluster cannot reach +`portal.influxdata.com` on port `80` or `443`. + +#### Ports + +Meta nodes communicate over ports `8088`, `8089`, and `8091`. + +For licensing purposes, meta nodes must also be able to reach `portal.influxdata.com` +on port `80` or `443`. +If the meta nodes cannot reach `portal.influxdata.com` on port `80` or `443`, +you'll need to set the `license-path` setting instead of the `license-key` +setting in the meta node configuration file. + +#### User account + +The installation package creates an `influxdb` user on the operating system. +The `influxdb` user runs the InfluxDB meta service. +The `influxdb` user also owns certain files needed to start the service. +In some cases, local policies may prevent the local user account from being created and the service fails to start. +Contact your systems administrator for assistance with this requirement. + +## Set up meta nodes + +1. [Add DNS entries](#add-dns-entries) +2. [Set up, configure, and start the meta services](#set-up-configure-and-start-the-meta-services) +3. [Join meta nodes to the cluster](#join-meta-nodes-to-the-cluster) + +### Add DNS entries + +Ensure that your servers' hostnames and IP addresses are added to your network's DNS environment. +The addition of DNS entries and IP assignment is usually site and policy specific; contact your DNS administrator for assistance as necessary. +Ultimately, use entries similar to the following (hostnames and domain IP addresses are representative). + +| Record Type | Hostname | IP | +|:------------|:---------------------------------:|--------------:| +| `A` | `enterprise-meta-01.mydomain.com` | `` | +| `A` | `enterprise-meta-02.mydomain.com` | `` | +| `A` | `enterprise-meta-03.mydomain.com` | `` | + + +#### Verify DNS resolution + +Before proceeding with the installation, verify on each server that the other +servers are resolvable. Here is an example set of shell commands using `ping`: + +``` +ping -qc 1 enterprise-meta-01 +ping -qc 1 enterprise-meta-02 +ping -qc 1 enterprise-meta-03 +``` + +We highly recommend that each server be able to resolve the IP from the hostname alone as shown here. +Resolve any connectivity issues before proceeding with the installation. +A healthy cluster requires that every meta node can communicate with every other +meta node. + +### Set up, configure, and start the meta services + +Perform the following steps on each meta server. + +#### I. Download and install the meta service + +##### Ubuntu & Debian (64-bit) + +``` +wget https://dl.influxdata.com/enterprise/releases/influxdb-meta_{{< latest-patch >}}-c{{< latest-patch >}}_amd64.deb +sudo dpkg -i influxdb-meta_{{< latest-patch >}}-c{{< latest-patch >}}_amd64.deb +``` + +##### RedHat & CentOS (64-bit) + +``` +wget https://dl.influxdata.com/enterprise/releases/influxdb-meta-{{< latest-patch >}}_c{{< latest-patch >}}.x86_64.rpm +sudo yum localinstall influxdb-meta-{{< latest-patch >}}_c{{< latest-patch >}}.x86_64.rpm +``` + +##### Verify the authenticity of release download (recommended) + +For added security, follow these steps to verify the signature of your InfluxDB download with `gpg`. + +1. Download and import InfluxData's public key: + + ``` + curl -s https://repos.influxdata.com/influxdb.key | gpg --import + ``` + +2. Download the signature file for the release by adding `.asc` to the download URL. + For example: + + ``` + wget https://dl.influxdata.com/enterprise/releases/influxdb-meta-{{< latest-patch >}}_c{{< latest-patch >}}.x86_64.rpm.asc + ``` + +3. Verify the signature with `gpg --verify`: + + ``` + gpg --verify influxdb-meta-{{< latest-patch >}}_c{{< latest-patch >}}.x86_64.rpm.asc influxdb-meta-{{< latest-patch >}}_c{{< latest-patch >}}.x86_64.rpm + ``` + + The output from this command should include the following: + + ``` + gpg: Good signature from "InfluxDB Packaging Service " [unknown] + ``` + +#### II. Edit the configuration file + +In `/etc/influxdb/influxdb-meta.conf`: + +* Uncomment `hostname` and set to the full hostname of the meta node. +* Uncomment `internal-shared-secret` in the `[meta]` section and set it to a long pass phrase to be used in JWT authentication for intra-node communication. This value must the same for all of your meta nodes and match the `[meta] meta-internal-shared-secret` settings in the configuration files of your data nodes. +* Set `license-key` in the `[enterprise]` section to the license key you received on InfluxPortal **OR** `license-path` in the `[enterprise]` section to the local path to the JSON license file you received from InfluxData. + +{{% warn %}} +The `license-key` and `license-path` settings are mutually exclusive and one must remain set to the empty string. +{{% /warn %}} + +``` +# Hostname advertised by this host for remote addresses. This must be resolvable by all +# other nodes in the cluster +hostname="" + +[enterprise] + # license-key and license-path are mutually exclusive, use only one and leave the other blank + license-key = "" # Mutually exclusive with license-path + + # license-key and license-path are mutually exclusive, use only one and leave the other blank + license-path = "/path/to/readable/JSON.license.file" # Mutually exclusive with license-key +``` + +#### III. Start the meta service + +On sysvinit systems, enter: +``` +service influxdb-meta start +``` + +On systemd systems, enter: +``` +sudo systemctl start influxdb-meta +``` + +#### Verify meta node process +Check to see that the process is running by entering: + +``` +ps aux | grep -v grep | grep influxdb-meta +``` + +You should see output similar to: + +``` +influxdb 3207 0.8 4.4 483000 22168 ? Ssl 17:05 0:08 /usr/bin/influxd-meta -config /etc/influxdb/influxdb-meta.conf +``` + +> **Note:** It is possible to start the cluster with a single meta node but you +must pass the `-single-server flag` when starting the single meta node. +Please note that a cluster with only one meta node is **not** recommended for +production environments. + +### Join meta nodes to the cluster + +From one and only one meta node, join all meta nodes including itself. +In our example, from `enterprise-meta-01`, run: + +``` +influxd-ctl add-meta enterprise-meta-01:8091 +influxd-ctl add-meta enterprise-meta-02:8091 +influxd-ctl add-meta enterprise-meta-03:8091 +``` + +> **Note:** Please make sure that you specify the fully qualified host name of +the meta node during the join process. +Please do not specify `localhost` as this can cause cluster connection issues. + +The expected output is: +``` +Added meta node x at enterprise-meta-0x:8091 +``` + +#### Verify cluster + +To verify the cluster, run the following command on any meta node: + +``` +influxd-ctl show +``` + +The expected output is: + +``` +Data Nodes +========== +ID TCP Address Version + +Meta Nodes +========== +TCP Address Version +enterprise-meta-01:8091 {{< latest-patch >}}-c{{< latest-patch >}} +enterprise-meta-02:8091 {{< latest-patch >}}-c{{< latest-patch >}} +enterprise-meta-03:8091 {{< latest-patch >}}-c{{< latest-patch >}} +``` + +Note that your cluster must have at least three meta nodes. +If you do not see your meta nodes in the output, retry adding them to +the cluster. + +Once your meta nodes are part of your cluster move on to [the next steps to +set up your data nodes](/enterprise_influxdb/v1.9/install-and-deploy/installation/data_node_installation/). +Please do not continue to the next steps if your meta nodes are not part of the +cluster. diff --git a/content/enterprise_influxdb/v1.9/introduction/installation_requirements.md b/content/enterprise_influxdb/v1.9/introduction/installation_requirements.md new file mode 100644 index 000000000..d94c8a71d --- /dev/null +++ b/content/enterprise_influxdb/v1.9/introduction/installation_requirements.md @@ -0,0 +1,108 @@ +--- +title: Installation requirements +description: Requirements for installing and deploying InfluxDB Enterprise. +aliases: + - /enterprise_influxdb/v1.9/introduction/meta_node_installation/ + - /enterprise_influxdb/v1.9/introduction/data_node_installation/ + - /enterprise/v1.8/introduction/installation_guidelines/ + - /enterprise_influxdb/v1.9/introduction/installation_guidelines/ +menu: + enterprise_influxdb_1_9: + weight: 102 + parent: Introduction +--- + +Review the installation requirements below, and then check out available options to [install and deploy InfluxDB Enterprise](/enterprise_influxdb/v1.9/install-and-deploy/). For an overview of the architecture and concepts in an InfluxDB Enterprise cluster, review [Clustering in InfluxDB Enterprise](/enterprise_influxdb/v1.9/concepts/clustering/). + +## Requirements for InfluxDB Enterprise clusters + +InfluxDB Enterprise clusters require a license. To use a license key, all nodes in the cluster must be able to contact https://portal.influxdata.com via port `80` or port `443`. If nodes in the cluster cannot communicate with https://portal.influxdata.com, you must use the `license-path` configuration setting. For more information, see [Enterprise license settings](/enterprise_influxdb/v1.9/administration/config-data-nodes/#enterprise-license-settings). + +Nodes attempt to download a new license file for the given key every four hours. If a node cannot connect to the server and retrieve a new license file, the node uses the existing license file. After a license expires, nodes have the following grace periods: + +- If [InfluxDB daemon (`influxd`)](/enterprise_influxdb/v1.9/tools/influxd#sidebar) starts and fails to validate the license, the node has a 4-hour grace period. +- If `influxd` starts and validates the license, and then a later license check fails, the node has a 14-day grace period. + +### Frequently overlooked requirements + +The following are the most frequently overlooked requirements when installing a cluster. + +- [Ensure connectivity between machines](#ensure-connectivity-between-machines) +- [Synchronize time between hosts](#synchronize-time-between-hosts) +- [Use SSDs](#use-ssds) +- [Do not use NFS](#do-not-use-nfs-mounts) +- [Disable swap](#disable-swap) +- [Use three and only three meta nodes](#use-three-and-only-three-meta-nodes) +- [Meta and data nodes are fully independent](#meta-and-data-nodes-are-fully-independent) +- [Install Chronograf last](#install-chronograf-last) + +#### Ensure connectivity between machines + +All nodes in the cluster must be able to resolve each other by hostname or IP, +whichever is used in the configuration files. + +For simplicity, ensure that all nodes can reach all other nodes on ports `8086`, `8088`, `8089`, and `8091`. +If you alter the default ports in the configuration file(s), ensure the configured ports are open between the nodes. + +#### Synchronize time between hosts + +InfluxDB Enterprise uses hosts' local time in UTC to assign timestamps to data and for coordination purposes. +Use the Network Time Protocol (NTP) to synchronize time between hosts. + +#### Use SSDs + +Clusters require sustained availability of 1000-2000 IOPS from the attached storage. +SANs must guarantee at least 1000 IOPS is always available to InfluxDB Enterprise +nodes or they may not be sufficient. +SSDs are strongly recommended, and we have had no reports of IOPS contention from any customers running on SSDs. + +#### Do not use NFS + +For disk storage, use block devices only. +InfluxDB Enterprise does not support NFS (Network File System)-mounted devices. + +#### Disable swap + +To avoid potential disk contention when InfluxDB is under high load, +disable swap in your operating system settings. + +#### Use three and only three meta nodes + +Although technically the cluster can function with any number of meta nodes, the best practice is to ALWAYS have an odd number of meta nodes. +This allows the meta nodes to reach consensus. +An even number of meta nodes cannot achieve consensus because there can be no "deciding vote" cast between the nodes if they disagree. + +Therefore, the minimum number of meta nodes for a high availability (HA) installation is three. The typical HA installation for InfluxDB Enterprise deploys three meta nodes. + +Aside from three being a magic number, a three meta node cluster can tolerate the permanent loss of a single meta node with no degradation in any function or performance. +A replacement meta node can be added to restore the cluster to full redundancy. +A three meta node cluster that loses two meta nodes will still be able to handle +basic writes and queries, but no new shards, databases, users, etc. can be created. + +Running a cluster with five meta nodes does allow for the permanent loss of +two meta nodes without impact on the cluster, but it doubles the +Raft communication overhead. + +#### Meta and data nodes are fully independent + +Meta nodes run the Raft consensus protocol together, and manage the metastore of +all shared cluster information: cluster nodes, databases, retention policies, +shard groups, users, continuous queries, and subscriptions. + +Data nodes store the shard groups and respond to queries. +They request metastore information from the meta group as needed. + +There is no requirement at all for there to be a meta process on a data node, +or for there to be a meta process per data node. +Three meta nodes is enough for an arbitrary number of data nodes, and for best +redundancy, all nodes should run on independent servers. + +#### Install Chronograf last + +Chronograf should not be installed or configured until the +InfluxDB Enterprise cluster is fully functional. + +#### Set up monitoring + +Monitoring gives you visibility into the status and performance of your cluster. +See ["Monitor the InfluxData Platform"](/platform/monitoring/influxdata-platform/) for information on setting up monitoring for your InfluxDB Enterprise installation. diff --git a/content/enterprise_influxdb/v1.9/query_language/_index.md b/content/enterprise_influxdb/v1.9/query_language/_index.md new file mode 100644 index 000000000..7faf962fe --- /dev/null +++ b/content/enterprise_influxdb/v1.9/query_language/_index.md @@ -0,0 +1,87 @@ +--- +title: Influx Query Language (InfluxQL) +description: > + Influx Query Language (InfluxQL) is Influx DB's SQL-like query language. +menu: + enterprise_influxdb_1_9: + weight: 70 + identifier: InfluxQL +--- + +This section introduces InfluxQL, the InfluxDB SQL-like query language for +working with data in InfluxDB databases. + +## InfluxQL tutorial +The first seven documents in this section provide a tutorial-style introduction +to InfluxQL. +Feel free to download the dataset provided in +[Sample Data](/enterprise_influxdb/v1.9/query_language/data_download/) and follow along +with the documentation. + +#### Data exploration + +[Data exploration](/enterprise_influxdb/v1.9/query_language/explore-data/) covers the +query language basics for InfluxQL, including the +[`SELECT` statement](/enterprise_influxdb/v1.9/query_language/explore-data/#the-basic-select-statement), +[`GROUP BY` clauses](/enterprise_influxdb/v1.9/query_language/explore-data/#the-group-by-clause), +[`INTO` clauses](/enterprise_influxdb/v1.9/query_language/explore-data/#the-into-clause), and more. +See Data Exploration to learn about +[time syntax](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) and +[regular expressions](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions) in +queries. + +#### Schema exploration + +[Schema exploration](/enterprise_influxdb/v1.9/query_language/explore-schema/) covers +queries that are useful for viewing and exploring your +[schema](/enterprise_influxdb/v1.9/concepts/glossary/#schema). +See Schema Exploration for syntax explanations and examples of InfluxQL's `SHOW` +queries. + +#### Database management + +[Database management](/enterprise_influxdb/v1.9/query_language/manage-database/) covers InfluxQL for managing +[databases](/enterprise_influxdb/v1.9/concepts/glossary/#database) and +[retention policies](/enterprise_influxdb/v1.9/concepts/glossary/#retention-policy-rp) in +InfluxDB. +See Database Management for creating and dropping databases and retention +policies as well as deleting and dropping data. + +#### InfluxQL functions + +Covers all [InfluxQL functions](/enterprise_influxdb/v1.9/query_language/functions/). + +#### InfluxQL Continuous Queries + +[InfluxQL Continuous Queries](/enterprise_influxdb/v1.9/query_language/continuous_queries/) covers the +[basic syntax](/enterprise_influxdb/v1.9/query_language/continuous_queries/#basic-syntax) +, +[advanced syntax](/enterprise_influxdb/v1.9/query_language/continuous_queries/#advanced-syntax) +, +and +[common use cases](/enterprise_influxdb/v1.9/query_language/continuous_queries/#continuous-query-use-cases) +for +[Continuous Queries](/enterprise_influxdb/v1.9/concepts/glossary/#continuous-query-cq). +This page also describes how to +[`SHOW`](/enterprise_influxdb/v1.9/query_language/continuous_queries/#listing-continuous-queries) and +[`DROP`](/enterprise_influxdb/v1.9/query_language/continuous_queries/#deleting-continuous-queries) +Continuous Queries. + +#### InfluxQL mathematical operators + +[InfluxQL mathematical operators](/enterprise_influxdb/v1.9/query_language/math_operators/) +covers the use of mathematical operators in InfluxQL. + +#### Authentication and authorization + +[Authentication and authorization](/enterprise_influxdb/v1.9/administration/authentication_and_authorization/) covers how to +[set up authentication](/enterprise_influxdb/v1.9/administration/authentication_and_authorization/#set-up-authentication) +and how to +[authenticate requests](/enterprise_influxdb/v1.9/administration/authentication_and_authorization/#authenticate-requests) in InfluxDB. +This page also describes the different +[user types](/enterprise_influxdb/v1.9/administration/authentication_and_authorization/#user-types-and-privileges) and the InfluxQL for +[managing database users](/enterprise_influxdb/v1.9/administration/authentication_and_authorization/#user-management-commands). + +## InfluxQL reference + +The [reference documentation for InfluxQL](/enterprise_influxdb/v1.9/query_language/spec/). diff --git a/content/enterprise_influxdb/v1.9/query_language/continuous_queries.md b/content/enterprise_influxdb/v1.9/query_language/continuous_queries.md new file mode 100644 index 000000000..006e3a912 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/query_language/continuous_queries.md @@ -0,0 +1,987 @@ +--- +title: InfluxQL Continuous Queries +description: > + Continuous queries (CQ) are InfluxQL queries that run automatically and periodically on realtime data and store query results in a specified measurement. +menu: + enterprise_influxdb_1_9: + name: Continuous Queries + weight: 50 + parent: InfluxQL +v2: /influxdb/v2.0/process-data/ +--- + +## Introduction + +Continuous queries (CQ) are InfluxQL queries that run automatically and +periodically on realtime data and store query results in a +specified measurement. + + + + + + + + + + + + + + + + + +
        Basic SyntaxAdvanced SyntaxCQ Management
        Examples of Basic SyntaxExamples of Advanced SyntaxCQ Use Cases
        Common Issues with Basic SyntaxCommon Issues with Advanced SyntaxFurther information
        + +## Syntax + +### Basic syntax + +```sql +CREATE CONTINUOUS QUERY ON +BEGIN + +END +``` + +#### Description of basic syntax + +##### The `cq_query` + +The `cq_query` requires a +[function](/enterprise_influxdb/v1.9/concepts/glossary/#function), +an [`INTO` clause](/enterprise_influxdb/v1.9/query_language/spec/#clauses), +and a [`GROUP BY time()` clause](/enterprise_influxdb/v1.9/query_language/spec/#clauses): + +```sql +SELECT INTO FROM [WHERE ] GROUP BY time()[,] +``` + +>**Note:** Notice that the `cq_query` does not require a time range in a `WHERE` clause. +InfluxDB automatically generates a time range for the `cq_query` when it executes the CQ. +Any user-specified time ranges in the `cq_query`'s `WHERE` clause will be ignored +by the system. + +##### Schedule and coverage + +Continuous queries operate on real-time data. +They use the local server’s timestamp, the `GROUP BY time()` interval, and +InfluxDB database's preset time boundaries to determine when to execute and what time +range to cover in the query. + +CQs execute at the same interval as the `cq_query`'s `GROUP BY time()` interval, +and they run at the start of the InfluxDB database's preset time boundaries. +If the `GROUP BY time()` interval is one hour, the CQ executes at the start of +every hour. + +When the CQ executes, it runs a single query for the time range between +[`now()`](/enterprise_influxdb/v1.9/concepts/glossary/#now) and `now()` minus the +`GROUP BY time()` interval. +If the `GROUP BY time()` interval is one hour and the current time is 17:00, +the query's time range is between 16:00 and 16:59.999999999. + +#### Examples of basic syntax + +The examples below use the following sample data in the `transportation` +database. +The measurement `bus_data` stores 15-minute resolution data on the number of bus +`passengers` and `complaints`: + +```sql +name: bus_data +-------------- +time passengers complaints +2016-08-28T07:00:00Z 5 9 +2016-08-28T07:15:00Z 8 9 +2016-08-28T07:30:00Z 8 9 +2016-08-28T07:45:00Z 7 9 +2016-08-28T08:00:00Z 8 9 +2016-08-28T08:15:00Z 15 7 +2016-08-28T08:30:00Z 15 7 +2016-08-28T08:45:00Z 17 7 +2016-08-28T09:00:00Z 20 7 +``` + +##### Automatically downsampling data + +Use a simple CQ to automatically downsample data from a single field +and write the results to another measurement in the same database. + +```sql +CREATE CONTINUOUS QUERY "cq_basic" ON "transportation" +BEGIN + SELECT mean("passengers") INTO "average_passengers" FROM "bus_data" GROUP BY time(1h) +END +``` + +`cq_basic` calculates the average hourly number of passengers from the +`bus_data` measurement and stores the results in the `average_passengers` +measurement in the `transportation` database. + +`cq_basic` executes at one-hour intervals, the same interval as the +`GROUP BY time()` interval. +Every hour, `cq_basic` runs a single query that covers the time range between +`now()` and `now()` minus the `GROUP BY time()` interval, that is, the time +range between `now()` and one hour prior to `now()`. + +Annotated log output on the morning of August 28, 2016: + +```sql +> +At **8:00** `cq_basic` executes a query with the time range `time >= '7:00' AND time < '08:00'`. +`cq_basic` writes one point to the `average_passengers` measurement: +> + name: average_passengers + ------------------------ + time mean + 2016-08-28T07:00:00Z 7 +> +At **9:00** `cq_basic` executes a query with the time range `time >= '8:00' AND time < '9:00'`. +`cq_basic` writes one point to the `average_passengers` measurement: +> + name: average_passengers + ------------------------ + time mean + 2016-08-28T08:00:00Z 13.75 +``` + +Here are the results: + +```sql +> SELECT * FROM "average_passengers" +name: average_passengers +------------------------ +time mean +2016-08-28T07:00:00Z 7 +2016-08-28T08:00:00Z 13.75 +``` + +##### Automatically downsampling data into another retention policy + +[Fully qualify](/enterprise_influxdb/v1.9/query_language/explore-data/#the-basic-select-statement) +the destination measurement to store the downsampled data in a non-`DEFAULT` +[retention policy](/enterprise_influxdb/v1.9/concepts/glossary/#retention-policy-rp) (RP). + +```sql +CREATE CONTINUOUS QUERY "cq_basic_rp" ON "transportation" +BEGIN + SELECT mean("passengers") INTO "transportation"."three_weeks"."average_passengers" FROM "bus_data" GROUP BY time(1h) +END +``` + +`cq_basic_rp` calculates the average hourly number of passengers from the +`bus_data` measurement and stores the results in the `transportation` database, +the `three_weeks` RP, and the `average_passengers` measurement. + +`cq_basic_rp` executes at one-hour intervals, the same interval as the +`GROUP BY time()` interval. +Every hour, `cq_basic_rp` runs a single query that covers the time range between +`now()` and `now()` minus the `GROUP BY time()` interval, that is, the time +range between `now()` and one hour prior to `now()`. + +Annotated log output on the morning of August 28, 2016: + +```sql +> +At **8:00** `cq_basic_rp` executes a query with the time range `time >= '7:00' AND time < '8:00'`. +`cq_basic_rp` writes one point to the `three_weeks` RP and the `average_passengers` measurement: +> + name: average_passengers + ------------------------ + time mean + 2016-08-28T07:00:00Z 7 +> +At **9:00** `cq_basic_rp` executes a query with the time range +`time >= '8:00' AND time < '9:00'`. +`cq_basic_rp` writes one point to the `three_weeks` RP and the `average_passengers` measurement: +> + name: average_passengers + ------------------------ + time mean + 2016-08-28T08:00:00Z 13.75 +``` + +Here are the results: + +```sql +> SELECT * FROM "transportation"."three_weeks"."average_passengers" +name: average_passengers +------------------------ +time mean +2016-08-28T07:00:00Z 7 +2016-08-28T08:00:00Z 13.75 +``` + +`cq_basic_rp` uses CQs and retention policies to automatically downsample data +and keep those downsampled data for an alternative length of time. +See the [Downsampling and Data Retention](/enterprise_influxdb/v1.9/guides/downsampling_and_retention/) +guide for an in-depth discussion about this CQ use case. + +##### Automatically downsampling a database with backreferencing + +Use a function with a wildcard (`*`) and `INTO` query's +[backreferencing syntax](/enterprise_influxdb/v1.9/query_language/explore-data/#the-into-clause) +to automatically downsample data from all measurements and numerical fields in +a database. + +```sql +CREATE CONTINUOUS QUERY "cq_basic_br" ON "transportation" +BEGIN + SELECT mean(*) INTO "downsampled_transportation"."autogen".:MEASUREMENT FROM /.*/ GROUP BY time(30m),* +END +``` + +`cq_basic_br` calculates the 30-minute average of `passengers` and `complaints` +from every measurement in the `transportation` database (in this case, there's only the +`bus_data` measurement). +It stores the results in the `downsampled_transportation` database. + +`cq_basic_br` executes at 30 minutes intervals, the same interval as the +`GROUP BY time()` interval. +Every 30 minutes, `cq_basic_br` runs a single query that covers the time range +between `now()` and `now()` minus the `GROUP BY time()` interval, that is, +the time range between `now()` and 30 minutes prior to `now()`. + +Annotated log output on the morning of August 28, 2016: + +```sql +> +At **7:30**, `cq_basic_br` executes a query with the time range `time >= '7:00' AND time < '7:30'`. +`cq_basic_br` writes two points to the `bus_data` measurement in the `downsampled_transportation` database: +> + name: bus_data + -------------- + time mean_complaints mean_passengers + 2016-08-28T07:00:00Z 9 6.5 +> +At **8:00**, `cq_basic_br` executes a query with the time range `time >= '7:30' AND time < '8:00'`. +`cq_basic_br` writes two points to the `bus_data` measurement in the `downsampled_transportation` database: +> + name: bus_data + -------------- + time mean_complaints mean_passengers + 2016-08-28T07:30:00Z 9 7.5 +> +[...] +> +At **9:00**, `cq_basic_br` executes a query with the time range `time >= '8:30' AND time < '9:00'`. +`cq_basic_br` writes two points to the `bus_data` measurement in the `downsampled_transportation` database: +> + name: bus_data + -------------- + time mean_complaints mean_passengers + 2016-08-28T08:30:00Z 7 16 +``` + +Here are the results: + +```sql +> SELECT * FROM "downsampled_transportation."autogen"."bus_data" +name: bus_data +-------------- +time mean_complaints mean_passengers +2016-08-28T07:00:00Z 9 6.5 +2016-08-28T07:30:00Z 9 7.5 +2016-08-28T08:00:00Z 8 11.5 +2016-08-28T08:30:00Z 7 16 +``` + +##### Automatically downsampling data and configuring CQ time boundaries + +Use an +[offset interval](/enterprise_influxdb/v1.9/query_language/explore-data/#advanced-group-by-time-syntax) +in the `GROUP BY time()` clause to alter both the CQ's default execution time and +preset time boundaries. + +```sql +CREATE CONTINUOUS QUERY "cq_basic_offset" ON "transportation" +BEGIN + SELECT mean("passengers") INTO "average_passengers" FROM "bus_data" GROUP BY time(1h,15m) +END +``` + +`cq_basic_offset`calculates the average hourly number of passengers from the +`bus_data` measurement and stores the results in the `average_passengers` +measurement. + +`cq_basic_offset` executes at one-hour intervals, the same interval as the +`GROUP BY time()` interval. +The 15 minute offset interval forces the CQ to execute 15 minutes after the +default execution time; `cq_basic_offset` executes at 8:15 instead of 8:00. + +Every hour, `cq_basic_offset` runs a single query that covers the time range +between `now()` and `now()` minus the `GROUP BY time()` interval, that is, the +time range between `now()` and one hour prior to `now()`. +The 15 minute offset interval shifts forward the generated preset time boundaries in the +CQ's `WHERE` clause; `cq_basic_offset` queries between 7:15 and 8:14.999999999 instead of 7:00 and 7:59.999999999. + +Annotated log output on the morning of August 28, 2016: + +``` +> +At **8:15** `cq_basic_offset` executes a query with the time range `time >= '7:15' AND time < '8:15'`. +`cq_basic_offset` writes one point to the `average_passengers` measurement: +> + name: average_passengers + ------------------------ + time mean + 2016-08-28T07:15:00Z 7.75 +> +At **9:15** `cq_basic_offset` executes a query with the time range `time >= '8:15' AND time < '9:15'`. +`cq_basic_offset` writes one point to the `average_passengers` measurement: +> + name: average_passengers + ------------------------ + time mean + 2016-08-28T08:15:00Z 16.75 +``` + +Here are the results: + +```sql +> SELECT * FROM "average_passengers" +name: average_passengers +------------------------ +time mean +2016-08-28T07:15:00Z 7.75 +2016-08-28T08:15:00Z 16.75 +``` + +Notice that the timestamps are for 7:15 and 8:15 instead of 7:00 and 8:00. + +#### Common issues with basic syntax + +##### Handling time intervals with no data + +CQs do not write any results for a time interval if no data fall within that +time range. + +Note that the basic syntax does not support using +[`fill()`](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals-and-fill) +to change the value reported for intervals with no data. +Basic syntax CQs ignore `fill()` if it's included in the CQ query. +A possible workaround is to use the +[advanced CQ syntax](#advanced-syntax). + +##### Resampling previous time intervals + +The basic CQ runs a single query that covers the time range between `now()` +and `now()` minus the `GROUP BY time()` interval. +See the [advanced syntax](#advanced-syntax) for how to configure the query's +time range. + +##### Backfilling results for older data + +CQs operate on realtime data, that is, data with timestamps that occur +relative to [`now()`](/enterprise_influxdb/v1.9/concepts/glossary/#now). +Use a basic +[`INTO` query](/enterprise_influxdb/v1.9/query_language/explore-data/#the-into-clause) +to backfill results for data with older timestamps. + +##### Missing tags in the CQ results + +By default, all +[`INTO` queries](/enterprise_influxdb/v1.9/query_language/explore-data/#the-into-clause) +convert any tags in the source measurement to fields in the destination +measurement. + +Include `GROUP BY *` in the CQ to preserve tags in the destination measurement. + +### Advanced syntax + +```txt +CREATE CONTINUOUS QUERY ON +RESAMPLE EVERY FOR +BEGIN + +END +``` + +#### Description of advanced syntax + +##### The `cq_query` + +See [ Description of Basic Syntax](/enterprise_influxdb/v1.9/query_language/continuous_queries/#description-of-basic-syntax). + +##### Scheduling and coverage + +CQs operate on real-time data. With the advanced syntax, CQs use the local +server’s timestamp, the information in the `RESAMPLE` clause, and the InfluxDB +server's preset time boundaries to determine when to execute and what time range to +cover in the query. + +CQs execute at the same interval as the `EVERY` interval in the `RESAMPLE` +clause, and they run at the start of InfluxDB’s preset time boundaries. +If the `EVERY` interval is two hours, InfluxDB executes the CQ at the top of +every other hour. + +When the CQ executes, it runs a single query for the time range between +[`now()`](/enterprise_influxdb/v1.9/concepts/glossary/#now) and `now()` minus the `FOR` interval in the `RESAMPLE` clause. +If the `FOR` interval is two hours and the current time is 17:00, the query's +time range is between 15:00 and 16:59.999999999. + +Both the `EVERY` interval and the `FOR` interval accept +[duration literals](/enterprise_influxdb/v1.9/query_language/spec/#durations). +The `RESAMPLE` clause works with either or both of the `EVERY` and `FOR` intervals +configured. +CQs default to the relevant +[basic syntax behavior](/enterprise_influxdb/v1.9/query_language/continuous_queries/#description-of-basic-syntax) +if the `EVERY` interval or `FOR` interval is not provided (see the first issue in +[Common Issues with Advanced Syntax](/enterprise_influxdb/v1.9/query_language/continuous_queries/#common-issues-with-advanced-syntax) +for an anomalous case). + +#### Examples of advanced syntax + +The examples below use the following sample data in the `transportation` database. +The measurement `bus_data` stores 15-minute resolution data on the number of bus +`passengers`: + +```sql +name: bus_data +-------------- +time passengers +2016-08-28T06:30:00Z 2 +2016-08-28T06:45:00Z 4 +2016-08-28T07:00:00Z 5 +2016-08-28T07:15:00Z 8 +2016-08-28T07:30:00Z 8 +2016-08-28T07:45:00Z 7 +2016-08-28T08:00:00Z 8 +2016-08-28T08:15:00Z 15 +2016-08-28T08:30:00Z 15 +2016-08-28T08:45:00Z 17 +2016-08-28T09:00:00Z 20 +``` + +##### Configuring execution intervals + +Use an `EVERY` interval in the `RESAMPLE` clause to specify the CQ's execution +interval. + +```sql +CREATE CONTINUOUS QUERY "cq_advanced_every" ON "transportation" +RESAMPLE EVERY 30m +BEGIN + SELECT mean("passengers") INTO "average_passengers" FROM "bus_data" GROUP BY time(1h) +END +``` + +`cq_advanced_every` calculates the one-hour average of `passengers` +from the `bus_data` measurement and stores the results in the +`average_passengers` measurement in the `transportation` database. + +`cq_advanced_every` executes at 30-minute intervals, the same interval as the +`EVERY` interval. +Every 30 minutes, `cq_advanced_every` runs a single query that covers the time +range for the current time bucket, that is, the one-hour time bucket that +intersects with `now()`. + +Annotated log output on the morning of August 28, 2016: + +```sql +> +At **8:00**, `cq_advanced_every` executes a query with the time range `WHERE time >= '7:00' AND time < '8:00'`. +`cq_advanced_every` writes one point to the `average_passengers` measurement: +> + name: average_passengers + ------------------------ + time mean + 2016-08-28T07:00:00Z 7 +> +At **8:30**, `cq_advanced_every` executes a query with the time range `WHERE time >= '8:00' AND time < '9:00'`. +`cq_advanced_every` writes one point to the `average_passengers` measurement: +> + name: average_passengers + ------------------------ + time mean + 2016-08-28T08:00:00Z 12.6667 +> +At **9:00**, `cq_advanced_every` executes a query with the time range `WHERE time >= '8:00' AND time < '9:00'`. +`cq_advanced_every` writes one point to the `average_passengers` measurement: +> + name: average_passengers + ------------------------ + time mean + 2016-08-28T08:00:00Z 13.75 +``` + +Here are the results: + +```sql +> SELECT * FROM "average_passengers" +name: average_passengers +------------------------ +time mean +2016-08-28T07:00:00Z 7 +2016-08-28T08:00:00Z 13.75 +``` + +Notice that `cq_advanced_every` calculates the result for the 8:00 time interval +twice. +First, it runs at 8:30 and calculates the average for every available data point +between 8:00 and 9:00 (`8`,`15`, and `15`). +Second, it runs at 9:00 and calculates the average for every available data +point between 8:00 and 9:00 (`8`, `15`, `15`, and `17`). +Because of the way InfluxDB +[handles duplicate points](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#how-does-influxdb-handle-duplicate-points) +, the second result simply overwrites the first result. + +##### Configuring time ranges for resampling + +Use a `FOR` interval in the `RESAMPLE` clause to specify the length of the CQ's +time range. + +```sql +CREATE CONTINUOUS QUERY "cq_advanced_for" ON "transportation" +RESAMPLE FOR 1h +BEGIN + SELECT mean("passengers") INTO "average_passengers" FROM "bus_data" GROUP BY time(30m) +END +``` + +`cq_advanced_for` calculates the 30-minute average of `passengers` +from the `bus_data` measurement and stores the results in the `average_passengers` +measurement in the `transportation` database. + +`cq_advanced_for` executes at 30-minute intervals, the same interval as the +`GROUP BY time()` interval. +Every 30 minutes, `cq_advanced_for` runs a single query that covers the time +range between `now()` and `now()` minus the `FOR` interval, that is, the time +range between `now()` and one hour prior to `now()`. + +Annotated log output on the morning of August 28, 2016: + +```sql +> +At **8:00** `cq_advanced_for` executes a query with the time range `WHERE time >= '7:00' AND time < '8:00'`. +`cq_advanced_for` writes two points to the `average_passengers` measurement: +> + name: average_passengers + ------------------------ + time mean + 2016-08-28T07:00:00Z 6.5 + 2016-08-28T07:30:00Z 7.5 +> +At **8:30** `cq_advanced_for` executes a query with the time range `WHERE time >= '7:30' AND time < '8:30'`. +`cq_advanced_for` writes two points to the `average_passengers` measurement: +> + name: average_passengers + ------------------------ + time mean + 2016-08-28T07:30:00Z 7.5 + 2016-08-28T08:00:00Z 11.5 +> +At **9:00** `cq_advanced_for` executes a query with the time range `WHERE time >= '8:00' AND time < '9:00'`. +`cq_advanced_for` writes two points to the `average_passengers` measurement: +> + name: average_passengers + ------------------------ + time mean + 2016-08-28T08:00:00Z 11.5 + 2016-08-28T08:30:00Z 16 +``` + +Notice that `cq_advanced_for` will calculate the result for every time interval +twice. +The CQ calculates the average for the 7:30 time interval at 8:00 and at 8:30, +and it calculates the average for the 8:00 time interval at 8:30 and 9:00. + +Here are the results: + +```sql +> SELECT * FROM "average_passengers" +name: average_passengers +------------------------ +time mean +2016-08-28T07:00:00Z 6.5 +2016-08-28T07:30:00Z 7.5 +2016-08-28T08:00:00Z 11.5 +2016-08-28T08:30:00Z 16 +``` + +##### Configuring execution intervals and CQ time ranges + +Use an `EVERY` interval and `FOR` interval in the `RESAMPLE` clause to specify +the CQ's execution interval and the length of the CQ's time range. + +```sql +CREATE CONTINUOUS QUERY "cq_advanced_every_for" ON "transportation" +RESAMPLE EVERY 1h FOR 90m +BEGIN + SELECT mean("passengers") INTO "average_passengers" FROM "bus_data" GROUP BY time(30m) +END +``` + +`cq_advanced_every_for` calculates the 30-minute average of +`passengers` from the `bus_data` measurement and stores the results in the +`average_passengers` measurement in the `transportation` database. + +`cq_advanced_every_for` executes at one-hour intervals, the same interval as the +`EVERY` interval. +Every hour, `cq_advanced_every_for` runs a single query that covers the time +range between `now()` and `now()` minus the `FOR` interval, that is, the time +range between `now()` and 90 minutes prior to `now()`. + +Annotated log output on the morning of August 28, 2016: + +```sql +> +At **8:00** `cq_advanced_every_for` executes a query with the time range `WHERE time >= '6:30' AND time < '8:00'`. +`cq_advanced_every_for` writes three points to the `average_passengers` measurement: +> + name: average_passengers + ------------------------ + time mean + 2016-08-28T06:30:00Z 3 + 2016-08-28T07:00:00Z 6.5 + 2016-08-28T07:30:00Z 7.5 +> +At **9:00** `cq_advanced_every_for` executes a query with the time range `WHERE time >= '7:30' AND time < '9:00'`. +`cq_advanced_every_for` writes three points to the `average_passengers` measurement: +> + name: average_passengers + ------------------------ + time mean + 2016-08-28T07:30:00Z 7.5 + 2016-08-28T08:00:00Z 11.5 + 2016-08-28T08:30:00Z 16 +``` + +Notice that `cq_advanced_every_for` will calculate the result for every time +interval twice. +The CQ calculates the average for the 7:30 interval at 8:00 and 9:00. + +Here are the results: + +```sql +> SELECT * FROM "average_passengers" +name: average_passengers +------------------------ +time mean +2016-08-28T06:30:00Z 3 +2016-08-28T07:00:00Z 6.5 +2016-08-28T07:30:00Z 7.5 +2016-08-28T08:00:00Z 11.5 +2016-08-28T08:30:00Z 16 +``` + +##### Configuring CQ time ranges and filling empty results + +Use a `FOR` interval and `fill()` to change the value reported for time +intervals with no data. +Note that at least one data point must fall within the `FOR` interval for `fill()` +to operate. +If no data fall within the `FOR` interval the CQ writes no points to the +destination measurement. + +```sql +CREATE CONTINUOUS QUERY "cq_advanced_for_fill" ON "transportation" +RESAMPLE FOR 2h +BEGIN + SELECT mean("passengers") INTO "average_passengers" FROM "bus_data" GROUP BY time(1h) fill(1000) +END +``` + +`cq_advanced_for_fill` calculates the one-hour average of `passengers` from the +`bus_data` measurement and stores the results in the `average_passengers` +measurement in the `transportation` database. +Where possible, it writes the value `1000` for time intervals with no results. + +`cq_advanced_for_fill` executes at one-hour intervals, the same interval as the +`GROUP BY time()` interval. +Every hour, `cq_advanced_for_fill` runs a single query that covers the time +range between `now()` and `now()` minus the `FOR` interval, that is, the time +range between `now()` and two hours prior to `now()`. + +Annotated log output on the morning of August 28, 2016: + +```sql +> +At **6:00**, `cq_advanced_for_fill` executes a query with the time range `WHERE time >= '4:00' AND time < '6:00'`. +`cq_advanced_for_fill` writes nothing to `average_passengers`; `bus_data` has no data +that fall within that time range. +> +At **7:00**, `cq_advanced_for_fill` executes a query with the time range `WHERE time >= '5:00' AND time < '7:00'`. +`cq_advanced_for_fill` writes two points to `average_passengers`: +> + name: average_passengers + ------------------------ + time mean + 2016-08-28T05:00:00Z 1000 <------ fill(1000) + 2016-08-28T06:00:00Z 3 <------ average of 2 and 4 +> +[...] +> +At **11:00**, `cq_advanced_for_fill` executes a query with the time range `WHERE time >= '9:00' AND time < '11:00'`. +`cq_advanced_for_fill` writes two points to `average_passengers`: +> + name: average_passengers + ------------------------ + 2016-08-28T09:00:00Z 20 <------ average of 20 + 2016-08-28T10:00:00Z 1000 <------ fill(1000) +> +``` + +At **12:00**, `cq_advanced_for_fill` executes a query with the time range `WHERE time >= '10:00' AND time < '12:00'`. +`cq_advanced_for_fill` writes nothing to `average_passengers`; `bus_data` has no data +that fall within that time range. + +Here are the results: + +```sql +> SELECT * FROM "average_passengers" +name: average_passengers +------------------------ +time mean +2016-08-28T05:00:00Z 1000 +2016-08-28T06:00:00Z 3 +2016-08-28T07:00:00Z 7 +2016-08-28T08:00:00Z 13.75 +2016-08-28T09:00:00Z 20 +2016-08-28T10:00:00Z 1000 +``` + +> **Note:** `fill(previous)` doesn’t fill the result for a time interval if the +previous value is outside the query’s time range. +See [Frequently Asked Questions](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#why-does-fill-previous-return-empty-results) +for more information. + +#### Common issues with advanced syntax + +##### If the `EVERY` interval is greater than the `GROUP BY time()` interval + +If the `EVERY` interval is greater than the `GROUP BY time()` interval, the CQ +executes at the same interval as the `EVERY` interval and runs a single query +that covers the time range between `now()` and `now()` minus the `EVERY` +interval (not between `now()` and `now()` minus the `GROUP BY time()` interval). + +For example, if the `GROUP BY time()` interval is `5m` and the `EVERY` interval +is `10m`, the CQ executes every ten minutes. +Every ten minutes, the CQ runs a single query that covers the time range +between `now()` and `now()` minus the `EVERY` interval, that is, the time +range between `now()` and ten minutes prior to `now()`. + +This behavior is intentional and prevents the CQ from missing data between +execution times. + +##### If the `FOR` interval is less than the execution interval + +If the `FOR` interval is less than the `GROUP BY time()` interval or, if +specified, the `EVERY` interval, InfluxDB returns the following error: + +```sql +error parsing query: FOR duration must be >= GROUP BY time duration: must be a minimum of got +``` + +To avoid missing data between execution times, the `FOR` interval must be equal +to or greater than the `GROUP BY time()` interval or, if specified, the `EVERY` +interval. + +Currently, this is the intended behavior. +GitHub Issue [#6963](https://github.com/influxdata/influxdb/issues/6963) +outlines a feature request for CQs to support gaps in data coverage. + +## Continuous query management + +Only admin users are allowed to work with CQs. For more on user privileges, see [Authentication and Authorization](/enterprise_influxdb/v1.9/administration/authentication_and_authorization/#user-types-and-privileges). + +### Listing continuous queries + +List every CQ on an InfluxDB instance with: + +```sql +SHOW CONTINUOUS QUERIES +``` + +`SHOW CONTINUOUS QUERIES` groups results by database. + +##### Examples + +The output shows that the `telegraf` and `mydb` databases have CQs: + +```sql +> SHOW CONTINUOUS QUERIES +name: _internal +--------------- +name query + + +name: telegraf +-------------- +name query +idle_hands CREATE CONTINUOUS QUERY idle_hands ON telegraf BEGIN SELECT min(usage_idle) INTO telegraf.autogen.min_hourly_cpu FROM telegraf.autogen.cpu GROUP BY time(1h) END +feeling_used CREATE CONTINUOUS QUERY feeling_used ON telegraf BEGIN SELECT mean(used) INTO downsampled_telegraf.autogen.:MEASUREMENT FROM telegraf.autogen./.*/ GROUP BY time(1h) END + + +name: downsampled_telegraf +-------------------------- +name query + + +name: mydb +---------- +name query +vampire CREATE CONTINUOUS QUERY vampire ON mydb BEGIN SELECT count(dracula) INTO mydb.autogen.all_of_them FROM mydb.autogen.one GROUP BY time(5m) END +``` + +### Deleting continuous queries + +Delete a CQ from a specific database with: + +```sql +DROP CONTINUOUS QUERY ON +``` + +`DROP CONTINUOUS QUERY` returns an empty result. + +##### Examples + +Drop the `idle_hands` CQ from the `telegraf` database: + +```sql +> DROP CONTINUOUS QUERY "idle_hands" ON "telegraf"` +> +``` + +### Altering continuous queries + +CQs cannot be altered once they're created. +To change a CQ, you must `DROP` and re`CREATE` it with the updated settings. + +### Continuous query statistics + +If `query-stats-enabled` is set to `true` in your `influxdb.conf` or using the `INFLUXDB_CONTINUOUS_QUERIES_QUERY_STATS_ENABLED` environment variable, data will be written to `_internal` with information about when continuous queries ran and their duration. +Information about CQ configuration settings is available in the [Configuration](/enterprise_influxdb/v1.9/administration/config/#continuous-queries-settings) documentation. + +> **Note:** `_internal` houses internal system data and is meant for internal use. +The structure of and data stored in `_internal` can change at any time. +Use of this data falls outside the scope of official InfluxData support. + +## Continuous query use cases + +### Downsampling and Data Retention + +Use CQs with InfluxDB database +[retention policies](/enterprise_influxdb/v1.9/concepts/glossary/#retention-policy-rp) +(RPs) to mitigate storage concerns. +Combine CQs and RPs to automatically downsample high precision data to a lower +precision and remove the dispensable, high precision data from the database. + +See the +[Downsampling and data retention](/enterprise_influxdb/v1.9/guides/downsampling_and_retention/) +guide for a detailed walkthrough of this common use case. + +### Precalculating expensive queries + +Shorten query runtimes by pre-calculating expensive queries with CQs. +Use a CQ to automatically downsample commonly-queried, high precision data to a +lower precision. +Queries on lower precision data require fewer resources and return faster. + +**Tip:** Pre-calculate queries for your preferred graphing tool to accelerate +the population of graphs and dashboards. + +### Substituting for a `HAVING` clause + +InfluxQL does not support [`HAVING` clauses](https://en.wikipedia.org/wiki/Having_%28SQL%29). +Get the same functionality by creating a CQ to aggregate the data and querying +the CQ results to apply the `HAVING` clause. + +> **Note:** InfluxQL supports [subqueries](/enterprise_influxdb/v1.9/query_language/explore-data/#subqueries) which also offer similar functionality to `HAVING` clauses. +See [Data Exploration](/enterprise_influxdb/v1.9/query_language/explore-data/#subqueries) for more information. + +##### Example + +InfluxDB does not accept the following query with a `HAVING` clause. +The query calculates the average number of `bees` at `30` minute intervals and +requests averages that are greater than `20`. + +```sql +SELECT mean("bees") FROM "farm" GROUP BY time(30m) HAVING mean("bees") > 20 +``` + +To get the same results: + +**1. Create a CQ** + +This step performs the `mean("bees")` part of the query above. +Because this step creates CQ you only need to execute it once. + +The following CQ automatically calculates the average number of `bees` at +`30` minutes intervals and writes those averages to the `mean_bees` field in the +`aggregate_bees` measurement. + +```sql +CREATE CONTINUOUS QUERY "bee_cq" ON "mydb" BEGIN SELECT mean("bees") AS "mean_bees" INTO "aggregate_bees" FROM "farm" GROUP BY time(30m) END +``` + +**2. Query the CQ results** + +This step performs the `HAVING mean("bees") > 20` part of the query above. + +Query the data in the measurement `aggregate_bees` and request values of the `mean_bees` field that are greater than `20` in the `WHERE` clause: + +```sql +SELECT "mean_bees" FROM "aggregate_bees" WHERE "mean_bees" > 20 +``` + +### Substituting for nested functions + +Some InfluxQL functions +[support nesting](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#which-influxql-functions-support-nesting) +of other functions. +Most do not. +If your function does not support nesting, you can get the same functionality using a CQ to calculate +the inner-most function. +Then simply query the CQ results to calculate the outer-most function. + +> **Note:** InfluxQL supports [subqueries](/enterprise_influxdb/v1.9/query_language/explore-data/#subqueries) which also offer the same functionality as nested functions. +See [Data Exploration](/enterprise_influxdb/v1.9/query_language/explore-data/#subqueries) for more information. + +##### Example + +InfluxDB does not accept the following query with a nested function. +The query calculates the number of non-null values +of `bees` at `30` minute intervals and the average of those counts: + +```sql +SELECT mean(count("bees")) FROM "farm" GROUP BY time(30m) +``` + +To get the same results: + +**1. Create a CQ** + +This step performs the `count("bees")` part of the nested function above. +Because this step creates a CQ you only need to execute it once. + +The following CQ automatically calculates the number of non-null values of `bees` at `30` minute intervals +and writes those counts to the `count_bees` field in the `aggregate_bees` measurement. + +```sql +CREATE CONTINUOUS QUERY "bee_cq" ON "mydb" BEGIN SELECT count("bees") AS "count_bees" INTO "aggregate_bees" FROM "farm" GROUP BY time(30m) END +``` + +**2. Query the CQ results** + +This step performs the `mean([...])` part of the nested function above. + +Query the data in the measurement `aggregate_bees` to calculate the average of the +`count_bees` field: + +```sql +SELECT mean("count_bees") FROM "aggregate_bees" WHERE time >= AND time <= +``` + +## Further information + +To see how to combine two InfluxDB features, CQs, and retention policies, +to periodically downsample data and automatically expire the dispensable high +precision data, see [Downsampling and data retention](/enterprise_influxdb/v1.9/guides/downsampling_and_retention/). + +Kapacitor, InfluxData's data processing engine, can do the same work as +continuous queries in InfluxDB databases. + +To learn when to use Kapacitor instead of InfluxDB and how to perform the same CQ +functionality with a TICKscript, see [examples of continuous queries in Kapacitor](/{{< latest "kapacitor" >}}/guides/continuous_queries/). diff --git a/content/enterprise_influxdb/v1.9/query_language/explore-data.md b/content/enterprise_influxdb/v1.9/query_language/explore-data.md new file mode 100644 index 000000000..0a3478740 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/query_language/explore-data.md @@ -0,0 +1,3328 @@ +--- +title: Explore data using InfluxQL +description: > + Explore time series data using InfluxData's SQL-like query language. Understand how to use the SELECT statement to query data from measurements, tags, and fields. +menu: + enterprise_influxdb_1_9: + name: Explore data + weight: 20 + parent: InfluxQL +aliases: + - /enterprise_influxdb/v1.9/query_language/data_exploration/ +v2: /influxdb/v2.0/query-data/flux/query-fields/ +--- + +InfluxQL is an SQL-like query language for interacting with data in InfluxDB. +The following sections detail InfluxQL's `SELECT` statement and useful query syntax +for exploring your data. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
        The Basics:Configure Query Results:General Tips on Query Syntax:
        The SELECT statementORDER BY time DESCTime Syntax
        The WHERE clauseThe LIMIT and SLIMIT clausesRegular Expressions
        The GROUP BY clauseThe OFFSET and SOFFSET clausesData types and cast operations
        The INTO clauseThe Time Zone clauseMerge behavior
        Multiple statements
        Subqueries
        + +### Sample data + +This document uses publicly available data from the +[National Oceanic and Atmospheric Administration's (NOAA) Center for Operational Oceanographic Products and Services](http://tidesandcurrents.noaa.gov/stations.html?type=Water+Levels). +See the [Sample Data](/enterprise_influxdb/v1.9/query_language/data_download/) page to download +the data and follow along with the example queries in the sections below. + +Start by logging into the Influx CLI: + +```bash +$ influx -precision rfc3339 -database NOAA_water_database +Connected to http://localhost:8086 version 1.8.x +InfluxDB shell 1.8.x +> +``` + +Next, get acquainted with this subsample of the data in the `h2o_feet` measurement: + +name: h2o_feet + +| time | level description | location | water_level | +| ------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------ | +| 2015-08-18T00:00:00Z | between 6 and 9 feet | coyote_creek | 8.12 | +| 2015-08-18T00:00:00Z | below 3 feet | santa_monica | 2.064 | +| 2015-08-18T00:06:00Z | between 6 and 9 feet | coyote_creek | 8.005 | +| 2015-08-18T00:06:00Z | below 3 feet | santa_monica | 2.116 | +| 2015-08-18T00:12:00Z | between 6 and 9 feet | coyote_creek | 7.887 | +| 2015-08-18T00:12:00Z | below 3 feet | santa_monica | 2.028 | + +The data in the `h2o_feet` [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) +occur at six-minute time intervals. +The measurement has one [tag key](/enterprise_influxdb/v1.9/concepts/glossary/#tag-key) +(`location`) which has two [tag values](/enterprise_influxdb/v1.9/concepts/glossary/#tag-value): +`coyote_creek` and `santa_monica`. +The measurement also has two [fields](/enterprise_influxdb/v1.9/concepts/glossary/#field): +`level description` stores string [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value) +and `water_level` stores float field values. +All of these data is in the `NOAA_water_database` [database](/enterprise_influxdb/v1.9/concepts/glossary/#database). + +> **Disclaimer:** The `level description` field isn't part of the original NOAA data - we snuck it in there for the sake of having a field key with a special character and string field values. + +## The basic SELECT statement + +The `SELECT` statement queries data from a particular [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) or measurements. + +### Syntax + +```sql +SELECT [,,] FROM [,] +``` + +The `SELECT` statement requires a `SELECT` clause and a `FROM` clause. + +#### `SELECT` clause + +The `SELECT` clause supports several formats for specifying data: + +`SELECT *` +          Returns all [fields](/enterprise_influxdb/v1.9/concepts/glossary/#field) and [tags](/enterprise_influxdb/v1.9/concepts/glossary/#tag). + +`SELECT ""` +          Returns a specific field. + +`SELECT "",""` +          Returns more than one field. + +`SELECT "",""` +          Returns a specific field and tag. +The `SELECT` clause must specify at least one field when it includes a tag. + +`SELECT ""::field,""::tag` +          Returns a specific field and tag. +The `::[field | tag]` syntax specifies the [identifier's](/enterprise_influxdb/v1.9/concepts/glossary/#identifier) type. +Use this syntax to differentiate between field keys and tag keys that have the same name. + +Other supported features: +[Arithmetic operations](/enterprise_influxdb/v1.9/query_language/math_operators/), +[Functions](/enterprise_influxdb/v1.9/query_language/functions/), +[Basic cast operations](#data-types-and-cast-operations), +[Regular expressions](#regular-expressions) + +> **Note:** The SELECT statement cannot include an aggregate function **and** a non-aggregate function, field key, or tag key. For more information, see [error about mixing aggregate and non-aggregate queries](/enterprise_influxdb/v1.9/troubleshooting/errors/#error-parsing-query-mixing-aggregate-and-non-aggregate-queries-is-not-supported). + +#### `FROM` clause + +The `FROM` clause supports several formats for specifying a [measurement(s)](/enterprise_influxdb/v1.9/concepts/glossary/#measurement): + +`FROM ` +           +Returns data from a single measurement. +If you're using the [CLI](/enterprise_influxdb/v1.9/tools/use-influx/) InfluxDB queries the measurement in the +[`USE`d](/enterprise_influxdb/v1.9/tools/use-influx/#commands) +[database](/enterprise_influxdb/v1.9/concepts/glossary/#database) and the `DEFAULT` [retention policy](/enterprise_influxdb/v1.9/concepts/glossary/#retention-policy-rp). +If you're using the [InfluxDB API](/enterprise_influxdb/v1.9/tools/api/) InfluxDB queries the +measurement in the database specified in the [`db` query string parameter](/enterprise_influxdb/v1.9/tools/api/#query-string-parameters) +and the `DEFAULT` retention policy. + +`FROM ,` +           +Returns data from more than one measurement. + +`FROM ..` +           +Returns data from a fully qualified measurement. +Fully qualify a measurement by specifying its database and retention policy. + +`FROM ..` +           +Returns data from a measurement in a user-specified [database](/enterprise_influxdb/v1.9/concepts/glossary/#database) and the `DEFAULT` +[retention policy](/enterprise_influxdb/v1.9/concepts/glossary/#retention-policy-rp). + +Other supported features: +[Regular Expressions](#regular-expressions) + +#### Quoting + +[Identifiers](/enterprise_influxdb/v1.9/concepts/glossary/#identifier) **must** be double quoted if they contain characters other than `[A-z,0-9,_]`, if they +begin with a digit, or if they are an [InfluxQL keyword](https://github.com/influxdata/influxql/blob/master/README.md#keywords). +While not always necessary, we recommend that you double quote identifiers. + +> **Note:** The quoting syntax for queries differs from the [line protocol](/enterprise_influxdb/v1.9/concepts/glossary/#influxdb-line-protocol). +Please review the [rules for single and double-quoting](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#when-should-i-single-quote-and-when-should-i-double-quote-in-queries) in queries. + +### Examples + +#### Select all fields and tags from a single measurement + +```sql +> SELECT * FROM "h2o_feet" + +name: h2o_feet +-------------- +time level description location water_level +2015-08-18T00:00:00Z below 3 feet santa_monica 2.064 +2015-08-18T00:00:00Z between 6 and 9 feet coyote_creek 8.12 +[...] +2015-09-18T21:36:00Z between 3 and 6 feet santa_monica 5.066 +2015-09-18T21:42:00Z between 3 and 6 feet santa_monica 4.938 +``` + +The query selects all [fields](/enterprise_influxdb/v1.9/concepts/glossary/#field) and +[tags](/enterprise_influxdb/v1.9/concepts/glossary/#tag) from the `h2o_feet` +[measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +If you're using the [CLI](/enterprise_influxdb/v1.9/tools/use-influx/) be sure to enter +`USE NOAA_water_database` before you run the query. +The CLI queries the data in the `USE`d database and the +`DEFAULT` [retention policy](/enterprise_influxdb/v1.9/concepts/glossary/#retention-policy-rp). +If you're using the [InfluxDB API](/enterprise_influxdb/v1.9/tools/api/) be sure to set the +`db` [query string parameter](/enterprise_influxdb/v1.9/tools/api/#query-string-parameters) +to `NOAA_water_database`. +If you do not set the `rp` query string parameter, the InfluxDB API automatically +queries the database's `DEFAULT` retention policy. + +#### Select specific tags and fields from a single measurement + +```sql +> SELECT "level description","location","water_level" FROM "h2o_feet" + +name: h2o_feet +-------------- +time level description location water_level +2015-08-18T00:00:00Z below 3 feet santa_monica 2.064 +2015-08-18T00:00:00Z between 6 and 9 feet coyote_creek 8.12 +[...] +2015-09-18T21:36:00Z between 3 and 6 feet santa_monica 5.066 +2015-09-18T21:42:00Z between 3 and 6 feet santa_monica 4.938 +``` + +The query selects the `level description` field, the `location` tag, and the +`water_level` field. +Note that the `SELECT` clause must specify at least one field when it includes +a tag. + +#### Select specific tags and fields from a single measurement, and provide their identifier type + +```sql +> SELECT "level description"::field,"location"::tag,"water_level"::field FROM "h2o_feet" + +name: h2o_feet +-------------- +time level description location water_level +2015-08-18T00:00:00Z below 3 feet santa_monica 2.064 +2015-08-18T00:00:00Z between 6 and 9 feet coyote_creek 8.12 +[...] +2015-09-18T21:36:00Z between 3 and 6 feet santa_monica 5.066 +2015-09-18T21:42:00Z between 3 and 6 feet santa_monica 4.938 +``` + +The query selects the `level description` field, the `location` tag, and the +`water_level` field from the `h2o_feet` measurement. +The `::[field | tag]` syntax specifies if the +[identifier](/enterprise_influxdb/v1.9/concepts/glossary/#identifier) is a field or tag. +Use `::[field | tag]` to differentiate between [an identical field key and tag key ](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#how-do-i-query-data-with-an-identical-tag-key-and-field-key). +That syntax is not required for most use cases. + +#### Select all fields from a single measurement + +```sql +> SELECT *::field FROM "h2o_feet" + +name: h2o_feet +-------------- +time level description water_level +2015-08-18T00:00:00Z below 3 feet 2.064 +2015-08-18T00:00:00Z between 6 and 9 feet 8.12 +[...] +2015-09-18T21:36:00Z between 3 and 6 feet 5.066 +2015-09-18T21:42:00Z between 3 and 6 feet 4.938 +``` + +The query selects all fields from the `h2o_feet` measurement. +The `SELECT` clause supports combining the `*` syntax with the `::` syntax. + +#### Select a specific field from a measurement and perform basic arithmetic + +```sql +> SELECT ("water_level" * 2) + 4 FROM "h2o_feet" + +name: h2o_feet +-------------- +time water_level +2015-08-18T00:00:00Z 20.24 +2015-08-18T00:00:00Z 8.128 +[...] +2015-09-18T21:36:00Z 14.132 +2015-09-18T21:42:00Z 13.876 +``` + +The query multiplies `water_level`'s field values by two and adds four to those +values. +Note that InfluxDB follows the standard order of operations. +See [Mathematical Operators](/enterprise_influxdb/v1.9/query_language/math_operators/) +for more on supported operators. + +#### Select all data from more than one measurement + +```sql +> SELECT * FROM "h2o_feet","h2o_pH" + +name: h2o_feet +-------------- +time level description location pH water_level +2015-08-18T00:00:00Z below 3 feet santa_monica 2.064 +2015-08-18T00:00:00Z between 6 and 9 feet coyote_creek 8.12 +[...] +2015-09-18T21:36:00Z between 3 and 6 feet santa_monica 5.066 +2015-09-18T21:42:00Z between 3 and 6 feet santa_monica 4.938 + +name: h2o_pH +------------ +time level description location pH water_level +2015-08-18T00:00:00Z santa_monica 6 +2015-08-18T00:00:00Z coyote_creek 7 +[...] +2015-09-18T21:36:00Z santa_monica 8 +2015-09-18T21:42:00Z santa_monica 7 +``` + +The query selects all fields and tags from two measurements: `h2o_feet` and +`h2o_pH`. +Separate multiple measurements with a comma (`,`). + +#### Select all data from a fully qualified measurement + +```sql +> SELECT * FROM "NOAA_water_database"."autogen"."h2o_feet" + +name: h2o_feet +-------------- +time level description location water_level +2015-08-18T00:00:00Z below 3 feet santa_monica 2.064 +2015-08-18T00:00:00Z between 6 and 9 feet coyote_creek 8.12 +[...] +2015-09-18T21:36:00Z between 3 and 6 feet santa_monica 5.066 +2015-09-18T21:42:00Z between 3 and 6 feet santa_monica 4.938 +``` + +The query selects data in the `NOAA_water_database`, the `autogen` retention +policy, and the measurement `h2o_feet`. + +In the CLI, fully qualify a measurement to query data in a database other +than the `USE`d database and in a retention policy other than the +`DEFAULT` retention policy. +In the InfluxDB API, fully qualify a measurement in place of using the `db` +and `rp` query string parameters if desired. + +#### Select all data from a measurement in a particular database + +```sql +> SELECT * FROM "NOAA_water_database".."h2o_feet" + +name: h2o_feet +-------------- +time level description location water_level +2015-08-18T00:00:00Z below 3 feet santa_monica 2.064 +2015-08-18T00:00:00Z between 6 and 9 feet coyote_creek 8.12 +[...] +2015-09-18T21:36:00Z between 3 and 6 feet santa_monica 5.066 +2015-09-18T21:42:00Z between 3 and 6 feet santa_monica 4.938 +``` + +The query selects data in the `NOAA_water_database`, the `DEFAULT` retention +policy, and the `h2o_feet` measurement. +The `..` indicates the `DEFAULT` retention policy for the specified database. + +In the CLI, specify the database to query data in a database other than the +`USE`d database. +In the InfluxDB API, specify the database in place of using the `db` query +string parameter if desired. + +### Common issues with the SELECT statement + +#### Selecting tag keys in the SELECT clause + +A query requires at least one [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +in the `SELECT` clause to return data. +If the `SELECT` clause only includes a single [tag key](/enterprise_influxdb/v1.9/concepts/glossary/#tag-key) or several tag keys, the +query returns an empty response. +This behavior is a result of how the system stores data. + +##### Example + +The following query returns no data because it specifies a single tag key (`location`) in +the `SELECT` clause: + +```sql +> SELECT "location" FROM "h2o_feet" +> +``` + +To return any data associated with the `location` tag key, the query's `SELECT` +clause must include at least one field key (`water_level`): + +```sql +> SELECT "water_level","location" FROM "h2o_feet" +name: h2o_feet +time water_level location +---- ----------- -------- +2015-08-18T00:00:00Z 8.12 coyote_creek +2015-08-18T00:00:00Z 2.064 santa_monica +[...] +2015-09-18T21:36:00Z 5.066 santa_monica +2015-09-18T21:42:00Z 4.938 santa_monica +``` + +## The `WHERE` clause + +The `WHERE` filters data based on +[fields](/enterprise_influxdb/v1.9/concepts/glossary/#field), +[tags](/enterprise_influxdb/v1.9/concepts/glossary/#tag), and/or +[timestamps](/enterprise_influxdb/v1.9/concepts/glossary/#timestamp). + +Tired of reading? Check out this InfluxQL Short: +
        +
        + + +### Syntax + +``` +SELECT_clause FROM_clause WHERE [(AND|OR) [...]] +``` + +The `WHERE` clause supports `conditional_expression`s on fields, tags, and +timestamps. + +>**Note** InfluxDB does not support using OR in the WHERE clause to specify multiple time ranges. For example, InfluxDB returns an empty response for the following query: + +`> SELECT * FROM "absolutismus" WHERE time = '2016-07-31T20:07:00Z' OR time = '2016-07-31T23:07:17Z'` + +#### Fields + +``` +field_key ['string' | boolean | float | integer] +``` + +The `WHERE` clause supports comparisons against string, boolean, float, +and integer [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). + +Single quote string field values in the `WHERE` clause. +Queries with unquoted string field values or double quoted string field values +will not return any data and, in most cases, +[will not return an error](#common-issues-with-the-where-clause). + +##### Supported operators + +| Operator | Meaning | +|:--------:|:-------- | +| `=` | equal to | +| `<>` | not equal to | +| `!=` | not equal to | +| `>` | greater than | +| `>=` | greater than or equal to | +| `<` | less than | +| `<=` | less than or equal to | + +Other supported features: +[Arithmetic Operations](/enterprise_influxdb/v1.9/query_language/math_operators/), +[Regular Expressions](#regular-expressions) + +#### Tags + +```sql +tag_key ['tag_value'] +``` + +Single quote [tag values](/enterprise_influxdb/v1.9/concepts/glossary/#tag-value) in +the `WHERE` clause. +Queries with unquoted tag values or double quoted tag values will not return +any data and, in most cases, +[will not return an error](#common-issues-with-the-where-clause). + +##### Supported operators + +| Operator | Meaning | +|:--------:|:------- | +| `=` | equal to | +| `<>` | not equal to | +| `!=` | not equal to | + +Other supported features: +[Regular Expressions](#regular-expressions) + +#### Timestamps + +For most `SELECT` statements, the default time range is between [`1677-09-21 00:12:43.145224194` and `2262-04-11T23:47:16.854775806Z` UTC](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#what-are-the-minimum-and-maximum-timestamps-that-influxdb-can-store). +For `SELECT` statements with a [`GROUP BY time()` clause](#group-by-time-intervals), the default time +range is between `1677-09-21 00:12:43.145224194` UTC and [`now()`](/enterprise_influxdb/v1.9/concepts/glossary/#now). + +The [Time Syntax](#time-syntax) section on this page +details how to specify alternative time ranges in the `WHERE` clause. + +### Examples + +#### Select data that have specific field key-values + +```sql +> SELECT * FROM "h2o_feet" WHERE "water_level" > 8 + +name: h2o_feet +-------------- +time level description location water_level +2015-08-18T00:00:00Z between 6 and 9 feet coyote_creek 8.12 +2015-08-18T00:06:00Z between 6 and 9 feet coyote_creek 8.005 +[...] +2015-09-18T00:12:00Z between 6 and 9 feet coyote_creek 8.189 +2015-09-18T00:18:00Z between 6 and 9 feet coyote_creek 8.084 +``` + +The query returns data from the `h2o_feet` +[measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) with +[field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value) of `water_level` +that are greater than eight. + +#### Select data that have a specific string field key-value + +```sql +> SELECT * FROM "h2o_feet" WHERE "level description" = 'below 3 feet' + +name: h2o_feet +-------------- +time level description location water_level +2015-08-18T00:00:00Z below 3 feet santa_monica 2.064 +2015-08-18T00:06:00Z below 3 feet santa_monica 2.116 +[...] +2015-09-18T14:06:00Z below 3 feet santa_monica 2.999 +2015-09-18T14:36:00Z below 3 feet santa_monica 2.907 +``` + +The query returns data from the `h2o_feet` measurement with field values of +`level description` that equal the `below 3 feet` string. +InfluxQL requires single quotes around string field values in the `WHERE` +clause. + +#### Select data that have a specific field key-value and perform basic arithmetic + +```sql +> SELECT * FROM "h2o_feet" WHERE "water_level" + 2 > 11.9 + +name: h2o_feet +-------------- +time level description location water_level +2015-08-29T07:06:00Z at or greater than 9 feet coyote_creek 9.902 +2015-08-29T07:12:00Z at or greater than 9 feet coyote_creek 9.938 +2015-08-29T07:18:00Z at or greater than 9 feet coyote_creek 9.957 +2015-08-29T07:24:00Z at or greater than 9 feet coyote_creek 9.964 +2015-08-29T07:30:00Z at or greater than 9 feet coyote_creek 9.954 +2015-08-29T07:36:00Z at or greater than 9 feet coyote_creek 9.941 +2015-08-29T07:42:00Z at or greater than 9 feet coyote_creek 9.925 +2015-08-29T07:48:00Z at or greater than 9 feet coyote_creek 9.902 +2015-09-02T23:30:00Z at or greater than 9 feet coyote_creek 9.902 +``` + +The query returns data from the `h2o_feet` measurement with field values of +`water_level` plus two that are greater than 11.9. +Note that InfluxDB follows the standard order of operations +See [Mathematical Operators](/enterprise_influxdb/v1.9/query_language/math_operators/) +for more on supported operators. + +#### Select data that have a specific tag key-value + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE "location" = 'santa_monica' + +name: h2o_feet +-------------- +time water_level +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +[...] +2015-09-18T21:36:00Z 5.066 +2015-09-18T21:42:00Z 4.938 +``` + +The query returns data from the `h2o_feet` measurement where the +[tag key](/enterprise_influxdb/v1.9/concepts/glossary/#tag-key) `location` is set to `santa_monica`. +InfluxQL requires single quotes around tag values in the `WHERE` clause. + +#### Select data that have specific field key-values and tag key-values + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE "location" <> 'santa_monica' AND (water_level < -0.59 OR water_level > 9.95) + +name: h2o_feet +-------------- +time water_level +2015-08-29T07:18:00Z 9.957 +2015-08-29T07:24:00Z 9.964 +2015-08-29T07:30:00Z 9.954 +2015-08-29T14:30:00Z -0.61 +2015-08-29T14:36:00Z -0.591 +2015-08-30T15:18:00Z -0.594 +``` + +The query returns data from the `h2o_feet` measurement where the tag key +`location` is not set to `santa_monica` and where the field values of +`water_level` are either less than -0.59 or greater than 9.95. +The `WHERE` clause supports the operators `AND` and `OR`, and supports +separating logic with parentheses. + +#### Select data that have specific timestamps + +```sql +> SELECT * FROM "h2o_feet" WHERE time > now() - 7d +``` + +The query returns data from the `h2o_feet` measurement that have [timestamps](/enterprise_influxdb/v1.9/concepts/glossary/#timestamp) +within the past seven days. +The [Time Syntax](#time-syntax) section on this page +offers in-depth information on supported time syntax in the `WHERE` clause. + +### Common issues with the `WHERE` clause + +#### A `WHERE` clause query unexpectedly returns no data + +In most cases, this issue is the result of missing single quotes around +[tag values](/enterprise_influxdb/v1.9/concepts/glossary/#tag-value) +or string [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). +Queries with unquoted or double quoted tag values or string field values will +not return any data and, in most cases, will not return an error. + +The first two queries in the code block below attempt to specify the tag value +`santa_monica` without any quotes and with double quotes. +Those queries return no results. +The third query single quotes `santa_monica` (this is the supported syntax) +and returns the expected results. + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE "location" = santa_monica + +> SELECT "water_level" FROM "h2o_feet" WHERE "location" = "santa_monica" + +> SELECT "water_level" FROM "h2o_feet" WHERE "location" = 'santa_monica' + +name: h2o_feet +-------------- +time water_level +2015-08-18T00:00:00Z 2.064 +[...] +2015-09-18T21:42:00Z 4.938 +``` + +The first two queries in the code block below attempt to specify the string +field value `at or greater than 9 feet` without any quotes and with double +quotes. +The first query returns an error because the string field value includes +white spaces. +The second query returns no results. +The third query single quotes `at or greater than 9 feet` (this is the +supported syntax) and returns the expected results. + +```sql +> SELECT "level description" FROM "h2o_feet" WHERE "level description" = at or greater than 9 feet + +ERR: error parsing query: found than, expected ; at line 1, char 86 + +> SELECT "level description" FROM "h2o_feet" WHERE "level description" = "at or greater than 9 feet" + +> SELECT "level description" FROM "h2o_feet" WHERE "level description" = 'at or greater than 9 feet' + +name: h2o_feet +-------------- +time level description +2015-08-26T04:00:00Z at or greater than 9 feet +[...] +2015-09-15T22:42:00Z at or greater than 9 feet +``` + +# The GROUP BY clause + +The `GROUP BY` clause groups query results by: + +- one or more specified [tags](/enterprise_influxdb/v1.9/concepts/glossary/#tag) +- specified time interval + +>**Note:** You cannot use `GROUP BY` to group fields. + + + + + + + + + + + + + + +
        GROUP BY tags +
        GROUP BY time intervals: + Basic SyntaxAdvanced SyntaxGROUP BY time intervals and fill()
        + +## GROUP BY tags + +`GROUP BY ` groups query results by one or more specified [tags](/enterprise_influxdb/v1.9/concepts/glossary/#tag). + +Watch InfluxQL short about `GROUP BY` with tags: +
        +
        + + +#### Syntax + +```sql +SELECT_clause FROM_clause [WHERE_clause] GROUP BY [* | [,` +   Groups results by a specific tag + +`GROUP BY ,` +   Groups results by more than one tag. +The order of the [tag keys](/enterprise_influxdb/v1.9/concepts/glossary/#tag-key) is irrelevant. + +If the query includes a [`WHERE` clause](#the-where-clause) the `GROUP BY` +clause must appear after the `WHERE` clause. + +Other supported features: [Regular Expressions](#regular-expressions) + +#### Examples + +##### Group query results by a single tag + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" GROUP BY "location" + +name: h2o_feet +tags: location=coyote_creek +time mean +---- ---- +1970-01-01T00:00:00Z 5.359342451341401 + + +name: h2o_feet +tags: location=santa_monica +time mean +---- ---- +1970-01-01T00:00:00Z 3.530863470081006 +``` + +The query uses an InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions/) +to calculate the average `water_level` for each +[tag value](/enterprise_influxdb/v1.9/concepts/glossary/#tag-value) of `location` in +the `h2o_feet` [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). +InfluxDB returns results in two [series](/enterprise_influxdb/v1.9/concepts/glossary/#series): one for each tag value of `location`. + +>**Note:** In InfluxDB, [epoch 0](https://en.wikipedia.org/wiki/Unix_time) (`1970-01-01T00:00:00Z`) is often used as a null timestamp equivalent. +If you request a query that has no timestamp to return, such as an [aggregation function](/enterprise_influxdb/v1.9/query_language/functions/) with an unbounded time range, InfluxDB returns epoch 0 as the timestamp. + +##### Group query results by more than one tag + +```sql +> SELECT MEAN("index") FROM "h2o_quality" GROUP BY "location","randtag" + +name: h2o_quality +tags: location=coyote_creek, randtag=1 +time mean +---- ---- +1970-01-01T00:00:00Z 50.69033760186263 + +name: h2o_quality +tags: location=coyote_creek, randtag=2 +time mean +---- ---- +1970-01-01T00:00:00Z 49.661867544220485 + +name: h2o_quality +tags: location=coyote_creek, randtag=3 +time mean +---- ---- +1970-01-01T00:00:00Z 49.360939907550076 + +name: h2o_quality +tags: location=santa_monica, randtag=1 +time mean +---- ---- +1970-01-01T00:00:00Z 49.132712456344585 + +name: h2o_quality +tags: location=santa_monica, randtag=2 +time mean +---- ---- +1970-01-01T00:00:00Z 50.2937984496124 + +name: h2o_quality +tags: location=santa_monica, randtag=3 +time mean +---- ---- +1970-01-01T00:00:00Z 49.99919903884662 +``` + +The query uses an InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions/) to calculate the average `index` for +each combination of the `location` [tag](/enterprise_influxdb/v1.9/concepts/glossary/#tag) and the `randtag` tag in the +`h2o_quality` measurement. +Separate multiple tags with a comma in the `GROUP BY` clause. + +##### Group query results by all tags + +```sql +> SELECT MEAN("index") FROM "h2o_quality" GROUP BY * + +name: h2o_quality +tags: location=coyote_creek, randtag=1 +time mean +---- ---- +1970-01-01T00:00:00Z 50.55405446521169 + + +name: h2o_quality +tags: location=coyote_creek, randtag=2 +time mean +---- ---- +1970-01-01T00:00:00Z 50.49958856271162 + + +name: h2o_quality +tags: location=coyote_creek, randtag=3 +time mean +---- ---- +1970-01-01T00:00:00Z 49.5164137518956 + + +name: h2o_quality +tags: location=santa_monica, randtag=1 +time mean +---- ---- +1970-01-01T00:00:00Z 50.43829082296367 + + +name: h2o_quality +tags: location=santa_monica, randtag=2 +time mean +---- ---- +1970-01-01T00:00:00Z 52.0688508894012 + + +name: h2o_quality +tags: location=santa_monica, randtag=3 +time mean +---- ---- +1970-01-01T00:00:00Z 49.29386362086556 +``` + +The query uses an InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions/) +to calculate the average `index` for every possible +[tag](/enterprise_influxdb/v1.9/concepts/glossary/#tag) combination in the `h2o_quality` +measurement. + +Note that the query results are identical to the results of the query in [Example 2](#examples-2) +where we explicitly specified the `location` and `randtag` tag keys. +This is because the `h2o_quality` measurement only has two tag keys. + +## GROUP BY time intervals + +`GROUP BY time()` queries group query results by a user-specified time interval. + +### Basic GROUP BY time() syntax + +#### Syntax + +```sql +SELECT () FROM_clause WHERE GROUP BY time(),[tag_key] [fill()] +``` + +Basic `GROUP BY time()` queries require an InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions/) +in the [`SELECT` clause](#the-basic-select-statement) and a time range in the +[`WHERE` clause](#the-where-clause). +Note that the `GROUP BY` clause must come after the `WHERE` clause. + +##### `time(time_interval)` + +The `time_interval` in the `GROUP BY time()` clause is a +[duration literal](/enterprise_influxdb/v1.9/query_language/spec/#durations). +It determines how InfluxDB groups query results over time. +For example, a `time_interval` of `5m` groups query results into five-minute +time groups across the time range specified in the [`WHERE` clause](#the-where-clause). + +##### `fill()` + +`fill()` is optional. +It changes the value reported for time intervals that have no data. +See [GROUP BY time intervals and `fill()`](#group-by-time-intervals-and-fill) +for more information. + +**Coverage:** + +Basic `GROUP BY time()` queries rely on the `time_interval` and on the InfluxDB database's +preset time boundaries to determine the raw data included in each time interval +and the timestamps returned by the query. + +#### Examples of basic syntax + +The examples below use the following subsample of the sample data: + +```sql +> SELECT "water_level","location" FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' + +name: h2o_feet +-------------- +time water_level location +2015-08-18T00:00:00Z 8.12 coyote_creek +2015-08-18T00:00:00Z 2.064 santa_monica +2015-08-18T00:06:00Z 8.005 coyote_creek +2015-08-18T00:06:00Z 2.116 santa_monica +2015-08-18T00:12:00Z 7.887 coyote_creek +2015-08-18T00:12:00Z 2.028 santa_monica +2015-08-18T00:18:00Z 7.762 coyote_creek +2015-08-18T00:18:00Z 2.126 santa_monica +2015-08-18T00:24:00Z 7.635 coyote_creek +2015-08-18T00:24:00Z 2.041 santa_monica +2015-08-18T00:30:00Z 7.5 coyote_creek +2015-08-18T00:30:00Z 2.051 santa_monica +``` + +##### Group query results into 12 minute intervals + +```sql +> SELECT COUNT("water_level") FROM "h2o_feet" WHERE "location"='coyote_creek' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' GROUP BY time(12m) + +name: h2o_feet +-------------- +time count +2015-08-18T00:00:00Z 2 +2015-08-18T00:12:00Z 2 +2015-08-18T00:24:00Z 2 +``` + +The query uses an InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions/) +to count the number of `water_level` points with the [tag](/enterprise_influxdb/v1.9/concepts/glossary/#tag) +`location = coyote_creek` and it group results into 12 minute intervals. + +The result for each [timestamp](/enterprise_influxdb/v1.9/concepts/glossary/#timestamp) +represents a single 12 minute interval. +The count for the first timestamp covers the raw data between `2015-08-18T00:00:00Z` +and up to, but not including, `2015-08-18T00:12:00Z`. +The count for the second timestamp covers the raw data between `2015-08-18T00:12:00Z` +and up to, but not including, `2015-08-18T00:24:00Z.` + +##### Group query results into 12 minutes intervals and by a tag key + +```sql +> SELECT COUNT("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' GROUP BY time(12m),"location" + +name: h2o_feet +tags: location=coyote_creek +time count +---- ----- +2015-08-18T00:00:00Z 2 +2015-08-18T00:12:00Z 2 +2015-08-18T00:24:00Z 2 + +name: h2o_feet +tags: location=santa_monica +time count +---- ----- +2015-08-18T00:00:00Z 2 +2015-08-18T00:12:00Z 2 +2015-08-18T00:24:00Z 2 +``` + +The query uses an InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions/) +to count the number of `water_level` points. +It groups results by the `location` tag and into 12 minute intervals. +Note that the time interval and the tag key are separated by a comma in the +`GROUP BY` clause. + +The query returns two [series](/enterprise_influxdb/v1.9/concepts/glossary/#series) of results: one for each +[tag value](/enterprise_influxdb/v1.9/concepts/glossary/#tag-value) of the `location` tag. +The result for each timestamp represents a single 12 minute interval. +The count for the first timestamp covers the raw data between `2015-08-18T00:00:00Z` +and up to, but not including, `2015-08-18T00:12:00Z`. +The count for the second timestamp covers the raw data between `2015-08-18T00:12:00Z` +and up to, but not including, `2015-08-18T00:24:00Z.` + +#### Common issues with basic syntax + +##### Unexpected timestamps and values in query results + +With the basic syntax, InfluxDB relies on the `GROUP BY time()` interval +and on the system's preset time boundaries to determine the raw data included +in each time interval and the timestamps returned by the query. +In some cases, this can lead to unexpected results. + +**Example** + +Raw data: + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE "location"='coyote_creek' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:18:00Z' +name: h2o_feet +-------------- +time water_level +2015-08-18T00:00:00Z 8.12 +2015-08-18T00:06:00Z 8.005 +2015-08-18T00:12:00Z 7.887 +2015-08-18T00:18:00Z 7.762 +``` + +Query and results: + +The following query covers a 12-minute time range and groups results into 12-minute time intervals, but it returns **two** results: + +```sql +> SELECT COUNT("water_level") FROM "h2o_feet" WHERE "location"='coyote_creek' AND time >= '2015-08-18T00:06:00Z' AND time < '2015-08-18T00:18:00Z' GROUP BY time(12m) + +name: h2o_feet +time count +---- ----- +2015-08-18T00:00:00Z 1 <----- Note that this timestamp occurs before the start of the query's time range +2015-08-18T00:12:00Z 1 +``` + +Explanation: + +InfluxDB uses preset round-number time boundaries for `GROUP BY` intervals that are +independent of any time conditions in the `WHERE` clause. +When it calculates the results, all returned data must occur within the query's +explicit time range but the `GROUP BY` intervals will be based on the preset +time boundaries. + +The table below shows the preset time boundary, the relevant `GROUP BY time()` interval, the +points included, and the returned timestamp for each `GROUP BY time()` +interval in the results. + +| Time Interval Number | Preset Time Boundary |`GROUP BY time()` Interval | Points Included | Returned Timestamp | +| :------------- | :------------- | :------------- | :------------- | :------------- | +| 1 | `time >= 2015-08-18T00:00:00Z AND time < 2015-08-18T00:12:00Z` | `time >= 2015-08-18T00:06:00Z AND time < 2015-08-18T00:12:00Z` | `8.005` | `2015-08-18T00:00:00Z` | +| 2 | `time >= 2015-08-12T00:12:00Z AND time < 2015-08-18T00:24:00Z` | `time >= 2015-08-12T00:12:00Z AND time < 2015-08-18T00:18:00Z` | `7.887` | `2015-08-18T00:12:00Z` | + +The first preset 12-minute time boundary begins at `00:00` and ends just before +`00:12`. +Only one raw point (`8.005`) falls both within the query's first `GROUP BY time()` interval and in that +first time boundary. +Note that while the returned timestamp occurs before the start of the query's time range, +the query result excludes data that occur before the query's time range. + +The second preset 12-minute time boundary begins at `00:12` and ends just before +`00:24`. +Only one raw point (`7.887`) falls both within the query's second `GROUP BY time()` interval and in that +second time boundary. + +The [advanced `GROUP BY time()` syntax](#advanced-group-by-time-syntax) allows users to shift +the start time of the InfluxDB database's preset time boundaries. +[Example 3](#examples-3) +in the Advanced Syntax section continues with the query shown here; +it shifts forward the preset time boundaries by six minutes such that +InfluxDB returns: + +```sql +name: h2o_feet +time count +---- ----- +2015-08-18T00:06:00Z 2 +``` + +### Advanced GROUP BY time() syntax + +#### Syntax + +```sql +SELECT () FROM_clause WHERE GROUP BY time(,),[tag_key] [fill()] +``` + +Advanced `GROUP BY time()` queries require an InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions/) +in the [`SELECT` clause](#the-basic-select-statement) and a time range in the +[`WHERE` clause](#the-where-clause). +Note that the `GROUP BY` clause must come after the `WHERE` clause. + +##### `time(time_interval,offset_interval)` + +See the [Basic GROUP BY time() Syntax](#basic-group-by-time-syntax) +for details on the `time_interval`. + +The `offset_interval` is a +[duration literal](/enterprise_influxdb/v1.9/query_language/spec/#durations). +It shifts forward or back the InfluxDB database's preset time boundaries. +The `offset_interval` can be positive or negative. + +##### `fill()` + +`fill()` is optional. +It changes the value reported for time intervals that have no data. +See [GROUP BY time intervals and `fill()`](#group-by-time-intervals-and-fill) +for more information. + +**Coverage:** + +Advanced `GROUP BY time()` queries rely on the `time_interval`, the `offset_interval` +, and on the InfluxDB database's preset time boundaries to determine the raw data included in each time interval +and the timestamps returned by the query. + +#### Examples of advanced syntax + +The examples below use the following subsample of the sample data: + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE "location"='coyote_creek' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:54:00Z' + +name: h2o_feet +-------------- +time water_level +2015-08-18T00:00:00Z 8.12 +2015-08-18T00:06:00Z 8.005 +2015-08-18T00:12:00Z 7.887 +2015-08-18T00:18:00Z 7.762 +2015-08-18T00:24:00Z 7.635 +2015-08-18T00:30:00Z 7.5 +2015-08-18T00:36:00Z 7.372 +2015-08-18T00:42:00Z 7.234 +2015-08-18T00:48:00Z 7.11 +2015-08-18T00:54:00Z 6.982 +``` + +##### Group query results into 18 minute intervals and shift the preset time boundaries forward + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE "location"='coyote_creek' AND time >= '2015-08-18T00:06:00Z' AND time <= '2015-08-18T00:54:00Z' GROUP BY time(18m,6m) + +name: h2o_feet +time mean +---- ---- +2015-08-18T00:06:00Z 7.884666666666667 +2015-08-18T00:24:00Z 7.502333333333333 +2015-08-18T00:42:00Z 7.108666666666667 +``` + +The query uses an InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions/) +to calculate the average `water_level`, grouping results into 18 minute +time intervals, and offsetting the preset time boundaries by six minutes. + +The time boundaries and returned timestamps for the query **without** the `offset_interval` adhere to the InfluxDB database's preset time boundaries. Let's first examine the results without the offset: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE "location"='coyote_creek' AND time >= '2015-08-18T00:06:00Z' AND time <= '2015-08-18T00:54:00Z' GROUP BY time(18m) + +name: h2o_feet +time mean +---- ---- +2015-08-18T00:00:00Z 7.946 +2015-08-18T00:18:00Z 7.6323333333333325 +2015-08-18T00:36:00Z 7.238666666666667 +2015-08-18T00:54:00Z 6.982 +``` + +The time boundaries and returned timestamps for the query **without** the +`offset_interval` adhere to the InfluxDB database's preset time boundaries: + +| Time Interval Number | Preset Time Boundary |`GROUP BY time()` Interval | Points Included | Returned Timestamp | +| :------------- | :------------- | :------------- | :------------- | :------------- | +| 1 | `time >= 2015-08-18T00:00:00Z AND time < 2015-08-18T00:18:00Z` | `time >= 2015-08-18T00:06:00Z AND time < 2015-08-18T00:18:00Z` | `8.005`,`7.887` | `2015-08-18T00:00:00Z` | +| 2 | `time >= 2015-08-18T00:18:00Z AND time < 2015-08-18T00:36:00Z` | <--- same | `7.762`,`7.635`,`7.5` | `2015-08-18T00:18:00Z` | +| 3 | `time >= 2015-08-18T00:36:00Z AND time < 2015-08-18T00:54:00Z` | <--- same | `7.372`,`7.234`,`7.11` | `2015-08-18T00:36:00Z` | +| 4 | `time >= 2015-08-18T00:54:00Z AND time < 2015-08-18T01:12:00Z` | `time = 2015-08-18T00:54:00Z` | `6.982` | `2015-08-18T00:54:00Z` | + +The first preset 18-minute time boundary begins at `00:00` and ends just before +`00:18`. +Two raw points (`8.005` and `7.887`) fall both within the first `GROUP BY time()` interval and in that +first time boundary. +Note that while the returned timestamp occurs before the start of the query's time range, +the query result excludes data that occur before the query's time range. + +The second preset 18-minute time boundary begins at `00:18` and ends just before +`00:36`. +Three raw points (`7.762` and `7.635` and `7.5`) fall both within the second `GROUP BY time()` interval and in that +second time boundary. In this case, the boundary time range and the interval's time range are the same. + +The fourth preset 18-minute time boundary begins at `00:54` and ends just before +`1:12:00`. +One raw point (`6.982`) falls both within the fourth `GROUP BY time()` interval and in that +fourth time boundary. + +The time boundaries and returned timestamps for the query **with** the +`offset_interval` adhere to the offset time boundaries: + +| Time Interval Number | Offset Time Boundary |`GROUP BY time()` Interval | Points Included | Returned Timestamp | +| :------------- | :------------- | :------------- | :------------- | ------------- | +| 1 | `time >= 2015-08-18T00:06:00Z AND time < 2015-08-18T00:24:00Z` | <--- same | `8.005`,`7.887`,`7.762` | `2015-08-18T00:06:00Z` | +| 2 | `time >= 2015-08-18T00:24:00Z AND time < 2015-08-18T00:42:00Z` | <--- same | `7.635`,`7.5`,`7.372` | `2015-08-18T00:24:00Z` | +| 3 | `time >= 2015-08-18T00:42:00Z AND time < 2015-08-18T01:00:00Z` | <--- same | `7.234`,`7.11`,`6.982` | `2015-08-18T00:42:00Z` | +| 4 | `time >= 2015-08-18T01:00:00Z AND time < 2015-08-18T01:18:00Z` | NA | NA | NA | + +The six-minute offset interval shifts forward the preset boundary's time range +such that the boundary time ranges and the relevant `GROUP BY time()` interval time ranges are +always the same. +With the offset, each interval performs the calculation on three points, and +the timestamp returned matches both the start of the boundary time range and the +start of the `GROUP BY time()` interval time range. + +Note that `offset_interval` forces the fourth time boundary to be outside +the query's time range so the query returns no results for that last interval. + +##### Group query results into 12 minute intervals and shift the preset time boundaries back + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE "location"='coyote_creek' AND time >= '2015-08-18T00:06:00Z' AND time <= '2015-08-18T00:54:00Z' GROUP BY time(18m,-12m) + +name: h2o_feet +time mean +---- ---- +2015-08-18T00:06:00Z 7.884666666666667 +2015-08-18T00:24:00Z 7.502333333333333 +2015-08-18T00:42:00Z 7.108666666666667 +``` + +The query uses an InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions/) +to calculate the average `water_level`, grouping results into 18 minute +time intervals, and offsetting the preset time boundaries by -12 minutes. + +> **Note:** The query in Example 2 returns the same results as the query in Example 1, but +the query in Example 2 uses a negative `offset_interval` instead of a positive +`offset_interval`. +> There are no performance differences between the two queries; feel free to choose the most +intuitive option when deciding between a positive and negative `offset_interval`. + +The time boundaries and returned timestamps for the query **without** the `offset_interval` adhere to InfluxDB database's preset time boundaries. Let's first examine the results without the offset: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE "location"='coyote_creek' AND time >= '2015-08-18T00:06:00Z' AND time <= '2015-08-18T00:54:00Z' GROUP BY time(18m) + +name: h2o_feet +time mean +---- ---- +2015-08-18T00:00:00Z 7.946 +2015-08-18T00:18:00Z 7.6323333333333325 +2015-08-18T00:36:00Z 7.238666666666667 +2015-08-18T00:54:00Z 6.982 +``` + +The time boundaries and returned timestamps for the query **without** the +`offset_interval` adhere to the InfluxDB database's preset time boundaries: + +| Time Interval Number | Preset Time Boundary |`GROUP BY time()` Interval | Points Included | Returned Timestamp | +| :------------- | :------------- | :------------- | :------------- | :------------- | +| 1 | `time >= 2015-08-18T00:00:00Z AND time < 2015-08-18T00:18:00Z` | `time >= 2015-08-18T00:06:00Z AND time < 2015-08-18T00:18:00Z` | `8.005`,`7.887` | `2015-08-18T00:00:00Z` | +| 2 | `time >= 2015-08-18T00:18:00Z AND time < 2015-08-18T00:36:00Z` | <--- same | `7.762`,`7.635`,`7.5` | `2015-08-18T00:18:00Z` | +| 3 | `time >= 2015-08-18T00:36:00Z AND time < 2015-08-18T00:54:00Z` | <--- same | `7.372`,`7.234`,`7.11` | `2015-08-18T00:36:00Z` | +| 4 | `time >= 2015-08-18T00:54:00Z AND time < 2015-08-18T01:12:00Z` | `time = 2015-08-18T00:54:00Z` | `6.982` | `2015-08-18T00:54:00Z` | + +The first preset 18-minute time boundary begins at `00:00` and ends just before +`00:18`. +Two raw points (`8.005` and `7.887`) fall both within the first `GROUP BY time()` interval and in that +first time boundary. +Note that while the returned timestamp occurs before the start of the query's time range, +the query result excludes data that occur before the query's time range. + +The second preset 18-minute time boundary begins at `00:18` and ends just before +`00:36`. +Three raw points (`7.762` and `7.635` and `7.5`) fall both within the second `GROUP BY time()` interval and in that +second time boundary. In this case, the boundary time range and the interval's time range are the same. + +The fourth preset 18-minute time boundary begins at `00:54` and ends just before +`1:12:00`. +One raw point (`6.982`) falls both within the fourth `GROUP BY time()` interval and in that +fourth time boundary. + +The time boundaries and returned timestamps for the query **with** the +`offset_interval` adhere to the offset time boundaries: + +| Time Interval Number | Offset Time Boundary |`GROUP BY time()` Interval | Points Included | Returned Timestamp | +| :------------- | :------------- | :------------- | :------------- | ------------- | +| 1 | `time >= 2015-08-17T23:48:00Z AND time < 2015-08-18T00:06:00Z` | NA | NA | NA | +| 2 | `time >= 2015-08-18T00:06:00Z AND time < 2015-08-18T00:24:00Z` | <--- same | `8.005`,`7.887`,`7.762` | `2015-08-18T00:06:00Z` | +| 3 | `time >= 2015-08-18T00:24:00Z AND time < 2015-08-18T00:42:00Z` | <--- same | `7.635`,`7.5`,`7.372` | `2015-08-18T00:24:00Z` | +| 4 | `time >= 2015-08-18T00:42:00Z AND time < 2015-08-18T01:00:00Z` | <--- same | `7.234`,`7.11`,`6.982` | `2015-08-18T00:42:00Z` | + +The negative 12-minute offset interval shifts back the preset boundary's time range +such that the boundary time ranges and the relevant `GROUP BY time()` interval time ranges are always the +same. +With the offset, each interval performs the calculation on three points, and +the timestamp returned matches both the start of the boundary time range and the +start of the `GROUP BY time()` interval time range. + +Note that `offset_interval` forces the first time boundary to be outside +the query's time range so the query returns no results for that first interval. + +##### Group query results into 12 minute intervals and shift the preset time boundaries forward + +This example is a continuation of the scenario outlined in [Common Issues with Basic Syntax](#common-issues-with-basic-syntax). + +```sql +> SELECT COUNT("water_level") FROM "h2o_feet" WHERE "location"='coyote_creek' AND time >= '2015-08-18T00:06:00Z' AND time < '2015-08-18T00:18:00Z' GROUP BY time(12m,6m) + +name: h2o_feet +time count +---- ----- +2015-08-18T00:06:00Z 2 +``` + +The query uses an InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions/) +to count the number of `water_level` points, grouping results into 12 minute +time intervals, and offsetting the preset time boundaries by six minutes. + +The time boundaries and returned timestamps for the query **without** the `offset_interval` adhere to InfluxDB database's preset time boundaries. Let's first examine the results without the offset: + +```sql +> SELECT COUNT("water_level") FROM "h2o_feet" WHERE "location"='coyote_creek' AND time >= '2015-08-18T00:06:00Z' AND time < '2015-08-18T00:18:00Z' GROUP BY time(12m) + +name: h2o_feet +time count +---- ----- +2015-08-18T00:00:00Z 1 +2015-08-18T00:12:00Z 1 +``` + +The time boundaries and returned timestamps for the query **without** the +`offset_interval` adhere to InfluxDB database's preset time boundaries: + +| Time Interval Number | Preset Time Boundary |`GROUP BY time()` Interval | Points Included | Returned Timestamp | +| :------------- | :------------- | :------------- | :------------- | :------------- | +| 1 | `time >= 2015-08-18T00:00:00Z AND time < 2015-08-18T00:12:00Z` | `time >= 2015-08-18T00:06:00Z AND time < 2015-08-18T00:12:00Z` | `8.005` | `2015-08-18T00:00:00Z` | +| 2 | `time >= 2015-08-12T00:12:00Z AND time < 2015-08-18T00:24:00Z` | `time >= 2015-08-12T00:12:00Z AND time < 2015-08-18T00:18:00Z` | `7.887` | `2015-08-18T00:12:00Z` | + +The first preset 12-minute time boundary begins at `00:00` and ends just before +`00:12`. +Only one raw point (`8.005`) falls both within the query's first `GROUP BY time()` interval and in that +first time boundary. +Note that while the returned timestamp occurs before the start of the query's time range, +the query result excludes data that occur before the query's time range. + +The second preset 12-minute time boundary begins at `00:12` and ends just before +`00:24`. +Only one raw point (`7.887`) falls both within the query's second `GROUP BY time()` interval and in that +second time boundary. + +The time boundaries and returned timestamps for the query **with** the +`offset_interval` adhere to the offset time boundaries: + +| Time Interval Number | Offset Time Boundary |`GROUP BY time()` Interval | Points Included | Returned Timestamp | +| :------------- | :------------- | :------------- | :------------- | :------------- | +| 1 | `time >= 2015-08-18T00:06:00Z AND time < 2015-08-18T00:18:00Z` | <--- same | `8.005`,`7.887` | `2015-08-18T00:06:00Z` | +| 2 | `time >= 2015-08-18T00:18:00Z AND time < 2015-08-18T00:30:00Z` | NA | NA | NA | + +The six-minute offset interval shifts forward the preset boundary's time range +such that the preset boundary time range and the relevant `GROUP BY time()` interval time range are the +same. +With the offset, the query returns a single result, and the timestamp returned +matches both the start of the boundary time range and the start of the `GROUP BY time()` interval +time range. + +Note that `offset_interval` forces the second time boundary to be outside +the query's time range so the query returns no results for that second interval. + +## `GROUP BY` time intervals and `fill()` + +`fill()` changes the value reported for time intervals that have no data. + +#### Syntax + +```sql +SELECT () FROM_clause WHERE GROUP BY time(time_interval,[)] +``` + +By default, a `GROUP BY time()` interval with no data reports `null` as its +value in the output column. +`fill()` changes the value reported for time intervals that have no data. +Note that `fill()` must go at the end of the `GROUP BY` clause if you're +`GROUP(ing) BY` several things (for example, both [tags](/enterprise_influxdb/v1.9/concepts/glossary/#tag) and a time interval). + +##### fill_option + +Any numerical value +              + Reports the given numerical value for time intervals with no data. + +`linear` +              +Reports the results of [linear interpolation](https://en.wikipedia.org/wiki/Linear_interpolation) for time intervals with no data. + +`none` +              +              +           +Reports no timestamp and no value for time intervals with no data. + +`null` +              +              +           +Reports null for time intervals with no data but returns a timestamp. This is the same as the default behavior. + +`previous` +              +              +    +Reports the value from the previous time interval for time intervals with no data. + +#### Examples + +{{< tabs-wrapper >}} +{{% tabs %}} +[Example 1: fill(100)](#) +[Example 2: fill(linear)](#) +[Example 3: fill(none)](#) +[Example 4: fill(null)](#) +[Example 5: fill(previous)](#) +{{% /tabs %}} +{{% tab-content %}} + +Without `fill(100)`: + +```sql +> SELECT MAX("water_level") FROM "h2o_feet" WHERE "location"='coyote_creek' AND time >= '2015-09-18T16:00:00Z' AND time <= '2015-09-18T16:42:00Z' GROUP BY time(12m) + +name: h2o_feet +-------------- +time max +2015-09-18T16:00:00Z 3.599 +2015-09-18T16:12:00Z 3.402 +2015-09-18T16:24:00Z 3.235 +2015-09-18T16:36:00Z +``` + +With `fill(100)`: + +```sql +> SELECT MAX("water_level") FROM "h2o_feet" WHERE "location"='coyote_creek' AND time >= '2015-09-18T16:00:00Z' AND time <= '2015-09-18T16:42:00Z' GROUP BY time(12m) fill(100) + +name: h2o_feet +-------------- +time max +2015-09-18T16:00:00Z 3.599 +2015-09-18T16:12:00Z 3.402 +2015-09-18T16:24:00Z 3.235 +2015-09-18T16:36:00Z 100 +``` + +`fill(100)` changes the value reported for the time interval with no data to `100`. + +{{% /tab-content %}} + +{{% tab-content %}} + +Without `fill(linear)`: + +```sql +> SELECT MEAN("tadpoles") FROM "pond" WHERE time >= '2016-11-11T21:00:00Z' AND time <= '2016-11-11T22:06:00Z' GROUP BY time(12m) + +name: pond +time mean +---- ---- +2016-11-11T21:00:00Z 1 +2016-11-11T21:12:00Z +2016-11-11T21:24:00Z 3 +2016-11-11T21:36:00Z +2016-11-11T21:48:00Z +2016-11-11T22:00:00Z 6 +``` + +With `fill(linear)`: + +```sql +> SELECT MEAN("tadpoles") FROM "pond" WHERE time >= '2016-11-11T21:00:00Z' AND time <= '2016-11-11T22:06:00Z' GROUP BY time(12m) fill(linear) + +name: pond +time mean +---- ---- +2016-11-11T21:00:00Z 1 +2016-11-11T21:12:00Z 2 +2016-11-11T21:24:00Z 3 +2016-11-11T21:36:00Z 4 +2016-11-11T21:48:00Z 5 +2016-11-11T22:00:00Z 6 +``` + +`fill(linear)` changes the value reported for the time interval with no data +to the results of [linear interpolation](https://en.wikipedia.org/wiki/Linear_interpolation). + +> **Note:** The data in Example 2 are not in `NOAA_water_database`. +We had to create a dataset with less regular data to work with `fill(linear)`. + +{{% /tab-content %}} + +{{% tab-content %}} + +Without `fill(none)`: + +```sql +> SELECT MAX("water_level") FROM "h2o_feet" WHERE "location"='coyote_creek' AND time >= '2015-09-18T16:00:00Z' AND time <= '2015-09-18T16:42:00Z' GROUP BY time(12m) + +name: h2o_feet +-------------- +time max +2015-09-18T16:00:00Z 3.599 +2015-09-18T16:12:00Z 3.402 +2015-09-18T16:24:00Z 3.235 +2015-09-18T16:36:00Z +``` + +With `fill(none)`: + +```sql +> SELECT MAX("water_level") FROM "h2o_feet" WHERE "location"='coyote_creek' AND time >= '2015-09-18T16:00:00Z' AND time <= '2015-09-18T16:42:00Z' GROUP BY time(12m) fill(none) + +name: h2o_feet +-------------- +time max +2015-09-18T16:00:00Z 3.599 +2015-09-18T16:12:00Z 3.402 +2015-09-18T16:24:00Z 3.235 +``` + +`fill(none)` reports no value and no timestamp for the time interval with no data. + +{{% /tab-content %}} + +{{% tab-content %}} + +Without `fill(null)`: + +```sql +> SELECT MAX("water_level") FROM "h2o_feet" WHERE "location"='coyote_creek' AND time >= '2015-09-18T16:00:00Z' AND time <= '2015-09-18T16:42:00Z' GROUP BY time(12m) + +name: h2o_feet +-------------- +time max +2015-09-18T16:00:00Z 3.599 +2015-09-18T16:12:00Z 3.402 +2015-09-18T16:24:00Z 3.235 +2015-09-18T16:36:00Z +``` + +With `fill(null)`: + +```sql +> SELECT MAX("water_level") FROM "h2o_feet" WHERE "location"='coyote_creek' AND time >= '2015-09-18T16:00:00Z' AND time <= '2015-09-18T16:42:00Z' GROUP BY time(12m) fill(null) + +name: h2o_feet +-------------- +time max +2015-09-18T16:00:00Z 3.599 +2015-09-18T16:12:00Z 3.402 +2015-09-18T16:24:00Z 3.235 +2015-09-18T16:36:00Z +``` + +`fill(null)` reports `null` as the value for the time interval with no data. +That result matches the result of the query without `fill(null)`. + +{{% /tab-content %}} + +{{% tab-content %}} + +Without `fill(previous)`: + +```sql +> SELECT MAX("water_level") FROM "h2o_feet" WHERE "location"='coyote_creek' AND time >= '2015-09-18T16:00:00Z' AND time <= '2015-09-18T16:42:00Z' GROUP BY time(12m) + +name: h2o_feet +-------------- +time max +2015-09-18T16:00:00Z 3.599 +2015-09-18T16:12:00Z 3.402 +2015-09-18T16:24:00Z 3.235 +2015-09-18T16:36:00Z +``` + +With `fill(previous)`: + +```sql +> SELECT MAX("water_level") FROM "h2o_feet" WHERE "location"='coyote_creek' AND time >= '2015-09-18T16:00:00Z' AND time <= '2015-09-18T16:42:00Z' GROUP BY time(12m) fill(previous) + +name: h2o_feet +-------------- +time max +2015-09-18T16:00:00Z 3.599 +2015-09-18T16:12:00Z 3.402 +2015-09-18T16:24:00Z 3.235 +2015-09-18T16:36:00Z 3.235 +``` + +`fill(previous)` changes the value reported for the time interval with no data to `3.235`, +the value from the previous time interval. + +{{% /tab-content %}} +{{< /tabs-wrapper >}} + +#### Common issues with `fill()` + +##### Queries with `fill()` when no data fall within the query's time range + +Currently, queries ignore `fill()` if no data fall within the query's time range. +This is the expected behavior. An open +[feature request](https://github.com/influxdata/influxdb/issues/6967) on GitHub +proposes that `fill()` should force a return of values even if the query's time +range covers no data. + +**Example** + +The following query returns no data because `water_level` has no points within +the query's time range. +Note that `fill(800)` has no effect on the query results. + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE "location" = 'coyote_creek' AND time >= '2015-09-18T22:00:00Z' AND time <= '2015-09-18T22:18:00Z' GROUP BY time(12m) fill(800) +> +``` + +##### Queries with `fill(previous)` when the previous result falls outside the query's time range + +`fill(previous)` doesn’t fill the result for a time interval if the previous +value is outside the query’s time range. + +**Example** + +The following query covers the time range between `2015-09-18T16:24:00Z` and `2015-09-18T16:54:00Z`. +Note that `fill(previous)` fills the result for `2015-09-18T16:36:00Z` with the +result from `2015-09-18T16:24:00Z`. + +```sql +> SELECT MAX("water_level") FROM "h2o_feet" WHERE location = 'coyote_creek' AND time >= '2015-09-18T16:24:00Z' AND time <= '2015-09-18T16:54:00Z' GROUP BY time(12m) fill(previous) + +name: h2o_feet +-------------- +time max +2015-09-18T16:24:00Z 3.235 +2015-09-18T16:36:00Z 3.235 +2015-09-18T16:48:00Z 4 +``` + +The next query shortens the time range in the previous query. +It now covers the time between `2015-09-18T16:36:00Z` and `2015-09-18T16:54:00Z`. +Note that `fill(previous)` doesn't fill the result for `2015-09-18T16:36:00Z` with the +result from `2015-09-18T16:24:00Z`; the result for `2015-09-18T16:24:00Z` is outside the query's +shorter time range. + +```sql +> SELECT MAX("water_level") FROM "h2o_feet" WHERE location = 'coyote_creek' AND time >= '2015-09-18T16:36:00Z' AND time <= '2015-09-18T16:54:00Z' GROUP BY time(12m) fill(previous) + +name: h2o_feet +-------------- +time max +2015-09-18T16:36:00Z +2015-09-18T16:48:00Z 4 +``` + +##### `fill(linear)` when the previous or following result falls outside the query's time range + +`fill(linear)` doesn't fill the result for a time interval with no data if the +previous result or the following result is outside the query's time range. + +**Example** + +The following query covers the time range between `2016-11-11T21:24:00Z` and +`2016-11-11T22:06:00Z`. Note that `fill(linear)` fills the results for the +`2016-11-11T21:36:00Z` time interval and the `2016-11-11T21:48:00Z` time interval +using the values from the `2016-11-11T21:24:00Z` time interval and the +`2016-11-11T22:00:00Z` time interval. + +```sql +> SELECT MEAN("tadpoles") FROM "pond" WHERE time > '2016-11-11T21:24:00Z' AND time <= '2016-11-11T22:06:00Z' GROUP BY time(12m) fill(linear) + +name: pond +time mean +---- ---- +2016-11-11T21:24:00Z 3 +2016-11-11T21:36:00Z 4 +2016-11-11T21:48:00Z 5 +2016-11-11T22:00:00Z 6 +``` + +The next query shortens the time range in the previous query. +It now covers the time between `2016-11-11T21:36:00Z` and `2016-11-11T22:06:00Z`. +Note that `fill()` previous doesn't fill the results for the `2016-11-11T21:36:00Z` +time interval and the `2016-11-11T21:48:00Z` time interval; the result for +`2016-11-11T21:24:00Z` is outside the query's shorter time range and InfluxDB +cannot perform the linear interpolation. + +```sql +> SELECT MEAN("tadpoles") FROM "pond" WHERE time >= '2016-11-11T21:36:00Z' AND time <= '2016-11-11T22:06:00Z' GROUP BY time(12m) fill(linear) +name: pond +time mean +---- ---- +2016-11-11T21:36:00Z +2016-11-11T21:48:00Z +2016-11-11T22:00:00Z 6 +``` + +> **Note:** The data in Issue 3 are not in `NOAA_water_database`. +> We had to create a dataset with less regular data to work with `fill(linear)`. + +# The INTO clause + +The `INTO` clause writes query results to a user-specified [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +### Syntax + +```sql +SELECT_clause INTO FROM_clause [WHERE_clause] [GROUP_BY_clause] +``` + +The `INTO` clause supports several formats for specifying a [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement): + +`INTO ` +           +Writes data to the specified measurement. +If you're using the [CLI](/enterprise_influxdb/v1.9/tools/use-influx/) InfluxDB writes the data to the measurement in the +[`USE`d](/enterprise_influxdb/v1.9/tools/use-influx/#commands) +[database](/enterprise_influxdb/v1.9/concepts/glossary/#database) and the `DEFAULT` [retention policy](/enterprise_influxdb/v1.9/concepts/glossary/#retention-policy-rp). +If you're using the [InfluxDB API](/enterprise_influxdb/v1.9/tools/api/) InfluxDB writes the data to the +measurement in the database specified in the [`db` query string parameter](/enterprise_influxdb/v1.9/tools/api/#query-string-parameters) +and the `DEFAULT` retention policy. + +`INTO ..` +           +Writes data to a fully qualified measurement. +Fully qualify a measurement by specifying its database and retention policy. + +`INTO ..` +           +Writes data to a measurement in a user-specified database and the `DEFAULT` +retention policy. + +`INTO ..:MEASUREMENT FROM //` +           +Writes data to all measurements in the user-specified database and +retention policy that match the [regular expression](#regular-expressions) in the `FROM` clause. +`:MEASUREMENT` is a backreference to each measurement matched in the `FROM` clause. + +### Examples + +#### Rename a database + +```sql +> SELECT * INTO "copy_NOAA_water_database"."autogen".:MEASUREMENT FROM "NOAA_water_database"."autogen"./.*/ GROUP BY * + +name: result +time written +---- ------- +0 76290 +``` + +Directly renaming a database in InfluxDB is not possible, so a common use for the `INTO` clause is to move data from one database to another. +The query above writes all data in the `NOAA_water_database` and `autogen` retention policy to the `copy_NOAA_water_database` database and the `autogen` retention policy. + +The [backreference](#examples-5) syntax (`:MEASUREMENT`) maintains the source measurement names in the destination database. +Note that both the `copy_NOAA_water_database` database and its `autogen` retention policy must exist prior to running the `INTO` query. +See [Database Management](/enterprise_influxdb/v1.9/query_language/manage-database/) +for how to manage databases and retention policies. + +The `GROUP BY *` clause [preserves tags](#missing-data) in the source database as tags in the destination database. +The following query does not maintain the series context for tags; tags will be stored as fields in the destination database (`copy_NOAA_water_database`): + +```sql +SELECT * INTO "copy_NOAA_water_database"."autogen".:MEASUREMENT FROM "NOAA_water_database"."autogen"./.*/ +``` + +When moving large amounts of data, we recommend sequentially running `INTO` queries for different measurements and using time boundaries in the [`WHERE` clause](#time-syntax). +This prevents your system from running out of memory. +The codeblock below provides sample syntax for those queries: + +``` +SELECT * +INTO .. +FROM .. +WHERE time > now() - 100w AND time < now() - 90w GROUP BY * + +SELECT * +INTO .. +FROM ..} +WHERE time > now() - 90w AND < now() - 80w GROUP BY * + +SELECT * +INTO .. +FROM .. +WHERE time > now() - 80w AND time < now() - 70w GROUP BY * +``` + +#### Write the results of a query to a measurement + +```sql +> SELECT "water_level" INTO "h2o_feet_copy_1" FROM "h2o_feet" WHERE "location" = 'coyote_creek' + +name: result +------------ +time written +1970-01-01T00:00:00Z 7604 + +> SELECT * FROM "h2o_feet_copy_1" + +name: h2o_feet_copy_1 +--------------------- +time water_level +2015-08-18T00:00:00Z 8.12 +[...] +2015-09-18T16:48:00Z 4 +``` + +The query writes its results a new [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement): `h2o_feet_copy_1`. +If you're using the [CLI](/enterprise_influxdb/v1.9/tools/use-influx/), InfluxDB writes the data to +the `USE`d [database](/enterprise_influxdb/v1.9/concepts/glossary/#database) and the `DEFAULT` [retention policy](/enterprise_influxdb/v1.9/concepts/glossary/#retention-policy-rp). +If you're using the [InfluxDB API](/enterprise_influxdb/v1.9/tools/api/), InfluxDB writes the +data to the database and retention policy specified in the `db` and `rp` +[query string parameters](/enterprise_influxdb/v1.9/tools/api/#query-string-parameters). +If you do not set the `rp` query string parameter, the InfluxDB API automatically +writes the data to the database's `DEFAULT` retention policy. + +The response shows the number of points (`7605`) that InfluxDB writes to `h2o_feet_copy_1`. +The timestamp in the response is meaningless; InfluxDB uses epoch 0 +(`1970-01-01T00:00:00Z`) as a null timestamp equivalent. + +#### Write the results of a query to a fully qualified measurement + +```sql +> SELECT "water_level" INTO "where_else"."autogen"."h2o_feet_copy_2" FROM "h2o_feet" WHERE "location" = 'coyote_creek' + +name: result +------------ +time written +1970-01-01T00:00:00Z 7604 + +> SELECT * FROM "where_else"."autogen"."h2o_feet_copy_2" + +name: h2o_feet_copy_2 +--------------------- +time water_level +2015-08-18T00:00:00Z 8.12 +[...] +2015-09-18T16:48:00Z 4 +``` + +The query writes its results to a new measurement: `h2o_feet_copy_2`. +InfluxDB writes the data to the `where_else` database and to the `autogen` +retention policy. +Note that both `where_else` and `autogen` must exist prior to running the `INTO` +query. +See [Database Management](/enterprise_influxdb/v1.9/query_language/manage-database/) +for how to manage databases and retention policies. + +The response shows the number of points (`7605`) that InfluxDB writes to `h2o_feet_copy_2`. +The timestamp in the response is meaningless; InfluxDB uses epoch 0 +(`1970-01-01T00:00:00Z`) as a null timestamp equivalent. + +#### Write aggregated results to a measurement (downsampling) + +```sql +> SELECT MEAN("water_level") INTO "all_my_averages" FROM "h2o_feet" WHERE "location" = 'coyote_creek' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' GROUP BY time(12m) + +name: result +------------ +time written +1970-01-01T00:00:00Z 3 + +> SELECT * FROM "all_my_averages" + +name: all_my_averages +--------------------- +time mean +2015-08-18T00:00:00Z 8.0625 +2015-08-18T00:12:00Z 7.8245 +2015-08-18T00:24:00Z 7.5675 +``` + +The query aggregates data using an +InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions) and a [`GROUP BY +time()` clause](#group-by-time-intervals). +It also writes its results to the `all_my_averages` measurement. + +The response shows the number of points (`3`) that InfluxDB writes to `all_my_averages`. +The timestamp in the response is meaningless; InfluxDB uses epoch 0 +(`1970-01-01T00:00:00Z`) as a null timestamp equivalent. + +The query is an example of downsampling: taking higher precision data, +aggregating those data to a lower precision, and storing the lower precision +data in the database. +Downsampling is a common use case for the `INTO` clause. + +#### Write aggregated results for more than one measurement to a different database (downsampling with backreferencing) + +```sql +> SELECT MEAN(*) INTO "where_else"."autogen".:MEASUREMENT FROM /.*/ WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:06:00Z' GROUP BY time(12m) + +name: result +time written +---- ------- +1970-01-01T00:00:00Z 5 + +> SELECT * FROM "where_else"."autogen"./.*/ + +name: average_temperature +time mean_degrees mean_index mean_pH mean_water_level +---- ------------ ---------- ------- ---------------- +2015-08-18T00:00:00Z 78.5 + +name: h2o_feet +time mean_degrees mean_index mean_pH mean_water_level +---- ------------ ---------- ------- ---------------- +2015-08-18T00:00:00Z 5.07625 + +name: h2o_pH +time mean_degrees mean_index mean_pH mean_water_level +---- ------------ ---------- ------- ---------------- +2015-08-18T00:00:00Z 6.75 + +name: h2o_quality +time mean_degrees mean_index mean_pH mean_water_level +---- ------------ ---------- ------- ---------------- +2015-08-18T00:00:00Z 51.75 + +name: h2o_temperature +time mean_degrees mean_index mean_pH mean_water_level +---- ------------ ---------- ------- ---------------- +2015-08-18T00:00:00Z 63.75 +``` + +The query aggregates data using an +InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions) and a [`GROUP BY +time()` clause](#group-by-time-intervals). +It aggregates data in every measurement that matches the [regular expression](#regular-expressions) +in the `FROM` clause and writes the results to measurements with the same name in the +`where_else` database and the `autogen` retention policy. +Note that both `where_else` and `autogen` must exist prior to running the `INTO` +query. +See [Database management](/enterprise_influxdb/v1.9/query_language/manage-database/) +for how to manage databases and retention policies. + +The response shows the number of points (`5`) that InfluxDB writes to the `where_else` +database and the `autogen` retention policy. +The timestamp in the response is meaningless; InfluxDB uses epoch 0 +(`1970-01-01T00:00:00Z`) as a null timestamp equivalent. + +The query is an example of downsampling with backreferencing. +It takes higher precision data from more than one measurement, +aggregates those data to a lower precision, and stores the lower precision +data in the database. +Downsampling with backreferencing is a common use case for the `INTO` clause. + +### Common issues with the `INTO` clause + +#### Missing data + +If an `INTO` query includes a [tag key](/enterprise_influxdb/v1.9/concepts/glossary#tag-key) in the [`SELECT` clause](#the-basic-select-statement), the query converts [tags](/enterprise_influxdb/v1.9/concepts/glossary#tag) in the current +measurement to [fields](/enterprise_influxdb/v1.9/concepts/glossary#field) in the destination measurement. +This can cause InfluxDB to overwrite [points](/enterprise_influxdb/v1.9/concepts/glossary#point) that were previously differentiated +by a [tag value](/enterprise_influxdb/v1.9/concepts/glossary#tag-value). +Note that this behavior does not apply to queries that use the [`TOP()`](/enterprise_influxdb/v1.9/query_language/functions/#top) or [`BOTTOM()`](/enterprise_influxdb/v1.9/query_language/functions/#bottom) functions. +The +[Frequently Asked Questions](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#why-are-my-into-queries-missing-data) +document describes that behavior in detail. + +To preserve tags in the current measurement as tags in the destination measurement, +[`GROUP BY` the relevant tag key](#group-by-tags) or `GROUP BY *` in the `INTO` query. + +#### Automating queries with the `INTO` clause + +The `INTO` clause section in this document shows how to manually implement +queries with an `INTO` clause. +See the [Continuous Queries](/enterprise_influxdb/v1.9/query_language/continuous_queries/) +documentation for how to automate `INTO` clause queries on realtime data. +Among [other uses](/enterprise_influxdb/v1.9/query_language/continuous_queries/#continuous-query-use-cases), +Continuous Queries automate the downsampling process. + +## ORDER BY time DESC + +By default, InfluxDB returns results in ascending time order; the first [point](/enterprise_influxdb/v1.9/concepts/glossary/#point) +returned has the oldest [timestamp](/enterprise_influxdb/v1.9/concepts/glossary/#timestamp) and +the last point returned has the most recent timestamp. +`ORDER BY time DESC` reverses that order such that InfluxDB returns the points +with the most recent timestamps first. + +### Syntax + +```sql +SELECT_clause [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] ORDER BY time DESC +``` + +`ORDER by time DESC` must appear after the [`GROUP BY` clause](#the-group-by-clause) +if the query includes a `GROUP BY` clause. +`ORDER by time DESC` must appear after the [`WHERE` clause](#the-where-clause) +if the query includes a `WHERE` clause and no `GROUP BY` clause. + +### Examples + +#### Return the newest points first + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE "location" = 'santa_monica' ORDER BY time DESC + +name: h2o_feet +time water_level +---- ----------- +2015-09-18T21:42:00Z 4.938 +2015-09-18T21:36:00Z 5.066 +[...] +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:00:00Z 2.064 +``` + +The query returns the points with the most recent timestamps from the +`h2o_feet` [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) first. +Without `ORDER by time DESC`, the query would return `2015-08-18T00:00:00Z` +first and `2015-09-18T21:42:00Z` last. + +#### Return the newest points first and include a GROUP BY time() clause + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:42:00Z' GROUP BY time(12m) ORDER BY time DESC + +name: h2o_feet +time mean +---- ---- +2015-08-18T00:36:00Z 4.6825 +2015-08-18T00:24:00Z 4.80675 +2015-08-18T00:12:00Z 4.950749999999999 +2015-08-18T00:00:00Z 5.07625 +``` + +The query uses an InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions) +and a time interval in the [GROUP BY clause](#group-by-time-intervals) +to calculate the average `water_level` for each twelve-minute +interval in the query's time range. +`ORDER BY time DESC` returns the most recent 12-minute time intervals +first. + +Without `ORDER BY time DESC`, the query would return +`2015-08-18T00:00:00Z` first and `2015-08-18T00:36:00Z` last. + +# The LIMIT and SLIMIT clauses + +`LIMIT` and `SLIMIT` limit the number of +[points](/enterprise_influxdb/v1.9/concepts/glossary/#point) and the number of +[series](/enterprise_influxdb/v1.9/concepts/glossary/#series) returned per query. + +## The LIMIT clause + +`LIMIT ` returns the first `N` [points](/enterprise_influxdb/v1.9/concepts/glossary/#point) from the specified [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +### Syntax + +```sql +SELECT_clause [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] LIMIT +``` + +`N` specifies the number of [points](/enterprise_influxdb/v1.9/concepts/glossary/#point) to return from the specified [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). +If `N` is greater than the number of points in a measurement, InfluxDB returns +all points from that series. + +Note that the `LIMIT` clause must appear in the order outlined in the syntax above. + +### Examples + +#### Limit the number of points returned + +```sql +> SELECT "water_level","location" FROM "h2o_feet" LIMIT 3 + +name: h2o_feet +time water_level location +---- ----------- -------- +2015-08-18T00:00:00Z 8.12 coyote_creek +2015-08-18T00:00:00Z 2.064 santa_monica +2015-08-18T00:06:00Z 8.005 coyote_creek +``` + +The query returns the three oldest [points](/enterprise_influxdb/v1.9/concepts/glossary/#point) (determined by timestamp) from the `h2o_feet` [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +#### Limit the number points returned and include a GROUP BY clause + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:42:00Z' GROUP BY *,time(12m) LIMIT 2 + +name: h2o_feet +tags: location=coyote_creek +time mean +---- ---- +2015-08-18T00:00:00Z 8.0625 +2015-08-18T00:12:00Z 7.8245 + +name: h2o_feet +tags: location=santa_monica +time mean +---- ---- +2015-08-18T00:00:00Z 2.09 +2015-08-18T00:12:00Z 2.077 +``` + +The query uses an InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions) +and a [GROUP BY clause](#group-by-time-intervals) +to calculate the average `water_level` for each [tag](/enterprise_influxdb/v1.9/concepts/glossary/#tag) and for each twelve-minute +interval in the query's time range. +`LIMIT 2` requests the two oldest twelve-minute averages (determined by timestamp). + +Note that without `LIMIT 2`, the query would return four points per [series](/enterprise_influxdb/v1.9/concepts/glossary/#series); +one for each twelve-minute interval in the query's time range. + +## The `SLIMIT` clause + +`SLIMIT ` returns every [point](/enterprise_influxdb/v1.9/concepts/glossary/#point) from \ [series](/enterprise_influxdb/v1.9/concepts/glossary/#series) in the specified [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +### Syntax + +```sql +SELECT_clause [INTO_clause] FROM_clause [WHERE_clause] GROUP BY *[,time()] [ORDER_BY_clause] SLIMIT +``` + +`N` specifies the number of [series](/enterprise_influxdb/v1.9/concepts/glossary/#series) to return from the specified [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). +If `N` is greater than the number of series in a measurement, InfluxDB returns +all series from that measurement. + +There is an [ongoing issue](https://github.com/influxdata/influxdb/issues/7571) that requires queries with `SLIMIT` to include `GROUP BY *`. +Note that the `SLIMIT` clause must appear in the order outlined in the syntax above. + +### Examples + +#### Limit the number of series returned + +```sql +> SELECT "water_level" FROM "h2o_feet" GROUP BY * SLIMIT 1 + +name: h2o_feet +tags: location=coyote_creek +time water_level +---- ----- +2015-08-18T00:00:00Z 8.12 +2015-08-18T00:06:00Z 8.005 +2015-08-18T00:12:00Z 7.887 +[...] +2015-09-18T16:12:00Z 3.402 +2015-09-18T16:18:00Z 3.314 +2015-09-18T16:24:00Z 3.235 +``` + +The query returns all `water_level` [points](/enterprise_influxdb/v1.9/concepts/glossary/#point) from one of the [series](/enterprise_influxdb/v1.9/concepts/glossary/#series) associated +with the `h2o_feet` [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +#### Limit the number of series returned and include a GROUP BY time() clause + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:42:00Z' GROUP BY *,time(12m) SLIMIT 1 + +name: h2o_feet +tags: location=coyote_creek +time mean +---- ---- +2015-08-18T00:00:00Z 8.0625 +2015-08-18T00:12:00Z 7.8245 +2015-08-18T00:24:00Z 7.5675 +2015-08-18T00:36:00Z 7.303 +``` + +The query uses an InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions) +and a time interval in the [GROUP BY clause](#group-by-time-intervals) +to calculate the average `water_level` for each twelve-minute +interval in the query's time range. +`SLIMIT 1` requests a single series associated with the `h2o_feet` measurement. + +Note that without `SLIMIT 1`, the query would return results for the two series +associated with the `h2o_feet` measurement: `location=coyote_creek` and +`location=santa_monica`. + +## LIMIT and SLIMIT + +`LIMIT ` followed by `SLIMIT ` returns the first \ [points](/enterprise_influxdb/v1.9/concepts/glossary/#point) from \ [series](/enterprise_influxdb/v1.9/concepts/glossary/#series) in the specified measurement. + +### Syntax + +```sql +SELECT_clause [INTO_clause] FROM_clause [WHERE_clause] GROUP BY *[,time()] [ORDER_BY_clause] LIMIT SLIMIT +``` + +`N1` specifies the number of [points](/enterprise_influxdb/v1.9/concepts/glossary/#point) to return per [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). +If `N1` is greater than the number of points in a measurement, InfluxDB returns all points from that measurement. + +`N2` specifies the number of series to return from the specified [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). +If `N2` is greater than the number of series in a measurement, InfluxDB returns all series from that measurement. + +There is an [ongoing issue](https://github.com/influxdata/influxdb/issues/7571) that requires queries with `LIMIT` and `SLIMIT` to include `GROUP BY *`. +Note that the `LIMIT` and `SLIMIT` clauses must appear in the order outlined in the syntax above. + +### Examples + +#### Limit the number of points and series returned + +```sql +> SELECT "water_level" FROM "h2o_feet" GROUP BY * LIMIT 3 SLIMIT 1 + +name: h2o_feet +tags: location=coyote_creek +time water_level +---- ----------- +2015-08-18T00:00:00Z 8.12 +2015-08-18T00:06:00Z 8.005 +2015-08-18T00:12:00Z 7.887 +``` + +The query returns the three oldest [points](/enterprise_influxdb/v1.9/concepts/glossary/#point) (determined by timestamp) from one +of the [series](/enterprise_influxdb/v1.9/concepts/glossary/#series) associated with the +[measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) `h2o_feet`. + +#### Limit the number of points and series returned and include a GROUP BY time() clause + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:42:00Z' GROUP BY *,time(12m) LIMIT 2 SLIMIT 1 + +name: h2o_feet +tags: location=coyote_creek +time mean +---- ---- +2015-08-18T00:00:00Z 8.0625 +2015-08-18T00:12:00Z 7.8245 +``` + +The query uses an InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions) +and a time interval in the [GROUP BY clause](#group-by-time-intervals) +to calculate the average `water_level` for each twelve-minute +interval in the query's time range. +`LIMIT 2` requests the two oldest twelve-minute averages (determined by +timestamp) and `SLIMIT 1` requests a single series +associated with the `h2o_feet` measurement. + +Note that without `LIMIT 2 SLIMIT 1`, the query would return four points +for each of the two series associated with the `h2o_feet` measurement. + +## The OFFSET and SOFFSET clauses + +`OFFSET` and `SOFFSET` paginates [points](/enterprise_influxdb/v1.9/concepts/glossary/#point) and [series](/enterprise_influxdb/v1.9/concepts/glossary/#series) returned. + + + + + + +
        The OFFSET clauseThe SOFFSET clause
        + +## The `OFFSET` clause + +`OFFSET ` paginates `N` [points](/enterprise_influxdb/v1.9/concepts/glossary/#point) in the query results. + +### Syntax + +```sql +SELECT_clause [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] LIMIT_clause OFFSET [SLIMIT_clause] +``` + +`N` specifies the number of [points](/enterprise_influxdb/v1.9/concepts/glossary/#point) to paginate. +The `OFFSET` clause requires a [`LIMIT` clause](#the-limit-clause). +Using the `OFFSET` clause without a `LIMIT` clause can cause [inconsistent +query results](https://github.com/influxdata/influxdb/issues/7577). + +> **Note:** InfluxDB returns no results if the `WHERE` clause includes a time +range and the `OFFSET` clause would cause InfluxDB to return points with +timestamps outside of that time range. + +### Examples + +#### Paginate points + +```sql +> SELECT "water_level","location" FROM "h2o_feet" LIMIT 3 OFFSET 3 + +name: h2o_feet +time water_level location +---- ----------- -------- +2015-08-18T00:06:00Z 2.116 santa_monica +2015-08-18T00:12:00Z 7.887 coyote_creek +2015-08-18T00:12:00Z 2.028 santa_monica +``` + +The query returns the fourth, fifth, and sixth [points](/enterprise_influxdb/v1.9/concepts/glossary/#point) from the `h2o_feet` [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). +If the query did not include `OFFSET 3`, it would return the first, second, +and third points from that measurement. + +#### Paginate points and include several clauses + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:42:00Z' GROUP BY *,time(12m) ORDER BY time DESC LIMIT 2 OFFSET 2 SLIMIT 1 + +name: h2o_feet +tags: location=coyote_creek +time mean +---- ---- +2015-08-18T00:12:00Z 7.8245 +2015-08-18T00:00:00Z 8.0625 +``` + +This example is pretty involved, so here's the clause-by-clause breakdown: + +The [`SELECT` clause](#the-basic-select-statement) specifies an InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions). +The [`FROM` clause](#the-basic-select-statement) specifies a single measurement. +The [`WHERE` clause](#the-where-clause) specifies the time range for the query. +The [`GROUP BY` clause](#the-group-by-clause) groups results by all tags (`*`) and into 12-minute intervals. +The [`ORDER BY time DESC` clause](#order-by-time-desc) returns results in descending timestamp order. +The [`LIMIT 2` clause](#the-limit-clause) limits the number of points returned to two. +The `OFFSET 2` clause excludes the first two averages from the query results. +The [`SLIMIT 1` clause](#the-slimit-clause) limits the number of series returned to one. + +Without `OFFSET 2`, the query would return the first two averages of the query results: + +```sql +name: h2o_feet +tags: location=coyote_creek +time mean +---- ---- +2015-08-18T00:36:00Z 7.303 +2015-08-18T00:24:00Z 7.5675 +``` + +## The `SOFFSET` clause + +`SOFFSET ` paginates `N` [series](/enterprise_influxdb/v1.9/concepts/glossary/#series) in the query results. + +### Syntax + +```sql +SELECT_clause [INTO_clause] FROM_clause [WHERE_clause] GROUP BY *[,time(time_interval)] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] SLIMIT_clause SOFFSET +``` + +`N` specifies the number of [series](/enterprise_influxdb/v1.9/concepts/glossary/#series) to paginate. +The `SOFFSET` clause requires an [`SLIMIT` clause](#the-slimit-clause). +Using the `SOFFSET` clause without an `SLIMIT` clause can cause [inconsistent +query results](https://github.com/influxdata/influxdb/issues/7578). +There is an [ongoing issue](https://github.com/influxdata/influxdb/issues/7571) that requires queries with `SLIMIT` to include `GROUP BY *`. + +> **Note:** InfluxDB returns no results if the `SOFFSET` clause paginates +through more than the total number of series. + +### Examples + +#### Paginate series + +```sql +> SELECT "water_level" FROM "h2o_feet" GROUP BY * SLIMIT 1 SOFFSET 1 + +name: h2o_feet +tags: location=santa_monica +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +[...] +2015-09-18T21:36:00Z 5.066 +2015-09-18T21:42:00Z 4.938 +``` + +The query returns data for the [series](/enterprise_influxdb/v1.9/concepts/glossary/#series) associated with the `h2o_feet` +[measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) and the `location = santa_monica` [tag](/enterprise_influxdb/v1.9/concepts/glossary/#tag). +Without `SOFFSET 1`, the query returns data for the series associated with the +`h2o_feet` measurement and the `location = coyote_creek` tag. + +#### Paginate series and include all clauses + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:42:00Z' GROUP BY *,time(12m) ORDER BY time DESC LIMIT 2 OFFSET 2 SLIMIT 1 SOFFSET 1 + +name: h2o_feet +tags: location=santa_monica +time mean +---- ---- +2015-08-18T00:12:00Z 2.077 +2015-08-18T00:00:00Z 2.09 +``` + +This example is pretty involved, so here's the clause-by-clause breakdown: + +The [`SELECT` clause](#the-basic-select-statement) specifies an InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions). +The [`FROM` clause](#the-basic-select-statement) specifies a single measurement. +The [`WHERE` clause](#the-where-clause) specifies the time range for the query. +The [`GROUP BY` clause](#the-group-by-clause) groups results by all tags (`*`) and into 12-minute intervals. +The [`ORDER BY time DESC` clause](#order-by-time-desc) returns results in descending timestamp order. +The [`LIMIT 2` clause](#the-limit-clause) limits the number of points returned to two. +The [`OFFSET 2` clause](#the-offset-clause) excludes the first two averages from the query results. +The [`SLIMIT 1` clause](#the-slimit-clause) limits the number of series returned to one. +The `SOFFSET 1` clause paginates the series returned. + +Without `SOFFSET 1`, the query would return the results for a different series: + +```sql +name: h2o_feet +tags: location=coyote_creek +time mean +---- ---- +2015-08-18T00:12:00Z 7.8245 +2015-08-18T00:00:00Z 8.0625 +``` + +## The Time Zone clause + +The `tz()` clause returns the UTC offset for the specified timezone. + +### Syntax + +```sql +SELECT_clause [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] tz('') +``` + +By default, InfluxDB stores and returns timestamps in UTC. +The `tz()` clause includes the UTC offset or, if applicable, the UTC Daylight Savings Time (DST) offset to the query's returned timestamps. +The returned timestamps must be in [RFC3339 format](/enterprise_influxdb/v1.9/query_language/explore-data/#configuring-the-returned-timestamps) for the UTC offset or UTC DST to appear. +The `time_zone` parameter follows the TZ syntax in the [Internet Assigned Numbers Authority time zone database](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) and it requires single quotes. + +### Examples + +#### Return the UTC offset for Chicago's time zone + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:18:00Z' tz('America/Chicago') + +name: h2o_feet +time water_level +---- ----------- +2015-08-17T19:00:00-05:00 2.064 +2015-08-17T19:06:00-05:00 2.116 +2015-08-17T19:12:00-05:00 2.028 +2015-08-17T19:18:00-05:00 2.126 +``` + +The query results include the UTC offset (`-05:00`) for the `America/Chicago` time zone in the timestamps. + +## Time syntax + +For most `SELECT` statements, the default time range is between [`1677-09-21 00:12:43.145224194` and `2262-04-11T23:47:16.854775806Z` UTC](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#what-are-the-minimum-and-maximum-timestamps-that-influxdb-can-store). +For `SELECT` statements with a [`GROUP BY time()` clause](#group-by-time-intervals), +the default time range is between `1677-09-21 00:12:43.145224194` UTC and [`now()`](/enterprise_influxdb/v1.9/concepts/glossary/#now). +The following sections detail how to specify alternative time ranges in the `SELECT` +statement's [`WHERE` clause](#the-where-clause). + + + + + + + +
        Absolute timeRelative timeCommon issues with time syntax
        + +Tired of reading? Check out this InfluxQL Short: +
        +
        + + +## Absolute time + +Specify absolute time with date-time strings and epoch time. + +### Syntax + +```sql +SELECT_clause FROM_clause WHERE time ['' | '' | ] [AND ['' | '' | ] [...]] +``` + +#### Supported operators + +| Operator | Meaning | +|:--------:|:------- | +| `=` | equal to | +| `<>` | not equal to | +| `!=` | not equal to | +| `>` | greater than | +| `>=` | greater than or equal to | +| `<` | less than | +| `<=` | less than or equal to | + +Currently, InfluxDB does not support using `OR` with absolute time in the `WHERE` +clause. See the [Frequently Asked Questions](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#why-is-my-query-with-a-where-or-time-clause-returning-empty-results) +document and the [GitHub Issue](https://github.com/influxdata/influxdb/issues/7530) +for more information. + +#### `rfc3339_date_time_string` + +```sql +'YYYY-MM-DDTHH:MM:SS.nnnnnnnnnZ' +``` + +`.nnnnnnnnn` is optional and is set to `.000000000` if not included. +The [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) date-time string requires single quotes. + +#### `rfc3339_like_date_time_string` + +```sql +'YYYY-MM-DD HH:MM:SS.nnnnnnnnn' +``` + +`HH:MM:SS.nnnnnnnnn.nnnnnnnnn` is optional and is set to `00:00:00.000000000` if not included. +The RFC3339-like date-time string requires single quotes. + +#### `epoch_time` + +Epoch time is the amount of time that has elapsed since 00:00:00 +Coordinated Universal Time (UTC), Thursday, 1 January 1970. + +By default, InfluxDB assumes that all epoch timestamps are in nanoseconds. +Include a [duration literal](/enterprise_influxdb/v1.9/query_language/spec/#durations) +at the end of the epoch timestamp to indicate a precision other than nanoseconds. + +#### Basic arithmetic + +All timestamp formats support basic arithmetic. +Add (`+`) or subtract (`-`) a time from a timestamp with a [duration literal](/enterprise_influxdb/v1.9/query_language/spec/#durations). +Note that InfluxQL requires a whitespace between the `+` or `-` and the +duration literal. + +### Examples + +#### Specify a time range with RFC3339 date-time strings + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00.000000000Z' AND time <= '2015-08-18T00:12:00Z' + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +``` + +The query returns data with timestamps between August 18, 2015 at 00:00:00.000000000 and +August 18, 2015 at 00:12:00. +The nanosecond specification in the first timestamp (`.000000000`) +is optional. + +Note that the single quotes around the RFC3339 date-time strings are required. + +#### Specify a time range with RFC3339-like date-time strings + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18' AND time <= '2015-08-18 00:12:00' + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +``` + +The query returns data with timestamps between August 18, 2015 at 00:00:00 and August 18, 2015 +at 00:12:00. +The first date-time string does not include a time; InfluxDB assumes the time +is 00:00:00. + +Note that the single quotes around the RFC3339-like date-time strings are +required. + +#### Specify a time range with epoch timestamps + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= 1439856000000000000 AND time <= 1439856720000000000 + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +``` + +The query returns data with timestamps that occur between August 18, 2015 +at 00:00:00 and August 18, 2015 at 00:12:00. +By default InfluxDB assumes epoch timestamps are in nanoseconds. + +#### Specify a time range with second-precision epoch timestamps + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= 1439856000s AND time <= 1439856720s + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +``` + +The query returns data with timestamps that occur between August 18, 2015 +at 00:00:00 and August 18, 2015 at 00:12:00. +The `s` [duration literal](/enterprise_influxdb/v1.9/query_language/spec/#durations) at the +end of the epoch timestamps indicate that the epoch timestamps are in seconds. + +#### Perform basic arithmetic on an RFC3339-like date-time string + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE time > '2015-09-18T21:24:00Z' + 6m + +name: h2o_feet +time water_level +---- ----------- +2015-09-18T21:36:00Z 5.066 +2015-09-18T21:42:00Z 4.938 +``` + +The query returns data with timestamps that occur at least six minutes after +September 18, 2015 at 21:24:00. +Note that the whitespace between the `+` and `6m` is required. + +#### Perform basic arithmetic on an epoch timestamp + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE time > 24043524m - 6m + +name: h2o_feet +time water_level +---- ----------- +2015-09-18T21:24:00Z 5.013 +2015-09-18T21:30:00Z 5.01 +2015-09-18T21:36:00Z 5.066 +2015-09-18T21:42:00Z 4.938 +``` + +The query returns data with timestamps that occur at least six minutes before +September 18, 2015 at 21:24:00. +Note that the whitespace between the `-` and `6m` is required. + +## Relative time + +Use [`now()`](/enterprise_influxdb/v1.9/concepts/glossary/#now) to query data with [timestamps](/enterprise_influxdb/v1.9/concepts/glossary/#timestamp) relative to the server's current timestamp. + +### Syntax + +```sql +SELECT_clause FROM_clause WHERE time now() [[ - | + ] ] [(AND|OR) now() [...]] +``` + +`now()` is the Unix time of the server at the time the query is executed on that server. +The whitespace between `-` or `+` and the [duration literal](/enterprise_influxdb/v1.9/query_language/spec/#durations) is required. + +#### Supported operators +| Operator | Meaning | +|:--------:|:------- | +| `=` | equal to | +| `<>` | not equal to | +| `!=` | not equal to | +| `>` | greater than | +| `>=` | greater than or equal to | +| `<` | less than | +| `<=` | less than or equal to | + +#### `duration_literal` + +- microseconds: `u` or `µ` +- milliseconds: `ms` +- seconds`s` +- minutes`m` +- hours:`h` +- days:`d` +- weeks:`w` + +### Examples + +#### Specify a time range with relative time + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE time > now() - 1h +``` + +The query returns data with timestamps that occur within the past hour. +The whitespace between `-` and `1h` is required. + +#### Specify a time range with absolute time and relative time + +```sql +> SELECT "level description" FROM "h2o_feet" WHERE time > '2015-09-18T21:18:00Z' AND time < now() + 1000d + +name: h2o_feet +time level description +---- ----------------- +2015-09-18T21:24:00Z between 3 and 6 feet +2015-09-18T21:30:00Z between 3 and 6 feet +2015-09-18T21:36:00Z between 3 and 6 feet +2015-09-18T21:42:00Z between 3 and 6 feet +``` + +The query returns data with timestamps that occur between September 18, 2015 +at 21:18:00 and 1000 days from `now()`. +The whitespace between `+` and `1000d` is required. + +## Common issues with time syntax + +### Using `OR` to select time multiple time intervals + +InfluxDB does not support using the `OR` operator in the `WHERE` clause to specify multiple time intervals. + +For more information, see [Frequently asked questions](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#why-is-my-query-with-a-where-or-time-clause-returning-empty-results). + +### Querying data that occur after `now()` with a `GROUP BY time()` clause + +Most `SELECT` statements have a default time range between [`1677-09-21 00:12:43.145224194` and `2262-04-11T23:47:16.854775806Z` UTC](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#what-are-the-minimum-and-maximum-timestamps-that-influxdb-can-store). +For `SELECT` statements with a [`GROUP BY time()` clause](#group-by-time-intervals), +the default time range is between `1677-09-21 00:12:43.145224194` UTC and [`now()`](/enterprise_influxdb/v1.9/concepts/glossary/#now). + +To query data with timestamps that occur after `now()`, `SELECT` statements with +a `GROUP BY time()` clause must provide an alternative upper bound in the +`WHERE` clause. + +#### Example + +Use the [CLI](/enterprise_influxdb/v1.9/tools/use-influx/) to write a point to the `NOAA_water_database` that occurs after `now()`: + +```sql +> INSERT h2o_feet,location=santa_monica water_level=3.1 1587074400000000000 +``` + +Run a `GROUP BY time()` query that covers data with timestamps between +`2015-09-18T21:30:00Z` and `now()`: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE "location"='santa_monica' AND time >= '2015-09-18T21:30:00Z' GROUP BY time(12m) fill(none) + +name: h2o_feet +time mean +---- ---- +2015-09-18T21:24:00Z 5.01 +2015-09-18T21:36:00Z 5.002 +``` + +Run a `GROUP BY time()` query that covers data with timestamps between +`2015-09-18T21:30:00Z` and 180 weeks from `now()`: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE "location"='santa_monica' AND time >= '2015-09-18T21:30:00Z' AND time <= now() + 180w GROUP BY time(12m) fill(none) + +name: h2o_feet +time mean +---- ---- +2015-09-18T21:24:00Z 5.01 +2015-09-18T21:36:00Z 5.002 +2020-04-16T22:00:00Z 3.1 +``` + +Note that the `WHERE` clause must provide an alternative **upper** bound to +override the default `now()` upper bound. The following query merely resets +the lower bound to `now()` such that the query's time range is between +`now()` and `now()`: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE "location"='santa_monica' AND time >= now() GROUP BY time(12m) fill(none) +> +``` + +### Configuring the returned timestamps + +The [CLI](/enterprise_influxdb/v1.9/tools/use-influx/) returns timestamps in +nanosecond epoch format by default. +Specify alternative formats with the +[`precision ` command](/enterprise_influxdb/v1.9/tools/use-influx/#influx-commands). +The [InfluxDB API](/enterprise_influxdb/v1.9/tools/api/) returns timestamps +in [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) format by default. +Specify alternative formats with the +[`epoch` query string parameter](/enterprise_influxdb/v1.9/tools/api/#query-string-parameters). + +## Regular expressions + +InfluxQL supports using regular expressions when specifying: + +* [field keys](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) and [tag keys](/enterprise_influxdb/v1.9/concepts/glossary/#tag-key) in the [`SELECT` clause](#the-basic-select-statement) +* [measurements](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) in the [`FROM` clause](#the-basic-select-statement) +* [tag values](/enterprise_influxdb/v1.9/concepts/glossary/#tag-value) and string [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value) in the [`WHERE` clause](#the-where-clause). +* [tag keys](/enterprise_influxdb/v1.9/concepts/glossary/#tag-key) in the [`GROUP BY` clause](#group-by-tags) + +Currently, InfluxQL does not support using regular expressions to match +non-string field values in the +`WHERE` clause, +[databases](/enterprise_influxdb/v1.9/concepts/glossary/#database), and +[retention polices](/enterprise_influxdb/v1.9/concepts/glossary/#retention-policy-rp). + +> **Note:** Regular expression comparisons are more computationally intensive than exact +string comparisons; queries with regular expressions are not as performant +as those without. + +### Syntax + +```sql +SELECT // FROM // WHERE [ // | //] GROUP BY // +``` + +Regular expressions are surrounded by `/` characters and use +[Golang's regular expression syntax](http://golang.org/pkg/regexp/syntax/). + +#### Supported operators + +`=~` matches against +`!~` doesn't match against + +### Examples + +#### Use a regular expression to specify field keys and tag keys in the SELECT clause + +```sql +> SELECT /l/ FROM "h2o_feet" LIMIT 1 + +name: h2o_feet +time level description location water_level +---- ----------------- -------- ----------- +2015-08-18T00:00:00Z between 6 and 9 feet coyote_creek 8.12 +``` + +The query selects all [field keys](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +and [tag keys](/enterprise_influxdb/v1.9/concepts/glossary/#tag-key) that include an `l`. +Note that the regular expression in the `SELECT` clause must match at least one +field key in order to return results for a tag key that matches the regular +expression. + +Currently, there is no syntax to distinguish between regular expressions for +field keys and regular expressions for tag keys in the `SELECT` clause. +The syntax `//::[field | tag]` is not supported. + +#### Use a regular expression to specify measurements in the FROM clause + +```sql +> SELECT MEAN("degrees") FROM /temperature/ + +name: average_temperature +time mean +---- ---- +1970-01-01T00:00:00Z 79.98472932232272 + +name: h2o_temperature +time mean +---- ---- +1970-01-01T00:00:00Z 64.98872722506226 +``` + +The query uses an InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions/) +to calculate the average `degrees` for every [measurement](/enterprise_influxdb/v1.9/concepts/glossary#measurement) in the `NOAA_water_database` +[database](/enterprise_influxdb/v1.9/concepts/glossary#database) that contains the word `temperature`. + +#### Use a regular expression to specify tag values in the WHERE clause + +```sql +> SELECT MEAN(water_level) FROM "h2o_feet" WHERE "location" =~ /[m]/ AND "water_level" > 3 + +name: h2o_feet +time mean +---- ---- +1970-01-01T00:00:00Z 4.47155532049926 +``` + +The query uses an InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions/) +to calculate the average `water_level` where the [tag value](/enterprise_influxdb/v1.9/concepts/glossary#tag-value) of `location` +includes an `m` and `water_level` is greater than three. + +#### Use a regular expression to specify a tag with no value in the WHERE clause + +```sql +> SELECT * FROM "h2o_feet" WHERE "location" !~ /./ +> +``` + +The query selects all data from the `h2o_feet` measurement where the `location` +[tag](/enterprise_influxdb/v1.9/concepts/glossary#tag) has no value. +Every data [point](/enterprise_influxdb/v1.9/concepts/glossary#point) in the `NOAA_water_database` has a tag value for `location`. + +It's possible to perform this same query without a regular expression. +See the +[Frequently Asked Questions](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#how-do-i-select-data-with-a-tag-that-has-no-value) +document for more information. + +#### Use a regular expression to specify a tag with a value in the WHERE clause + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE "location" =~ /./ + +name: h2o_feet +time mean +---- ---- +1970-01-01T00:00:00Z 4.442107025822523 +``` + +The query uses an InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions/) +to calculate the average `water_level` across all data that have a tag value for +`location`. + +#### Use a regular expression to specify a field value in the WHERE clause + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE "location" = 'santa_monica' AND "level description" =~ /between/ + +name: h2o_feet +time mean +---- ---- +1970-01-01T00:00:00Z 4.47155532049926 +``` + +The query uses an InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions/) +to calculate the average `water_level` for all data where the field value of +`level description` includes the word `between`. + +#### Use a regular expression to specify tag keys in the GROUP BY clause + +```sql +> SELECT FIRST("index") FROM "h2o_quality" GROUP BY /l/ + +name: h2o_quality +tags: location=coyote_creek +time first +---- ----- +2015-08-18T00:00:00Z 41 + +name: h2o_quality +tags: location=santa_monica +time first +---- ----- +2015-08-18T00:00:00Z 99 +``` + +The query uses an InfluxQL [function](/enterprise_influxdb/v1.9/query_language/functions/) +to select the first value of `index` for every tag that includes the letter `l` +in its tag key. + +## Data types and cast operations + +The [`SELECT` clause](#the-basic-select-statement) supports specifying a [field's](/enterprise_influxdb/v1.9/concepts/glossary/#field) type and basic cast +operations with the `::` syntax. + + + + + + +
        Data TypesCast Operations
        + +## Data types + +[Field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value) can be floats, integers, strings, or booleans. +The `::` syntax allows users to specify the field's type in a query. + +> **Note:** Generally, it is not necessary to specify the field value +type in the [`SELECT` clause](#the-basic-select-statement). +In most cases, InfluxDB rejects any writes that attempt to write a [field value](/enterprise_influxdb/v1.9/concepts/glossary/#field-value) +to a field that previously accepted field values of a different type. +> +It is possible for field value types to differ across [shard groups](/enterprise_influxdb/v1.9/concepts/glossary/#shard-group). +In these cases, it may be necessary to specify the field value type in the +`SELECT` clause. +Please see the +[Frequently Asked Questions](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#how-does-influxdb-handle-field-type-discrepancies-across-shards) +document for more information on how InfluxDB handles field value type discrepancies. + +### Syntax + +```sql +SELECT_clause :: FROM_clause +``` + +`type` can be `float`, `integer`, `string`, or `boolean`. +In most cases, InfluxDB returns no data if the `field_key` does not store data of the specified +`type`. See [Cast Operations](#cast-operations) for more information. + +### Example + +```sql +> SELECT "water_level"::float FROM "h2o_feet" LIMIT 4 + +name: h2o_feet +-------------- +time water_level +2015-08-18T00:00:00Z 8.12 +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 8.005 +2015-08-18T00:06:00Z 2.116 +``` + +The query returns values of the `water_level` field key that are floats. + +## Cast operations + +The `::` syntax allows users to perform basic cast operations in queries. +Currently, InfluxDB supports casting [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value) from integers to +floats or from floats to integers. + +### Syntax + +```sql +SELECT_clause :: FROM_clause +``` + +`type` can be `float` or `integer`. + +InfluxDB returns no data if the query attempts to cast an integer or float to a +string or boolean. + +### Examples + +#### Cast float field values to integers + +```sql +> SELECT "water_level"::integer FROM "h2o_feet" LIMIT 4 + +name: h2o_feet +-------------- +time water_level +2015-08-18T00:00:00Z 8 +2015-08-18T00:00:00Z 2 +2015-08-18T00:06:00Z 8 +2015-08-18T00:06:00Z 2 +``` + +The query returns the integer form of `water_level`'s float [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). + +#### Cast float field values to strings (this functionality is not supported) + +```sql +> SELECT "water_level"::string FROM "h2o_feet" LIMIT 4 +> +``` + +The query returns no data as casting a float field value to a string is not +yet supported. + +## Merge behavior + +In InfluxDB, queries merge [series](/enterprise_influxdb/v1.9/concepts/glossary/#series) +automatically. + +### Example + +The `h2o_feet` [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) in the `NOAA_water_database` is part of two [series](/enterprise_influxdb/v1.9/concepts/glossary/#series). +The first series is made up of the `h2o_feet` measurement and the `location = coyote_creek` [tag](/enterprise_influxdb/v1.9/concepts/glossary/#tag). +The second series is made of up the `h2o_feet` measurement and the `location = santa_monica` tag. + +The following query automatically merges those two series when it calculates the [average](/enterprise_influxdb/v1.9/query_language/functions/#mean) `water_level`: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" + +name: h2o_feet +-------------- +time mean +1970-01-01T00:00:00Z 4.442107025822521 +``` + +If you want the average `water_level` for the first series only, specify the relevant tag in the [`WHERE` clause](#the-where-clause): + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE "location" = 'coyote_creek' + +name: h2o_feet +-------------- +time mean +1970-01-01T00:00:00Z 5.359342451341401 +``` + +If you want the average `water_level` for each individual series, include a [`GROUP BY` clause](#group-by-tags): + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" GROUP BY "location" + +name: h2o_feet +tags: location=coyote_creek +time mean +---- ---- +1970-01-01T00:00:00Z 5.359342451341401 + +name: h2o_feet +tags: location=santa_monica +time mean +---- ---- +1970-01-01T00:00:00Z 3.530863470081006 +``` + +## Multiple statements + +Separate multiple [`SELECT` statements](#the-basic-select-statement) in a query with a semicolon (`;`). + +### Examples + +{{< tabs-wrapper >}} +{{% tabs %}} +[Example 1: CLI](#) +[Example 2: InfluxDB API](#) +{{% /tabs %}} + +{{% tab-content %}} + +In the InfluxDB [CLI](/enterprise_influxdb/v1.9/tools/use-influx/): + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet"; SELECT "water_level" FROM "h2o_feet" LIMIT 2 + +name: h2o_feet +time mean +---- ---- +1970-01-01T00:00:00Z 4.442107025822522 + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 8.12 +2015-08-18T00:00:00Z 2.064 +``` + +{{% /tab-content %}} + +{{% tab-content %}} + +With the [InfluxDB API](/enterprise_influxdb/v1.9/tools/api/): + +```json +{ + "results": [ + { + "statement_id": 0, + "series": [ + { + "name": "h2o_feet", + "columns": [ + "time", + "mean" + ], + "values": [ + [ + "1970-01-01T00:00:00Z", + 4.442107025822522 + ] + ] + } + ] + }, + { + "statement_id": 1, + "series": [ + { + "name": "h2o_feet", + "columns": [ + "time", + "water_level" + ], + "values": [ + [ + "2015-08-18T00:00:00Z", + 8.12 + ], + [ + "2015-08-18T00:00:00Z", + 2.064 + ] + ] + } + ] + } + ] +} +``` + +{{% /tab-content %}} +{{< /tabs-wrapper >}} + +## Subqueries + +A subquery is a query that is nested in the `FROM` clause of another query. +Use a subquery to apply a query as a condition in the enclosing query. +Subqueries offer functionality similar to nested functions and SQL +[`HAVING` clauses](https://en.wikipedia.org/wiki/Having_%28SQL%29). + +### Syntax + +```sql +SELECT_clause FROM ( SELECT_statement ) [...] +``` + +InfluxDB performs the subquery first and the main query second. + +The main query surrounds the subquery and requires at least the [`SELECT` clause](#the-basic-select-statement) and the [`FROM` clause](#the-basic-select-statement). +The main query supports all clauses listed in this document. + +The subquery appears in the main query's `FROM` clause, and it requires surrounding parentheses. +The subquery supports all clauses listed in this document. + +InfluxQL supports multiple nested subqueries per main query. +Sample syntax for multiple subqueries: + +```sql +SELECT_clause FROM ( SELECT_clause FROM ( SELECT_statement ) [...] ) [...] +``` + +### Examples + +#### Calculate the [`SUM()`](/enterprise_influxdb/v1.9/query_language/functions/#sum) of several [`MAX()`](/enterprise_influxdb/v1.9/query_language/functions/#max) values + +```sql +> SELECT SUM("max") FROM (SELECT MAX("water_level") FROM "h2o_feet" GROUP BY "location") + +name: h2o_feet +time sum +---- --- +1970-01-01T00:00:00Z 17.169 +``` + +The query returns the sum of the maximum `water_level` values across every tag value of `location`. + +InfluxDB first performs the subquery; it calculates the maximum value of `water_level` for each tag value of `location`: + +```sql +> SELECT MAX("water_level") FROM "h2o_feet" GROUP BY "location" +name: h2o_feet + +tags: location=coyote_creek +time max +---- --- +2015-08-29T07:24:00Z 9.964 + +name: h2o_feet +tags: location=santa_monica +time max +---- --- +2015-08-29T03:54:00Z 7.205 +``` + +Next, InfluxDB performs the main query and calculates the sum of those maximum values: `9.964` + `7.205` = `17.169`. +Notice that the main query specifies `max`, not `water_level`, as the field key in the `SUM()` function. + +#### Calculate the [`MEAN()`](/enterprise_influxdb/v1.9/query_language/functions/#mean) difference between two fields + +```sql +> SELECT MEAN("difference") FROM (SELECT "cats" - "dogs" AS "difference" FROM "pet_daycare") + +name: pet_daycare +time mean +---- ---- +1970-01-01T00:00:00Z 1.75 +``` + +The query returns the average of the differences between the number of `cats` and `dogs` in the `pet_daycare` measurement. + +InfluxDB first performs the subquery. +The subquery calculates the difference between the values in the `cats` field and the values in the `dogs` field, +and it names the output column `difference`: + +```sql +> SELECT "cats" - "dogs" AS "difference" FROM "pet_daycare" + +name: pet_daycare +time difference +---- ---------- +2017-01-20T00:55:56Z -1 +2017-01-21T00:55:56Z -49 +2017-01-22T00:55:56Z 66 +2017-01-23T00:55:56Z -9 +``` + +Next, InfluxDB performs the main query and calculates the average of those differences. +Notice that the main query specifies `difference` as the field key in the `MEAN()` function. + +#### Calculate several [`MEAN()`](/enterprise_influxdb/v1.9/query_language/functions/#mean) values and place a condition on those mean values + +```sql +> SELECT "all_the_means" FROM (SELECT MEAN("water_level") AS "all_the_means" FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' GROUP BY time(12m) ) WHERE "all_the_means" > 5 + +name: h2o_feet +time all_the_means +---- ------------- +2015-08-18T00:00:00Z 5.07625 +``` + +The query returns all mean values of the `water_level` field that are greater than five. + +InfluxDB first performs the subquery. +The subquery calculates `MEAN()` values of `water_level` from `2015-08-18T00:00:00Z` through `2015-08-18T00:30:00Z` and groups the results into 12-minute intervals. +It also names the output column `all_the_means`: + +```sql +> SELECT MEAN("water_level") AS "all_the_means" FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' GROUP BY time(12m) + +name: h2o_feet +time all_the_means +---- ------------- +2015-08-18T00:00:00Z 5.07625 +2015-08-18T00:12:00Z 4.950749999999999 +2015-08-18T00:24:00Z 4.80675 +``` + +Next, InfluxDB performs the main query and returns only those mean values that are greater than five. +Notice that the main query specifies `all_the_means` as the field key in the `SELECT` clause. + +#### Calculate the [`SUM()`](/enterprise_influxdb/v1.9/query_language/functions/#sum) of several [`DERIVATIVE()`](/enterprise_influxdb/v1.9/query_language/functions/#derivative) values + +```sql +> SELECT SUM("water_level_derivative") AS "sum_derivative" FROM (SELECT DERIVATIVE(MEAN("water_level")) AS "water_level_derivative" FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' GROUP BY time(12m),"location") GROUP BY "location" + +name: h2o_feet +tags: location=coyote_creek +time sum_derivative +---- -------------- +1970-01-01T00:00:00Z -0.4950000000000001 + +name: h2o_feet +tags: location=santa_monica +time sum_derivative +---- -------------- +1970-01-01T00:00:00Z -0.043999999999999595 +``` + +The query returns the sum of the derivative of average `water_level` values for each tag value of `location`. + +InfluxDB first performs the subquery. +The subquery calculates the derivative of average `water_level` values taken at 12-minute intervals. +It performs that calculation for each tag value of `location` and names the output column `water_level_derivative`: + +```sql +> SELECT DERIVATIVE(MEAN("water_level")) AS "water_level_derivative" FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' GROUP BY time(12m),"location" + +name: h2o_feet +tags: location=coyote_creek +time water_level_derivative +---- ---------------------- +2015-08-18T00:12:00Z -0.23800000000000043 +2015-08-18T00:24:00Z -0.2569999999999997 + +name: h2o_feet +tags: location=santa_monica +time water_level_derivative +---- ---------------------- +2015-08-18T00:12:00Z -0.0129999999999999 +2015-08-18T00:24:00Z -0.030999999999999694 +``` + +Next, InfluxDB performs the main query and calculates the sum of the `water_level_derivative` values for each tag value of `location`. +Notice that the main query specifies `water_level_derivative`, not `water_level` or `derivative`, as the field key in the `SUM()` function. + +### Common issues with subqueries + +#### Multiple SELECT statements in a subquery + +InfluxQL supports multiple nested subqueries per main query: + +```sql +SELECT_clause FROM ( SELECT_clause FROM ( SELECT_statement ) [...] ) [...] + ------------------ ---------------- + Subquery 1 Subquery 2 +``` + +InfluxQL does not support multiple [`SELECT` statements](#the-basic-select-statement) per subquery: + +```sql +SELECT_clause FROM (SELECT_statement; SELECT_statement) [...] +``` + +The system returns a parsing error if a subquery includes multiple `SELECT` statements. diff --git a/content/enterprise_influxdb/v1.9/query_language/explore-schema.md b/content/enterprise_influxdb/v1.9/query_language/explore-schema.md new file mode 100644 index 000000000..f4bd54b67 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/query_language/explore-schema.md @@ -0,0 +1,1289 @@ +--- +title: Explore your schema using InfluxQL +description: Useful query syntax for exploring schema in InfluxQL. +menu: + enterprise_influxdb_1_9: + name: Explore your schema + weight: 30 + parent: InfluxQL +aliases: + - /enterprise_influxdb/v1.9/query_language/schema_exploration/ +v2: /influxdb/v2.0/query-data/flux/explore-schema/ +--- + +InfluxQL is an SQL-like query language for interacting with data in InfluxDB. +The following sections cover useful query syntax for exploring your [schema](/enterprise_influxdb/v1.9/concepts/glossary/#schema). + + + + + + + + + + + + + + + + + +
        SHOW DATABASESSHOW RETENTION POLICIESSHOW SERIES
        SHOW MEASUREMENTSSHOW TAG KEYSSHOW TAG VALUES
        SHOW FIELD KEYSFilter meta queries by time
        + +**Sample data** + +The data used in this document are available for download on the [Sample Data](/enterprise_influxdb/v1.9/query_language/data_download/) page. + +Before proceeding, login to the Influx CLI. + +```bash +$ influx -precision rfc3339 +Connected to http://localhost:8086 version 1.4.x +InfluxDB shell 1.4.x +> +``` + +## `SHOW DATABASES` +Returns a list of all [databases](/enterprise_influxdb/v1.9/concepts/glossary/#database) on your instance. + +### Syntax + +```sql +SHOW DATABASES +``` + +### Examples + +#### Run a `SHOW DATABASES` query + +```sql +> SHOW DATABASES + +name: databases +name +---- +NOAA_water_database +_internal +``` + +The query returns database names in a tabular format. +This InfluxDB instance has two databases: `NOAA_water_database` and `_internal`. + +## `SHOW RETENTION POLICIES` + +Returns a list of [retention policies](/enterprise_influxdb/v1.9/concepts/glossary/#retention-policy-rp) for the specified [database](/enterprise_influxdb/v1.9/concepts/glossary/#database). + +### Syntax + +```sql +SHOW RETENTION POLICIES [ON ] +``` + +### Description of syntax + +`ON ` is optional. +If the query does not include `ON `, you must specify the +database with `USE ` in the [CLI](/enterprise_influxdb/v1.9/tools/use-influx/) or with the `db` query +string parameter in the [InfluxDB API](/enterprise_influxdb/v1.9/tools/api/#query-string-parameters) request. + +### Examples + +#### Run a `SHOW RETENTION POLICIES` query with the `ON` clause + +```sql +> SHOW RETENTION POLICIES ON NOAA_water_database + +name duration shardGroupDuration replicaN default +---- -------- ------------------ -------- ------- +autogen 0s 168h0m0s 1 true +``` + +The query returns the list of retention policies in the `NOAA_water_database` +database in tabular format. +The database has one retention policy called `autogen`. +The `autogen` retention policy has an infinite [duration](/enterprise_influxdb/v1.9/concepts/glossary/#duration), +a seven-day [shard group duration](/enterprise_influxdb/v1.9/concepts/glossary/#shard-group), +a [replication factor](/enterprise_influxdb/v1.9/concepts/glossary/#replication-factor) +of one, and it is the `DEFAULT` retention policy for the database. + +#### Run a `SHOW RETENTION POLICIES` query without the `ON` clause + +{{< tabs-wrapper >}} +{{% tabs %}} +[CLI](#) +[InfluxDB API](#) +{{% /tabs %}} +{{% tab-content %}} + +Specify the database with `USE ` + +```sql +> USE NOAA_water_database +Using database NOAA_water_database + +> SHOW RETENTION POLICIES + +name duration shardGroupDuration replicaN default +---- -------- ------------------ -------- ------- +autogen 0s 168h0m0s 1 true +``` + +{{% /tab-content %}} + +{{% tab-content %}} + +Specify the database with the `db` query string parameter: + +```bash +~# curl -G "http://localhost:8086/query?db=NOAA_water_database&pretty=true" --data-urlencode "q=SHOW RETENTION POLICIES" + +{ + "results": [ + { + "statement_id": 0, + "series": [ + { + "columns": [ + "name", + "duration", + "shardGroupDuration", + "replicaN", + "default" + ], + "values": [ + [ + "autogen", + "0s", + "168h0m0s", + 1, + true + ] + ] + } + ] + } + ] +} +``` + +{{% /tab-content %}} +{{< /tabs-wrapper >}} + +## `SHOW SERIES` + +Returns a list of [series](/enterprise_influxdb/v1.9/concepts/glossary/#series) for +the specified [database](/enterprise_influxdb/v1.9/concepts/glossary/#database). + +### Syntax + +```sql +SHOW SERIES [ON ] [FROM_clause] [WHERE [ '' | ]] [LIMIT_clause] [OFFSET_clause] +``` + +### Description of syntax + +`ON ` is optional. +If the query does not include `ON `, you must specify the +database with `USE ` in the [CLI](/enterprise_influxdb/v1.9/tools/use-influx/) or with the `db` query +string parameter in the [InfluxDB API](/enterprise_influxdb/v1.9/tools/api/#query-string-parameters) request. + +The `FROM`, `WHERE`, `LIMIT`, and `OFFSET` clauses are optional. +The `WHERE` clause supports tag comparisons; field comparisons are not +valid for the `SHOW SERIES` query. + +Supported operators in the `WHERE` clause: +`=`   equal to +`<>` not equal to +`!=` not equal to +`=~` matches against +`!~` doesn't match against + +See the Data Exploration page for documentation on the +[`FROM` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#the-basic-select-statement), +[`LIMIT` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-clause), +[`OFFSET` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-clause), +and on [Regular Expressions in Queries](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +### Examples + +#### Run a `SHOW SERIES` query with the `ON` clause + +```sql +// Returns series for all shards in the database +> SHOW SERIES ON NOAA_water_database + +key +--- +average_temperature,location=coyote_creek +average_temperature,location=santa_monica +h2o_feet,location=coyote_creek +h2o_feet,location=santa_monica +h2o_pH,location=coyote_creek +h2o_pH,location=santa_monica +h2o_quality,location=coyote_creek,randtag=1 +h2o_quality,location=coyote_creek,randtag=2 +h2o_quality,location=coyote_creek,randtag=3 +h2o_quality,location=santa_monica,randtag=1 +h2o_quality,location=santa_monica,randtag=2 +h2o_quality,location=santa_monica,randtag=3 +h2o_temperature,location=coyote_creek +h2o_temperature,location=santa_monica +``` + +The query's output is similar to the [line protocol](/enterprise_influxdb/v1.9/concepts/glossary/#influxdb-line-protocol) format. +Everything before the first comma is the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) name. +Everything after the first comma is either a [tag key](/enterprise_influxdb/v1.9/concepts/glossary/#tag-key) or a [tag value](/enterprise_influxdb/v1.9/concepts/glossary/#tag-value). +The `NOAA_water_database` has five different measurements and 14 different series. + +#### Run a `SHOW SERIES` query without the `ON` clause + +{{< tabs-wrapper >}} +{{% tabs %}} +[CLI](#) +[InfluxDB API](#) +{{% /tabs %}} +{{% tab-content %}} + +Specify the database with `USE ` + +```sql +> USE NOAA_water_database +Using database NOAA_water_database + +> SHOW SERIES + +key +--- +average_temperature,location=coyote_creek +average_temperature,location=santa_monica +h2o_feet,location=coyote_creek +h2o_feet,location=santa_monica +h2o_pH,location=coyote_creek +h2o_pH,location=santa_monica +h2o_quality,location=coyote_creek,randtag=1 +h2o_quality,location=coyote_creek,randtag=2 +h2o_quality,location=coyote_creek,randtag=3 +h2o_quality,location=santa_monica,randtag=1 +h2o_quality,location=santa_monica,randtag=2 +h2o_quality,location=santa_monica,randtag=3 +h2o_temperature,location=coyote_creek +h2o_temperature,location=santa_monica +``` + +{{% /tab-content %}} + +{{% tab-content %}} + +Specify the database with the `db` query string parameter: + +```bash +~# curl -G "http://localhost:8086/query?db=NOAA_water_database&pretty=true" --data-urlencode "q=SHOW SERIES" + +{ + "results": [ + { + "statement_id": 0, + "series": [ + { + "columns": [ + "key" + ], + "values": [ + [ + "average_temperature,location=coyote_creek" + ], + [ + "average_temperature,location=santa_monica" + ], + [ + "h2o_feet,location=coyote_creek" + ], + [ + "h2o_feet,location=santa_monica" + ], + [ + "h2o_pH,location=coyote_creek" + ], + [ + "h2o_pH,location=santa_monica" + ], + [ + "h2o_quality,location=coyote_creek,randtag=1" + ], + [ + "h2o_quality,location=coyote_creek,randtag=2" + ], + [ + "h2o_quality,location=coyote_creek,randtag=3" + ], + [ + "h2o_quality,location=santa_monica,randtag=1" + ], + [ + "h2o_quality,location=santa_monica,randtag=2" + ], + [ + "h2o_quality,location=santa_monica,randtag=3" + ], + [ + "h2o_temperature,location=coyote_creek" + ], + [ + "h2o_temperature,location=santa_monica" + ] + ] + } + ] + } + ] +} +``` + +{{% /tab-content %}} +{{< /tabs-wrapper >}} + +#### Run a `SHOW SERIES` query with several clauses + +```sql +> SHOW SERIES ON NOAA_water_database FROM "h2o_quality" WHERE "location" = 'coyote_creek' LIMIT 2 + +key +--- +h2o_quality,location=coyote_creek,randtag=1 +h2o_quality,location=coyote_creek,randtag=2 +``` + +The query returns all series in the `NOAA_water_database` database that are +associated with the `h2o_quality` measurement and the tag `location = coyote_creek`. +The `LIMIT` clause limits the number of series returned to two. + +#### Run a `SHOW SERIES` query limited by time + +Limit series returned within a specified shard group duration. + +```sql +// Returns all series in the current shard. +> SHOW SERIES ON NOAA_water_database WHERE time > now() - 1m + +key +--- +average_temperature,location=coyote_creek +h2o_feet,location=coyote_creek +h2o_pH,location=coyote_creek +h2o_quality,location=coyote_creek,randtag=1 +h2o_quality,location=coyote_creek,randtag=2 +h2o_quality,location=coyote_creek,randtag=3 +h2o_temperature,location=coyote_creek +``` + +The query above returns all series in the `NOAA_water_database` database in the current shard group. The `WHERE` clause limits results to series in the shard group that contain a timestamp in the last minute. Note, if a shard group duration is 7 days, results returned may be up to 7 days old. + +```sql +// Returns all series in shard groups that contain a timestamp in the last 28 days. +> SHOW SERIES ON NOAA_water_database WHERE time > now() - 28d + +key +--- +average_temperature,location=coyote_creek +average_temperature,location=santa_monica +h2o_feet,location=coyote_creek +h2o_feet,location=santa_monica +h2o_pH,location=coyote_creek +h2o_pH,location=santa_monica +h2o_quality,location=coyote_creek,randtag=1 +h2o_quality,location=coyote_creek,randtag=2 +h2o_quality,location=coyote_creek,randtag=3 +h2o_quality,location=santa_monica,randtag=1 +h2o_quality,location=santa_monica,randtag=2 +h2o_quality,location=santa_monica,randtag=3 +h2o_temperature,location=coyote_creek +h2o_temperature,location=santa_monica +``` + +Note, if the specified shard group duration is 7 days, the query above returns series for the last 3 or 4 shards. + +## `SHOW MEASUREMENTS` + +Returns a list of [measurements](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) +for the specified [database](/enterprise_influxdb/v1.9/concepts/glossary/#database). + +### Syntax + +```sql +SHOW MEASUREMENTS [ON ] [WITH MEASUREMENT ['' | ]] [WHERE ['' | ]] [LIMIT_clause] [OFFSET_clause] +``` + +### Description of Syntax + +`ON ` is optional. +If the query does not include `ON `, you must specify the +database with `USE ` in the [CLI](/enterprise_influxdb/v1.9/tools/use-influx/) or with the `db` query +string parameter in the [InfluxDB API](/enterprise_influxdb/v1.9/tools/api/#query-string-parameters) request. + +The `WITH`, `WHERE`, `LIMIT` and `OFFSET` clauses are optional. +The `WHERE` clause supports tag comparisons; field comparisons are not valid for the `SHOW MEASUREMENTS` query. + +Supported operators in the `WHERE` clause: +`=`   equal to +`<>` not equal to +`!=` not equal to +`=~` matches against +`!~` doesn't match against + +See the Data Exploration page for documentation on the +[`LIMIT` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-clause), +[`OFFSET` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-clause), +and on [Regular expressions in queries](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +### Examples + +#### Run a `SHOW MEASUREMENTS` query with the `ON` clause + +```sql +> SHOW MEASUREMENTS ON NOAA_water_database + +name: measurements +name +---- +average_temperature +h2o_feet +h2o_pH +h2o_quality +h2o_temperature +``` + +The query returns the list of measurements in the `NOAA_water_database` +database. +The database has five measurements: `average_temperature`, `h2o_feet`, +`h2o_pH`, `h2o_quality`, and `h2o_temperature`. + +#### Run a `SHOW MEASUREMENTS` query without the `ON` clause + +{{< tabs-wrapper >}} +{{% tabs %}} +[CLI](#) +[InfluxDB API](#) +{{% /tabs %}} + +{{% tab-content %}} + +Specify the database with `USE ` + +```sql +> USE NOAA_water_database +Using database NOAA_water_database + +> SHOW MEASUREMENTS +name: measurements +name +---- +average_temperature +h2o_feet +h2o_pH +h2o_quality +h2o_temperature +``` + +{{% /tab-content %}} + +{{% tab-content %}} + +Specify the database with the `db` query string parameter: +``` +~# curl -G "http://localhost:8086/query?db=NOAA_water_database&pretty=true" --data-urlencode "q=SHOW MEASUREMENTS" + +{ + { + "results": [ + { + "statement_id": 0, + "series": [ + { + "name": "measurements", + "columns": [ + "name" + ], + "values": [ + [ + "average_temperature" + ], + [ + "h2o_feet" + ], + [ + "h2o_pH" + ], + [ + "h2o_quality" + ], + [ + "h2o_temperature" + ] + ] + } + ] + } + ] + } +``` + +{{% /tab-content %}} +{{< /tabs-wrapper >}} + +#### Run a `SHOW MEASUREMENTS` query with several clauses (i) + +```sql +> SHOW MEASUREMENTS ON NOAA_water_database WITH MEASUREMENT =~ /h2o.*/ LIMIT 2 OFFSET 1 + +name: measurements +name +---- +h2o_pH +h2o_quality +``` + +The query returns the measurements in the `NOAA_water_database` database that +start with `h2o`. +The `LIMIT` and `OFFSET` clauses limit the number of measurement names returned to +two and offset the results by one, skipping the `h2o_feet` measurement. + +#### Run a `SHOW MEASUREMENTS` query with several clauses (ii) + +```sql +> SHOW MEASUREMENTS ON NOAA_water_database WITH MEASUREMENT =~ /h2o.*/ WHERE "randtag" =~ /\d/ + +name: measurements +name +---- +h2o_quality +``` + +The query returns all measurements in the `NOAA_water_database` that start +with `h2o` and have values for the tag key `randtag` that include an integer. + +## `SHOW TAG KEYS` + +Returns a list of [tag keys](/enterprise_influxdb/v1.9/concepts/glossary/#tag-key) +associated with the specified [database](/enterprise_influxdb/v1.9/concepts/glossary/#database). + +### Syntax + +```sql +SHOW TAG KEYS [ON ] [FROM_clause] WITH KEY [ [ "" | ] | [IN (""," ['' | ]] [LIMIT_clause] [OFFSET_clause] +``` + +### Description of syntax + +`ON ` is optional. +If the query does not include `ON `, you must specify the +database with `USE ` in the [CLI](/enterprise_influxdb/v1.9/tools/use-influx/) or with the `db` query +string parameter in the [InfluxDB API](/enterprise_influxdb/v1.9/tools/api/#query-string-parameters) request. + +The `FROM` clause and the `WHERE` clause are optional. +The `WHERE` clause supports tag comparisons; field comparisons are not +valid for the `SHOW TAG KEYS` query. + +Supported operators in the `WHERE` clause: +`=`   equal to +`<>` not equal to +`!=` not equal to +`=~` matches against +`!~` doesn't match against + +See the Data Exploration page for documentation on the +[`FROM` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#the-basic-select-statement), +[`LIMIT` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-clause), +[`OFFSET` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-clause), +and on [Regular Expressions in Queries](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +### Examples + +#### Run a `SHOW TAG KEYS` query with the `ON` clause + +```sql +> SHOW TAG KEYS ON "NOAA_water_database" + +name: average_temperature +tagKey +------ +location + +name: h2o_feet +tagKey +------ +location + +name: h2o_pH +tagKey +------ +location + +name: h2o_quality +tagKey +------ +location +randtag + +name: h2o_temperature +tagKey +------ +location +``` + +The query returns the list of tag keys in the `NOAA_water_database` database. +The output groups tag keys by measurement name; +it shows that every measurement has the `location` tag key and that the +`h2o_quality` measurement has an additional `randtag` tag key. + +#### Run a `SHOW TAG KEYS` query without the `ON` clause + +{{< tabs-wrapper >}} +{{% tabs %}} +[CLI](#) +[InfluxDB API](#) +{{% /tabs %}} +{{% tab-content %}} + +Specify the database with `USE ` + +```sql +> USE NOAA_water_database +Using database NOAA_water_database + +> SHOW TAG KEYS + +name: average_temperature +tagKey +------ +location + +name: h2o_feet +tagKey +------ +location + +name: h2o_pH +tagKey +------ +location + +name: h2o_quality +tagKey +------ +location +randtag + +name: h2o_temperature +tagKey +------ +location +``` + +{{% /tab-content %}} + +{{% tab-content %}} + +Specify the database with the `db` query string parameter: + +```sql +~# curl -G "http://localhost:8086/query?db=NOAA_water_database&pretty=true" --data-urlencode "q=SHOW TAG KEYS" + +{ + "results": [ + { + "statement_id": 0, + "series": [ + { + "name": "average_temperature", + "columns": [ + "tagKey" + ], + "values": [ + [ + "location" + ] + ] + }, + { + "name": "h2o_feet", + "columns": [ + "tagKey" + ], + "values": [ + [ + "location" + ] + ] + }, + { + "name": "h2o_pH", + "columns": [ + "tagKey" + ], + "values": [ + [ + "location" + ] + ] + }, + { + "name": "h2o_quality", + "columns": [ + "tagKey" + ], + "values": [ + [ + "location" + ], + [ + "randtag" + ] + ] + }, + { + "name": "h2o_temperature", + "columns": [ + "tagKey" + ], + "values": [ + [ + "location" + ] + ] + } + ] + } + ] +} +``` + +{{% /tab-content %}} +{{< /tabs-wrapper >}} + +#### Run a `SHOW TAG KEYS` query with several clauses + +```sql +> SHOW TAG KEYS ON "NOAA_water_database" FROM "h2o_quality" LIMIT 1 OFFSET 1 + +name: h2o_quality +tagKey +------ +randtag +``` + +The query returns tag keys from the `h2o_quality` measurement in the +`NOAA_water_database` database. +The `LIMIT` and `OFFSET` clauses limit the number of tag keys returned to one +and offsets the results by one. + +#### Run a `SHOW TAG KEYS` query with a `WITH KEY IN` clause + +```sql +> SHOW TAG KEYS ON "telegraf" WITH KEY IN ("host","region") + +"measurement","tagKey" +"cpu","host" +"cpu_load_short","host" +"cpu_load_short","region" +"disk","host" +"diskio","host" +"docker","host" +"docker_container_blkio","host" +"docker_container_cpu","host" +"docker_container_mem","host" +"docker_container_status","host" +``` + +## `SHOW TAG VALUES` + +Returns the list of [tag values](/enterprise_influxdb/v1.9/concepts/glossary/#tag-value) +for the specified [tag key(s)](/enterprise_influxdb/v1.9/concepts/glossary/#tag-key) in the database. + +### Syntax + +```sql +SHOW TAG VALUES [ON ][FROM_clause] WITH KEY [ [ "" | ] | [IN (""," ['' | ]] [LIMIT_clause] [OFFSET_clause] +``` + +### Description of syntax + +`ON ` is optional. +If the query does not include `ON `, you must specify the +database with `USE ` in the [CLI](/enterprise_influxdb/v1.9/tools/use-influx/) or with the `db` query string parameter in the [InfluxDB API](/enterprise_influxdb/v1.9/tools/api/#query-string-parameters) request. + +The `WITH` clause is required. +It supports specifying a single tag key, a regular expression, and multiple tag keys. + +The `FROM`, `WHERE`, `LIMIT`, and `OFFSET` clauses are optional. +The `WHERE` clause supports tag comparisons; field comparisons are not +valid for the `SHOW TAG KEYS` query. + +Supported operators in the `WITH` and `WHERE` clauses: +`=`   equal to +`<>` not equal to +`!=` not equal to +`=~` matches against +`!~` doesn't match against + +See the Data Exploration page for documentation on the +[`FROM` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#the-basic-select-statement), +[`LIMIT` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-clause), +[`OFFSET` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-clause), +and on [Regular Expressions in Queries](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +### Examples + +#### Run a `SHOW TAG VALUES` query with the `ON` clause + +```sql +> SHOW TAG VALUES ON "NOAA_water_database" WITH KEY = "randtag" + +name: h2o_quality +key value +--- ----- +randtag 1 +randtag 2 +randtag 3 +``` + +The query returns all tag values of the `randtag` tag key in the `NOAA_water_database` +database. +`SHOW TAG VALUES` groups query results by measurement name. + +#### Run a `SHOW TAG VALUES` query without the `ON` clause + +{{< tabs-wrapper >}} +{{% tabs %}} +[CLI](#) +[InfluxDB API](#) +{{% /tabs %}} +{{% tab-content %}} + +Specify the database with `USE ` + +```sql +> USE NOAA_water_database +Using database NOAA_water_database + +> SHOW TAG VALUES WITH KEY = "randtag" + +name: h2o_quality +key value +--- ----- +randtag 1 +randtag 2 +randtag 3 +``` + +{{% /tab-content %}} + +{{% tab-content %}} + +Specify the database with the `db` query string parameter: + +```bash +~# curl -G "http://localhost:8086/query?db=NOAA_water_database&pretty=true" --data-urlencode 'q=SHOW TAG VALUES WITH KEY = "randtag"' + +{ + "results": [ + { + "statement_id": 0, + "series": [ + { + "name": "h2o_quality", + "columns": [ + "key", + "value" + ], + "values": [ + [ + "randtag", + "1" + ], + [ + "randtag", + "2" + ], + [ + "randtag", + "3" + ] + ] + } + ] + } + ] +} +``` + +{{% /tab-content %}} +{{< /tabs-wrapper >}} + +#### Run a `SHOW TAG VALUES` query with several clauses + +```sql +> SHOW TAG VALUES ON "NOAA_water_database" WITH KEY IN ("location","randtag") WHERE "randtag" =~ /./ LIMIT 3 + +name: h2o_quality +key value +--- ----- +location coyote_creek +location santa_monica +randtag 1 +``` + +The query returns the tag values of the tag keys `location` and `randtag` for +all measurements in the `NOAA_water_database` database where `randtag` has tag values. +The `LIMIT` clause limits the number of tag values returned to three. + +## `SHOW FIELD KEYS` + +Returns the [field keys](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) and the +[data type](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types) of their +[field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). + +### Syntax + +```sql +SHOW FIELD KEYS [ON ] [FROM ] +``` + +### Description of syntax + +`ON ` is optional. +If the query does not include `ON `, you must specify the +database with `USE ` in the [CLI](/enterprise_influxdb/v1.9/tools/use-influx/) or with the `db` query +string parameter in the [InfluxDB API](/enterprise_influxdb/v1.9/tools/api/#query-string-parameters) request. + +The `FROM` clause is also optional. +See the Data Exploration page for documentation on the +[`FROM` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#the-basic-select-statement). + +> **Note:** A field's data type [can differ](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#how-does-influxdb-handle-field-type-discrepancies-across-shards) across +[shards](/enterprise_influxdb/v1.9/concepts/glossary/#shard). +If your field has more than one type, `SHOW FIELD KEYS` returns the type that +occurs first in the following list: float, integer, string, boolean. + +### Examples + +#### Run a `SHOW FIELD KEYS` query with the `ON` clause + +```sql +> SHOW FIELD KEYS ON "NOAA_water_database" + +name: average_temperature +fieldKey fieldType +-------- --------- +degrees float + +name: h2o_feet +fieldKey fieldType +-------- --------- +level description string +water_level float + +name: h2o_pH +fieldKey fieldType +-------- --------- +pH float + +name: h2o_quality +fieldKey fieldType +-------- --------- +index float + +name: h2o_temperature +fieldKey fieldType +-------- --------- +degrees float +``` + +The query returns the field keys and field value data types for each +measurement in the `NOAA_water_database` database. + +#### Run a `SHOW FIELD KEYS` query without the `ON` clause + +{{< tabs-wrapper >}} +{{% tabs %}} +[CLI](#) +[InfluxDB API](#) +{{% /tabs %}} +{{% tab-content %}} + +Specify the database with `USE ` + +```sql +> USE NOAA_water_database +Using database NOAA_water_database + +> SHOW FIELD KEYS + +name: average_temperature +fieldKey fieldType +-------- --------- +degrees float + +name: h2o_feet +fieldKey fieldType +-------- --------- +level description string +water_level float + +name: h2o_pH +fieldKey fieldType +-------- --------- +pH float + +name: h2o_quality +fieldKey fieldType +-------- --------- +index float + +name: h2o_temperature +fieldKey fieldType +-------- --------- +degrees float +``` + +{{% /tab-content %}} + +{{% tab-content %}} + +Specify the database with the `db` query string parameter: + +```bash +~# curl -G "http://localhost:8086/query?db=NOAA_water_database&pretty=true" --data-urlencode 'q=SHOW FIELD KEYS' + +{ + "results": [ + { + "statement_id": 0, + "series": [ + { + "name": "average_temperature", + "columns": [ + "fieldKey", + "fieldType" + ], + "values": [ + [ + "degrees", + "float" + ] + ] + }, + { + "name": "h2o_feet", + "columns": [ + "fieldKey", + "fieldType" + ], + "values": [ + [ + "level description", + "string" + ], + [ + "water_level", + "float" + ] + ] + }, + { + "name": "h2o_pH", + "columns": [ + "fieldKey", + "fieldType" + ], + "values": [ + [ + "pH", + "float" + ] + ] + }, + { + "name": "h2o_quality", + "columns": [ + "fieldKey", + "fieldType" + ], + "values": [ + [ + "index", + "float" + ] + ] + }, + { + "name": "h2o_temperature", + "columns": [ + "fieldKey", + "fieldType" + ], + "values": [ + [ + "degrees", + "float" + ] + ] + } + ] + } + ] +} +``` + +{{% /tab-content %}} +{{< /tabs-wrapper >}} + + +#### Run a `SHOW FIELD KEYS` query with the `FROM` clause + +```sql +> SHOW FIELD KEYS ON "NOAA_water_database" FROM "h2o_feet" + +name: h2o_feet +fieldKey fieldType +-------- --------- +level description string +water_level float +``` + +The query returns the fields keys and field value data types for the `h2o_feet` +measurement in the `NOAA_water_database` database. + +### Common Issues with `SHOW FIELD KEYS` + +#### SHOW FIELD KEYS and field type discrepancies + +Field value +[data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types) +cannot differ within a [shard](/enterprise_influxdb/v1.9/concepts/glossary/#shard) but they +can differ across shards. +`SHOW FIELD KEYS` returns every data type, across every shard, associated with +the field key. + +##### Example + +The `all_the_types` field stores four different data types: + +```sql +> SHOW FIELD KEYS + +name: mymeas +fieldKey fieldType +-------- --------- +all_the_types integer +all_the_types float +all_the_types string +all_the_types boolean +``` + +Note that `SHOW FIELD KEYS` handles field type discrepancies differently from +`SELECT` statements. +For more information, see the +[How does InfluxDB handle field type discrepancies across shards?](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#how-does-influxdb-handle-field-type-discrepancies-across-shards). + +### Filter meta queries by time + +When you filter meta queries by time, you may see results outside of your specified time. Meta query results are filtered at the shard level, so results can be approximately as granular as your shard group duration. If your time filter spans multiple shards, you'll get results from all shards with points in the specified time range. To review your shards and timestamps on points in the shard, run `SHOW SHARDS`. To learn more about shards and their duration, see [recommended shard groups durations](/enterprise_influxdb/v1.9/concepts/schema_and_data_layout/#shard-group-duration-recommendations). + +The example below shows how to filter `SHOW TAG KEYS` by approximately one hour using a 1h shard group duration. To filter other meta data, replace `SHOW TAG KEYS` with `SHOW TAG VALUES`, `SHOW SERIES`, `SHOW FIELD KEYS`, and so on. + +> **Note:** `SHOW MEASUREMENTS` cannot be filtered by time. + +#### Example filtering `SHOW TAG KEYS` by time + +1. Specify a shard duration on a new database or [alter an existing shard duration](/enterprise_influxdb/v1.9/query_language/manage-database/#modify-retention-policies-with-alter-retention-policy). To specify a 1h shard duration when creating a new database, run the following command: + + ```sh + > CREATE database mydb with duration 7d REPLICATION 1 SHARD DURATION 1h name myRP; + ``` + + > **Note:** The minimum shard duration is 1h. + +2. Verify the shard duration has the correct time interval (precision) by running the `SHOW SHARDS` command. The example below shows a shard duration with an hour precision. + + ```sh + > SHOW SHARDS + name: mydb + id database retention_policy shard_group start_time end_time expiry_time owners + -- -------- ---------------- ----------- ---------- -------- ----------- ------ + > precision h + ``` + +3. (Optional) Insert sample tag keys. This step is for demonstration purposes. If you already have tag keys (or other meta data) to search for, skip this step. + + ```sh + // Insert a sample tag called "test_key" into the "test" measurement, and then check the timestamp: + > INSERT test,test_key=hello value=1 + + > select * from test + name: test + time test_key value + ---- -------- ----- + 434820 hello 1 + + // Add new tag keys with timestamps one, two, and three hours earlier: + + > INSERT test,test_key_1=hello value=1 434819 + > INSERT test,test_key_2=hello value=1 434819 + > INSERT test,test_key_3_=hello value=1 434818 + > INSERT test,test_key_4=hello value=1 434817 + > INSERT test,test_key_5_=hello value=1 434817 + ``` + +4. To find tag keys within a shard duration, run one of the following commands: + + `SHOW TAG KEYS ON database-name ` OR + + `SELECT * FROM measurement ` + + The examples below use test data from step 3. + ```sh + //Using data from Step 3, show tag keys between now and an hour ago + > SHOW TAG KEYS ON mydb where time > now() -1h and time < now() + name: test + tagKey + ------ + test_key + test_key_1 + test_key_2 + + // Find tag keys between one and two hours ago + > SHOW TAG KEYS ON mydb where > time > now() -2h and time < now()-1h + name: test + tagKey + ------ + test_key_1 + test_key_2 + test_key_3 + + // Find tag keys between two and three hours ago + > SHOW TAG KEYS ON mydb where > time > now() -3h and time < now()-2h + name: test + tagKey + ------ + test_key_3 + test_key_4 + test_key_5 + + // For a specified measurement, find tag keys in a given shard by specifying the time boundaries of the shard + > SELECT * FROM test WHERE time >= '2019-08-09T00:00:00Z' and time < '2019-08-09T10:00:00Z' + name: test + time test_key_4 test_key_5 value + ---- ------------ ------------ ----- + 434817 hello 1 + 434817 hello 1 + + // For a specified database, find tag keys in a given shard by specifying the time boundaries of the shard + > SHOW TAG KEYS ON mydb WHERE time >= '2019-08-09T00:00:00Z' and time < '2019-08-09T10:00:00Z' + name: test + tagKey + ------ + test_key_4 + test_key_5 + ``` diff --git a/content/enterprise_influxdb/v1.9/query_language/functions.md b/content/enterprise_influxdb/v1.9/query_language/functions.md new file mode 100644 index 000000000..f7bbded77 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/query_language/functions.md @@ -0,0 +1,7197 @@ +--- +title: InfluxQL functions +description: > + Aggregate, select, transform, and predict data with InfluxQL functions. +menu: + enterprise_influxdb_1_9: + name: Functions + weight: 60 + parent: InfluxQL +--- + +Aggregate, select, transform, and predict data with InfluxQL functions. + +#### Content + +* [Aggregations](#aggregations) + * [COUNT()](#count) + * [DISTINCT()](#distinct) + * [INTEGRAL()](#integral) + * [MEAN()](#mean) + * [MEDIAN()](#median) + * [MODE()](#mode) + * [SPREAD()](#spread) + * [STDDEV()](#stddev) + * [SUM()](#sum) +* [Selectors](#selectors) + * [BOTTOM()](#bottom) + * [FIRST()](#first) + * [LAST()](#last) + * [MAX()](#max) + * [MIN()](#min) + * [PERCENTILE()](#percentile) + * [SAMPLE()](#sample) + * [TOP()](#top) +* [Transformations](#transformations) + * [ABS()](#abs) + * [ACOS()](#acos) + * [ASIN()](#asin) + * [ATAN()](#atan) + * [ATAN2()](#atan2) + * [CEIL()](#ceil) + * [COS()](#cos) + * [CUMULATIVE_SUM()](#cumulative-sum) + * [DERIVATIVE()](#derivative) + * [DIFFERENCE()](#difference) + * [ELAPSED()](#elapsed) + * [EXP()](#exp) + * [FLOOR()](#floor) + * [HISTOGRAM()](#histogram) + * [LN()](#ln) + * [LOG()](#log) + * [LOG2()](#log2) + * [LOG10()](#log10) + * [MOVING_AVERAGE()](#moving-average) + * [NON_NEGATIVE_DERIVATIVE()](#non-negative-derivative) + * [NON_NEGATIVE_DIFFERENCE()](#non-negative-difference) + * [POW()](#pow) + * [ROUND()](#round) + * [SIN()](#sin) + * [SQRT()](#sqrt) + * [TAN()](#tan) +* [Predictors](#predictors) + * [HOLT_WINTERS()](#holt-winters) +* [Technical Analysis](#technical-analysis) + * [CHANDE_MOMENTUM_OSCILLATOR()](#chande-momentum-oscillator) + * [EXPONENTIAL_MOVING_AVERAGE()](#exponential-moving-average) + * [DOUBLE_EXPONENTIAL_MOVING_AVERAGE()](#double-exponential-moving-average) + * [KAUFMANS_EFFICIENCY_RATIO()](#kaufmans-efficiency-ratio) + * [KAUFMANS_ADAPTIVE_MOVING_AVERAGE()](#kaufmans-adaptive-moving-average) + * [TRIPLE_EXPONENTIAL_MOVING_AVERAGE()](#triple-exponential-moving-average) + * [TRIPLE_EXPONENTIAL_DERIVATIVE()](#triple-exponential-derivative) + * [RELATIVE_STRENGTH_INDEX()](#relative-strength-index) +* [Other](#other) + * [Sample Data](#sample-data) + * [General Syntax for Functions](#general-syntax-for-functions) + * [Specify Multiple Functions in the SELECT clause](#specify-multiple-functions-in-the-select-clause) + * [Rename the Output Field Key](#rename-the-output-field-key) + * [Change the Values Reported for Intervals with no Data](#change-the-values-reported-for-intervals-with-no-data) + * [Common Issues with Functions](#common-issues-with-functions) + +## Aggregations + +### COUNT() + +Returns the number of non-null [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). + +#### Syntax + +``` +SELECT COUNT( [ * | | // ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +##### Nested Syntax + +``` +SELECT COUNT(DISTINCT( [ * | | // ] )) [...] +``` + +`COUNT(field_key)` +Returns the number of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`COUNT(/regular_expression/)` +Returns the number of field values associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +`COUNT(*)` +Returns the number of field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`COUNT()` supports all field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). +InfluxQL supports nesting [`DISTINCT()`](#distinct) with `COUNT()`. + +#### Examples + +##### Count the field values associated with a field key + +```sql +> SELECT COUNT("water_level") FROM "h2o_feet" + +name: h2o_feet +time count +---- ----- +1970-01-01T00:00:00Z 15258 +``` + +The query returns the number of non-null field values in the `water_level` field key in the `h2o_feet` measurement. + +##### Count the field values associated with each field key in a measurement + +```sql +> SELECT COUNT(*) FROM "h2o_feet" + +name: h2o_feet +time count_level description count_water_level +---- ----------------------- ----------------- +1970-01-01T00:00:00Z 15258 15258 +``` + +The query returns the number of non-null field values for each field key associated with the `h2o_feet` measurement. +The `h2o_feet` measurement has two field keys: `level description` and `water_level`. + +##### Count the field values associated with each field key that matches a regular expression + +```sql +> SELECT COUNT(/water/) FROM "h2o_feet" + +name: h2o_feet +time count_water_level +---- ----------------- +1970-01-01T00:00:00Z 15258 +``` + +The query returns the number of non-null field values for every field key that contains the word `water` in the `h2o_feet` measurement. + +##### Count the field values associated with a field key and include several clauses + +```sql +> SELECT COUNT("water_level") FROM "h2o_feet" WHERE time >= '2015-08-17T23:48:00Z' AND time <= '2015-08-18T00:54:00Z' GROUP BY time(12m),* fill(200) LIMIT 7 SLIMIT 1 + +name: h2o_feet +tags: location=coyote_creek +time count +---- ----- +2015-08-17T23:48:00Z 200 +2015-08-18T00:00:00Z 2 +2015-08-18T00:12:00Z 2 +2015-08-18T00:24:00Z 2 +2015-08-18T00:36:00Z 2 +2015-08-18T00:48:00Z 2 +``` + +The query returns the number of non-null field values in the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-17T23:48:00Z` and `2015-08-18T00:54:00Z` and [groups](/enterprise_influxdb/v1.9/query_language/explore-data/#the-group-by-clause) results into 12-minute time intervals and per tag. +The query [fills](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals-and-fill) empty time intervals with `200` and [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points and series returned to seven and one. + +##### Count the distinct field values associated with a field key + +```sql +> SELECT COUNT(DISTINCT("level description")) FROM "h2o_feet" + +name: h2o_feet +time count +---- ----- +1970-01-01T00:00:00Z 4 +``` + +The query returns the number of unique field values for the `level description` field key and the `h2o_feet` measurement. + +### Common Issues with COUNT() + +#### COUNT() and fill() + +Most InfluxQL functions report `null` values for time intervals with no data, and +[`fill()`](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals-and-fill) +replaces that `null` value with the `fill_option`. +`COUNT()` reports `0` for time intervals with no data, and `fill()` replaces any `0` values with the `fill_option`. + +##### Example + +The first query in the codeblock below does not include `fill()`. +The last time interval has no data so the reported value for that time interval is zero. +The second query includes `fill(800000)`; it replaces the zero in the last interval with `800000`. + +```sql +> SELECT COUNT("water_level") FROM "h2o_feet" WHERE time >= '2015-09-18T21:24:00Z' AND time <= '2015-09-18T21:54:00Z' GROUP BY time(12m) + +name: h2o_feet +time count +---- ----- +2015-09-18T21:24:00Z 2 +2015-09-18T21:36:00Z 2 +2015-09-18T21:48:00Z 0 + +> SELECT COUNT("water_level") FROM "h2o_feet" WHERE time >= '2015-09-18T21:24:00Z' AND time <= '2015-09-18T21:54:00Z' GROUP BY time(12m) fill(800000) + +name: h2o_feet +time count +---- ----- +2015-09-18T21:24:00Z 2 +2015-09-18T21:36:00Z 2 +2015-09-18T21:48:00Z 800000 +``` + +### DISTINCT() + +Returns the list of unique [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). + +#### Syntax + +``` +SELECT DISTINCT( [ | // ] ) FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +##### Nested Syntax + +``` +SELECT COUNT(DISTINCT( [ | // ] )) [...] +``` + +`DISTINCT(field_key)` +Returns the unique field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`DISTINCT()` supports all field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). +InfluxQL supports nesting `DISTINCT()` with [`COUNT()`](#count). + +#### Examples + +##### List the distinct field values associated with a field key + +```sql +> SELECT DISTINCT("level description") FROM "h2o_feet" + +name: h2o_feet +time distinct +---- -------- +1970-01-01T00:00:00Z between 6 and 9 feet +1970-01-01T00:00:00Z below 3 feet +1970-01-01T00:00:00Z between 3 and 6 feet +1970-01-01T00:00:00Z at or greater than 9 feet +``` + +The query returns a tabular list of the unique field values in the `level description` field key in the `h2o_feet` measurement. + +##### List the distinct field values associated with each field key in a measurement + +```sql +> SELECT DISTINCT(*) FROM "h2o_feet" + +name: h2o_feet +time distinct_level description distinct_water_level +---- -------------------------- -------------------- +1970-01-01T00:00:00Z between 6 and 9 feet 8.12 +1970-01-01T00:00:00Z between 3 and 6 feet 8.005 +1970-01-01T00:00:00Z at or greater than 9 feet 7.887 +1970-01-01T00:00:00Z below 3 feet 7.762 +[...] +``` + +The query returns a tabular list of the unique field values for each field key in the `h2o_feet` measurement. +The `h2o_feet` measurement has two field keys: `level description` and `water_level`. + +#### List the distinct field values associated with a field key and include several clauses + +```sql +> SELECT DISTINCT("level description") FROM "h2o_feet" WHERE time >= '2015-08-17T23:48:00Z' AND time <= '2015-08-18T00:54:00Z' GROUP BY time(12m),* SLIMIT 1 + +name: h2o_feet +tags: location=coyote_creek +time distinct +---- -------- +2015-08-18T00:00:00Z between 6 and 9 feet +2015-08-18T00:12:00Z between 6 and 9 feet +2015-08-18T00:24:00Z between 6 and 9 feet +2015-08-18T00:36:00Z between 6 and 9 feet +2015-08-18T00:48:00Z between 6 and 9 feet +``` + +The query returns a tabular list of the unique field values in the `level description` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-17T23:48:00Z` and `2015-08-18T00:54:00Z` and [groups](/enterprise_influxdb/v1.9/query_language/explore-data/#the-group-by-clause) results into 12-minute time intervals and per tag. +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of series returned to one. + +##### Count the distinct field values associated with a field key + +```sql +> SELECT COUNT(DISTINCT("level description")) FROM "h2o_feet" + +name: h2o_feet +time count +---- ----- +1970-01-01T00:00:00Z 4 +``` + +The query returns the number of unique field values in the `level description` field key and the `h2o_feet` measurement. + +### Common Issues with DISTINCT() + +#### DISTINCT() and the INTO clause + +Using `DISTINCT()` with the [`INTO` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#the-into-clause) can cause InfluxDB to overwrite points in the destination measurement. +`DISTINCT()` often returns several results with the same timestamp; InfluxDB assumes [points](/enterprise_influxdb/v1.9/concepts/glossary/#point) with the same [series](/enterprise_influxdb/v1.9/concepts/glossary/#series) and timestamp are duplicate points and simply overwrites any duplicate point with the most recent point in the destination measurement. + +##### Example + +The first query in the codeblock below uses the `DISTINCT()` function and returns four results. +Notice that each result has the same timestamp. +The second query adds an `INTO` clause to the initial query and writes the query results to the `distincts` measurement. +The last query in the code block selects all the data in the `distincts` measurement. + +The last query returns one point because the four initial results are duplicate points; they belong to the same series and have the same timestamp. +When the system encounters duplicate points, it simply overwrites the previous point with the most recent point. + +```sql +> SELECT DISTINCT("level description") FROM "h2o_feet" + +name: h2o_feet +time distinct +---- -------- +1970-01-01T00:00:00Z below 3 feet +1970-01-01T00:00:00Z between 6 and 9 feet +1970-01-01T00:00:00Z between 3 and 6 feet +1970-01-01T00:00:00Z at or greater than 9 feet + +> SELECT DISTINCT("level description") INTO "distincts" FROM "h2o_feet" + +name: result +time written +---- ------- +1970-01-01T00:00:00Z 4 + +> SELECT * FROM "distincts" + +name: distincts +time distinct +---- -------- +1970-01-01T00:00:00Z at or greater than 9 feet +``` + +### INTEGRAL() + +Returns the area under the curve for subsequent [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). + +#### Syntax + +``` +SELECT INTEGRAL( [ * | | // ] [ , ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +InfluxDB calculates the area under the curve for subsequent field values and converts those results into the summed area per `unit`. +The `unit` argument is an integer followed by a [duration literal](/enterprise_influxdb/v1.9/query_language/spec/#literals) and it is optional. +If the query does not specify the `unit`, the unit defaults to one second (`1s`). + +`INTEGRAL(field_key)` +Returns the area under the curve for subsequent field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`INTEGRAL(/regular_expression/)` +Returns the area under the curve for subsequent field values associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +`INTEGRAL(*)` +Returns the average field value associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`INTEGRAL()` does not support [`fill()`](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals-and-fill). `INTEGRAL()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +#### Examples + +Examples 1-5 use the following subsample of the [`NOAA_water_database` data](/enterprise_influxdb/v1.9/query_language/data_download/): + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +2015-08-18T00:18:00Z 2.126 +2015-08-18T00:24:00Z 2.041 +2015-08-18T00:30:00Z 2.051 +``` + +##### Calculate the integral for the field values associated with a field key + +```sql +> SELECT INTEGRAL("water_level") FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' + +name: h2o_feet +time integral +---- -------- +1970-01-01T00:00:00Z 3732.66 +``` + +The query returns the area under the curve (in seconds) for the field values associated with the `water_level` field key and in the `h2o_feet` measurement. + +##### Calculate the integral for the field values associated with a field key and specify the unit option + +```sql +> SELECT INTEGRAL("water_level",1m) FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' + +name: h2o_feet +time integral +---- -------- +1970-01-01T00:00:00Z 62.211 +``` + +The query returns the area under the curve (in minutes) for the field values associated with the `water_level` field key and in the `h2o_feet` measurement. + +##### Calculate the integral for the field values associated with each field key in a measurement and specify the unit option + +```sql +> SELECT INTEGRAL(*,1m) FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' + +name: h2o_feet +time integral_water_level +---- -------------------- +1970-01-01T00:00:00Z 62.211 +``` + +The query returns the area under the curve (in minutes) for the field values associated with each field key that stores numerical values in the `h2o_feet` measurement. +The `h2o_feet` measurement has on numerical field: `water_level`. + +#### Calculate the integral for the field values associated with each field key that matches a regular expression and specify the unit option + +```sql +> SELECT INTEGRAL(/water/,1m) FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' + +name: h2o_feet +time integral_water_level +---- -------------------- +1970-01-01T00:00:00Z 62.211 +``` + +The query returns the area under the curve (in minutes) for the field values associated with each field key that stores numerical values includes the word `water` in the `h2o_feet` measurement. + +#### Calculate the integral for the field values associated with a field key and include several clauses + +```sql +> SELECT INTEGRAL("water_level",1m) FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' GROUP BY time(12m) LIMIT 1 + +name: h2o_feet +time integral +---- -------- +2015-08-18T00:00:00Z 24.972 +``` + +The query returns the area under the curve (in minutes) for the field values associated with the `water_level` field key and in the `h2o_feet` measurement. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-18T00:00:00Z` and `2015-08-18T00:30:00Z`, [groups](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) results into 12-minute intervals, and [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of results returned to one. + +### MEAN() + +Returns the arithmetic mean (average) of [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). + +#### Syntax + +``` +SELECT MEAN( [ * | | // ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`MEAN(field_key)` +Returns the average field value associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`MEAN(/regular_expression/)` +Returns the average field value associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +`MEAN(*)` +Returns the average field value associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`MEAN()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +#### Examples + +##### Calculate the mean field value associated with a field key + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" + +name: h2o_feet +time mean +---- ---- +1970-01-01T00:00:00Z 4.442107025822522 +``` +The query returns the average field value in the `water_level` field key in the `h2o_feet` measurement. + +##### Calculate the mean field value associated with each field key in a measurement + +```sql +> SELECT MEAN(*) FROM "h2o_feet" + +name: h2o_feet +time mean_water_level +---- ---------------- +1970-01-01T00:00:00Z 4.442107025822522 +``` +The query returns the average field value for every field key that stores numerical values in the `h2o_feet` measurement. +The `h2o_feet` measurement has one numerical field: `water_level`. + +##### Calculate the mean field value associated with each field key that matches a regular expression + +```sql +> SELECT MEAN(/water/) FROM "h2o_feet" + +name: h2o_feet +time mean_water_level +---- ---------------- +1970-01-01T00:00:00Z 4.442107025822523 +``` + +The query returns the average field value for each field key that stores numerical values and includes the word `water` in the `h2o_feet` measurement. + +#### Calculate the mean field value associated with a field key and include several clauses + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-17T23:48:00Z' AND time <= '2015-08-18T00:54:00Z' GROUP BY time(12m),* fill(9.01) LIMIT 7 SLIMIT 1 + +name: h2o_feet +tags: location=coyote_creek +time mean +---- ---- +2015-08-17T23:48:00Z 9.01 +2015-08-18T00:00:00Z 8.0625 +2015-08-18T00:12:00Z 7.8245 +2015-08-18T00:24:00Z 7.5675 +2015-08-18T00:36:00Z 7.303 +2015-08-18T00:48:00Z 7.046 +``` + +The query returns the average of the values in the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-17T23:48:00Z` and `2015-08-18T00:54:00Z` and [groups](/enterprise_influxdb/v1.9/query_language/explore-data/#the-group-by-clause) results into 12-minute time intervals and per tag. +The query [fills](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals-and-fill) empty time intervals with `9.01` and [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points and series returned to seven and one. + +### MEDIAN() + +Returns the middle value from a sorted list of [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). + +#### Syntax + +``` +SELECT MEDIAN( [ * | | // ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`MEDIAN(field_key)` +Returns the middle field value associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`MEDIAN(/regular_expression/)` +Returns the middle field value associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +`MEDIAN(*)` +Returns the middle field value associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`MEDIAN()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +> **Note:** `MEDIAN()` is nearly equivalent to [`PERCENTILE(field_key, 50)`](#percentile), except `MEDIAN()` returns the average of the two middle field values if the field contains an even number of values. + +#### Examples + +##### Calculate the median field value associated with a field key + +```sql +> SELECT MEDIAN("water_level") FROM "h2o_feet" + +name: h2o_feet +time median +---- ------ +1970-01-01T00:00:00Z 4.124 +``` + +The query returns the middle field value in the `water_level` field key and in the `h2o_feet` measurement. + +##### Calculate the median field value associated with each field key in a measurement + +```sql +> SELECT MEDIAN(*) FROM "h2o_feet" + +name: h2o_feet +time median_water_level +---- ------------------ +1970-01-01T00:00:00Z 4.124 +``` + +The query returns the middle field value for every field key that stores numerical values in the `h2o_feet` measurement. +The `h2o_feet` measurement has one numerical field: `water_level`. + +##### Calculate the median field value associated with each field key that matches a regular expression + +```sql +> SELECT MEDIAN(/water/) FROM "h2o_feet" + +name: h2o_feet +time median_water_level +---- ------------------ +1970-01-01T00:00:00Z 4.124 +``` +The query returns the middle field value for every field key that stores numerical values and includes the word `water` in the `h2o_feet` measurement. + +#### Calculate the median field value associated with a field key and include several clauses + +```sql +> SELECT MEDIAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-17T23:48:00Z' AND time <= '2015-08-18T00:54:00Z' GROUP BY time(12m),* fill(700) LIMIT 7 SLIMIT 1 SOFFSET 1 + +name: h2o_feet +tags: location=santa_monica +time median +---- ------ +2015-08-17T23:48:00Z 700 +2015-08-18T00:00:00Z 2.09 +2015-08-18T00:12:00Z 2.077 +2015-08-18T00:24:00Z 2.0460000000000003 +2015-08-18T00:36:00Z 2.0620000000000003 +2015-08-18T00:48:00Z 700 +``` + +The query returns the middle field value in the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-17T23:48:00Z` and `2015-08-18T00:54:00Z` and [groups](/enterprise_influxdb/v1.9/query_language/explore-data/#the-group-by-clause) results into 12-minute time intervals and per tag. +The query [fills](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals-and-fill) empty time intervals with `700 `, [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points and series returned to seven and one, and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) the series returned by one. + +### MODE() + +Returns the most frequent value in a list of [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). + +#### Syntax + +``` +SELECT MODE( [ * | | // ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`MODE(field_key)` +Returns the most frequent field value associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`MODE(/regular_expression/)` +Returns the most frequent field value associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +`MODE(*)` +Returns the most frequent field value associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`MODE()` supports all field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +> **Note:** `MODE()` returns the field value with the earliest [timestamp](/enterprise_influxdb/v1.9/concepts/glossary/#timestamp) if there's a tie between two or more values for the maximum number of occurrences. + +#### Examples + +##### Calculate the mode field value associated with a field key + +```sql +> SELECT MODE("level description") FROM "h2o_feet" + +name: h2o_feet +time mode +---- ---- +1970-01-01T00:00:00Z between 3 and 6 feet +``` + +The query returns the most frequent field value in the `level description` field key and in the `h2o_feet` measurement. + +##### Calculate the mode field value associated with each field key in a measurement + +```sql +> SELECT MODE(*) FROM "h2o_feet" + +name: h2o_feet +time mode_level description mode_water_level +---- ---------------------- ---------------- +1970-01-01T00:00:00Z between 3 and 6 feet 2.69 +``` + +The query returns the most frequent field value for every field key in the `h2o_feet` measurement. +The `h2o_feet` measurement has two field keys: `level description` and `water_level`. + +##### Calculate the mode field value associated with each field key that matches a regular expression + +```sql +> SELECT MODE(/water/) FROM "h2o_feet" + +name: h2o_feet +time mode_water_level +---- ---------------- +1970-01-01T00:00:00Z 2.69 +``` + +The query returns the most frequent field value for every field key that includes the word `/water/` in the `h2o_feet` measurement. + +#### Calculate the mode field value associated with a field key and include several clauses + +```sql +> SELECT MODE("level description") FROM "h2o_feet" WHERE time >= '2015-08-17T23:48:00Z' AND time <= '2015-08-18T00:54:00Z' GROUP BY time(12m),* LIMIT 3 SLIMIT 1 SOFFSET 1 + +name: h2o_feet +tags: location=santa_monica +time mode +---- ---- +2015-08-17T23:48:00Z +2015-08-18T00:00:00Z below 3 feet +2015-08-18T00:12:00Z below 3 feet +``` + +The query returns the mode of the values associated with the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-17T23:48:00Z` and `2015-08-18T00:54:00Z` and [groups](/enterprise_influxdb/v1.9/query_language/explore-data/#the-group-by-clause) results into 12-minute time intervals and per tag. +The query [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points and series returned to three and one, and it [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) the series returned by one. + +### SPREAD() + +Returns the difference between the minimum and maximum [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). + +#### Syntax + +``` +SELECT SPREAD( [ * | | // ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`SPREAD(field_key)` +Returns the difference between the minimum and maximum field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`SPREAD(/regular_expression/)` +Returns the difference between the minimum and maximum field values associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +`SPREAD(*)` +Returns the difference between the minimum and maximum field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`SPREAD()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +#### Examples + +##### Calculate the spread for the field values associated with a field key + +```sql +> SELECT SPREAD("water_level") FROM "h2o_feet" + +name: h2o_feet +time spread +---- ------ +1970-01-01T00:00:00Z 10.574 +``` + +The query returns the difference between the minimum and maximum field values in the `water_level` field key and in the `h2o_feet` measurement. + +##### Calculate the spread for the field values associated with each field key in a measurement + +```sql +> SELECT SPREAD(*) FROM "h2o_feet" + +name: h2o_feet +time spread_water_level +---- ------------------ +1970-01-01T00:00:00Z 10.574 +``` + +The query returns the difference between the minimum and maximum field values for every field key that stores numerical values in the `h2o_feet` measurement. +The `h2o_feet` measurement has one numerical field: `water_level`. + +##### Calculate the spread for the field values associated with each field key that matches a regular expression + +```sql +> SELECT SPREAD(/water/) FROM "h2o_feet" + +name: h2o_feet +time spread_water_level +---- ------------------ +1970-01-01T00:00:00Z 10.574 +``` + +The query returns the difference between the minimum and maximum field values for every field key that stores numerical values and includes the word `water` in the `h2o_feet` measurement. + +#### Calculate the spread for the field values associated with a field key and include several clauses + +```sql +> SELECT SPREAD("water_level") FROM "h2o_feet" WHERE time >= '2015-08-17T23:48:00Z' AND time <= '2015-08-18T00:54:00Z' GROUP BY time(12m),* fill(18) LIMIT 3 SLIMIT 1 SOFFSET 1 + +name: h2o_feet +tags: location=santa_monica +time spread +---- ------ +2015-08-17T23:48:00Z 18 +2015-08-18T00:00:00Z 0.052000000000000046 +2015-08-18T00:12:00Z 0.09799999999999986 +``` + +The query returns the difference between the minimum and maximum field values in the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-17T23:48:00Z` and `2015-08-18T00:54:00Z `and [groups](/enterprise_influxdb/v1.9/query_language/explore-data/#the-group-by-clause) results into 12-minute time intervals and per tag. +The query [fills](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals-and-fill) empty time intervals with `18`, [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points and series returned to three and one, and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) the series returned by one. + +### STDDEV() + +Returns the standard deviation of [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). + +#### Syntax + +``` +SELECT STDDEV( [ * | | // ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`STDDEV(field_key)` +Returns the standard deviation of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`STDDEV(/regular_expression/)` +Returns the standard deviation of field values associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +`STDDEV(*)` +Returns the standard deviation of field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`STDDEV()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +#### Examples + +##### Calculate the standard deviation for the field values associated with a field key + +```sql +> SELECT STDDEV("water_level") FROM "h2o_feet" + +name: h2o_feet +time stddev +---- ------ +1970-01-01T00:00:00Z 2.279144584196141 +``` + +The query returns the standard deviation of the field values in the `water_level` field key and in the `h2o_feet` measurement. + +##### Calculate the standard deviation for the field values associated with each field key in a measurement + +```sql +> SELECT STDDEV(*) FROM "h2o_feet" + +name: h2o_feet +time stddev_water_level +---- ------------------ +1970-01-01T00:00:00Z 2.279144584196141 +``` + +The query returns the standard deviation of the field values for each field key that stores numerical values in the `h2o_feet` measurement. +The `h2o_feet` measurement has one numerical field: `water_level`. + +##### Calculate the standard deviation for the field values associated with each field key that matches a regular expression + +```sql +> SELECT STDDEV(/water/) FROM "h2o_feet" + +name: h2o_feet +time stddev_water_level +---- ------------------ +1970-01-01T00:00:00Z 2.279144584196141 +``` + +The query returns the standard deviation of the field values for each field key that stores numerical values and includes the word `water` in the `h2o_feet` measurement. + +#### Calculate the standard deviation for the field values associated with a field key and include several clauses + +```sql +> SELECT STDDEV("water_level") FROM "h2o_feet" WHERE time >= '2015-08-17T23:48:00Z' AND time <= '2015-08-18T00:54:00Z' GROUP BY time(12m),* fill(18000) LIMIT 2 SLIMIT 1 SOFFSET 1 + +name: h2o_feet +tags: location=santa_monica +time stddev +---- ------ +2015-08-17T23:48:00Z 18000 +2015-08-18T00:00:00Z 0.03676955262170051 +``` + +The query returns the standard deviation of the field values in the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-17T23:48:00Z` and `2015-08-18T00:54:00Z` and [groups](/enterprise_influxdb/v1.9/query_language/explore-data/#the-group-by-clause) results into 12-minute time intervals and per tag. +The query [fills](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals-and-fill) empty time intervals with `18000`, [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points and series returned to two and one, and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) the series returned by one. + +### SUM() + +Returns the sum of [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). + +#### Syntax + +``` +SELECT SUM( [ * | | // ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`SUM(field_key)` +Returns the sum of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`SUM(/regular_expression/)` +Returns the sum of field values associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +`SUM(*)` +Returns the sums of field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`SUM()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +#### Examples + +#### Calculate the sum of the field values associated with a field key + +```sql +> SELECT SUM("water_level") FROM "h2o_feet" + +name: h2o_feet +time sum +---- --- +1970-01-01T00:00:00Z 67777.66900000004 +``` + +The query returns the summed total of the field values in the `water_level` field key and in the `h2o_feet` measurement. + +#### Calculate the sum of the field values associated with each field key in a measurement + +```sql +> SELECT SUM(*) FROM "h2o_feet" + +name: h2o_feet +time sum_water_level +---- --------------- +1970-01-01T00:00:00Z 67777.66900000004 +``` + +The query returns the summed total of the field values for each field key that stores numerical values in the `h2o_feet` measurement. +The `h2o_feet` measurement has one numerical field: `water_level`. + +#### Calculate the sum of the field values associated with each field key that matches a regular expression + +```sql +> SELECT SUM(/water/) FROM "h2o_feet" + +name: h2o_feet +time sum_water_level +---- --------------- +1970-01-01T00:00:00Z 67777.66900000004 +``` + +The query returns the summed total of the field values for each field key that stores numerical values and includes the word `water` in the `h2o_feet` measurement. + +#### Calculate the sum of the field values associated with a field key and include several clauses + +```sql +> SELECT SUM("water_level") FROM "h2o_feet" WHERE time >= '2015-08-17T23:48:00Z' AND time <= '2015-08-18T00:54:00Z' GROUP BY time(12m),* fill(18000) LIMIT 4 SLIMIT 1 + +name: h2o_feet +tags: location=coyote_creek +time sum +---- --- +2015-08-17T23:48:00Z 18000 +2015-08-18T00:00:00Z 16.125 +2015-08-18T00:12:00Z 15.649 +2015-08-18T00:24:00Z 15.135 +``` + +The query returns the summed total of the field values in the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-17T23:48:00Z` and `2015-08-18T00:54:00Z` and [groups](/enterprise_influxdb/v1.9/query_language/explore-data/#the-group-by-clause) results into 12-minute time intervals and per tag. The query [fills](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals-and-fill) empty time intervals with 18000, and it [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points and series returned to four and one. + +## Selectors + +### BOTTOM() + +Returns the smallest `N` [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). + +#### Syntax + +``` +SELECT BOTTOM([,], )[,|] [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`BOTTOM(field_key,N)` +Returns the smallest N field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`BOTTOM(field_key,tag_key(s),N)` +Returns the smallest field value for N tag values of the [tag key](/enterprise_influxdb/v1.9/concepts/glossary/#tag-key). + +`BOTTOM(field_key,N),tag_key(s),field_key(s)` +Returns the smallest N field values associated with the field key in the parentheses and the relevant [tag](/enterprise_influxdb/v1.9/concepts/glossary/#tag) and/or [field](/enterprise_influxdb/v1.9/concepts/glossary/#field). + +`BOTTOM()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +> **Notes:** +> +* `BOTTOM()` returns the field value with the earliest timestamp if there's a tie between two or more values for the smallest value. +* `BOTTOM()` differs from other InfluxQL functions when combined with an [`INTO` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#the-into-clause). See the [Common Issues](#common-issues-with-bottom) section for more information. + +#### Examples + +##### Select the bottom three field values associated with a field key + +```sql +> SELECT BOTTOM("water_level",3) FROM "h2o_feet" + +name: h2o_feet +time bottom +---- ------ +2015-08-29T14:30:00Z -0.61 +2015-08-29T14:36:00Z -0.591 +2015-08-30T15:18:00Z -0.594 +``` + +The query returns the smallest three field values in the `water_level` field key and in the `h2o_feet` [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +##### Select the bottom field value associated with a field key for two tags + +```sql +> SELECT BOTTOM("water_level","location",2) FROM "h2o_feet" + +name: h2o_feet +time bottom location +---- ------ -------- +2015-08-29T10:36:00Z -0.243 santa_monica +2015-08-29T14:30:00Z -0.61 coyote_creek +``` + +The query returns the smallest field values in the `water_level` field key for two tag values associated with the `location` tag key. + +##### Select the bottom four field values associated with a field key and the relevant tags and fields + +```sql +> SELECT BOTTOM("water_level",4),"location","level description" FROM "h2o_feet" + +name: h2o_feet +time bottom location level description +---- ------ -------- ----------------- +2015-08-29T14:24:00Z -0.587 coyote_creek below 3 feet +2015-08-29T14:30:00Z -0.61 coyote_creek below 3 feet +2015-08-29T14:36:00Z -0.591 coyote_creek below 3 feet +2015-08-30T15:18:00Z -0.594 coyote_creek below 3 feet +``` + +The query returns the smallest four field values in the `water_level` field key and the relevant values of the `location` tag key and the `level description` field key. + +##### Select the bottom three field values associated with a field key and include several clauses + +```sql +> SELECT BOTTOM("water_level",3),"location" FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:54:00Z' GROUP BY time(24m) ORDER BY time DESC + +name: h2o_feet +time bottom location +---- ------ -------- +2015-08-18T00:48:00Z 1.991 santa_monica +2015-08-18T00:54:00Z 2.054 santa_monica +2015-08-18T00:54:00Z 6.982 coyote_creek +2015-08-18T00:24:00Z 2.041 santa_monica +2015-08-18T00:30:00Z 2.051 santa_monica +2015-08-18T00:42:00Z 2.057 santa_monica +2015-08-18T00:00:00Z 2.064 santa_monica +2015-08-18T00:06:00Z 2.116 santa_monica +2015-08-18T00:12:00Z 2.028 santa_monica +``` + +The query returns the smallest three values in the `water_level` field key for each 24-minute [interval](/enterprise_influxdb/v1.9/query_language/explore-data/#basic-group-by-time-syntax) between `2015-08-18T00:00:00Z` and `2015-08-18T00:54:00Z`. +It also returns results in [descending timestamp](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc) order. + +Notice that the [GROUP BY time() clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) does not override the points’ original timestamps. See [Issue 1](#bottom-with-a-group-by-time-clause) in the section below for a more detailed explanation of that behavior. + +#### Common Issues with `BOTTOM()` + +##### `BOTTOM()` with a `GROUP BY time()` clause + +Queries with `BOTTOM()` and a `GROUP BY time()` clause return the specified +number of points per `GROUP BY time()` interval. +For +[most `GROUP BY time()` queries](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals), +the returned timestamps mark the start of the `GROUP BY time()` interval. +`GROUP BY time()` queries with the `BOTTOM()` function behave differently; +they maintain the timestamp of the original data point. + +###### Example + +The query below returns two points per 18-minute +`GROUP BY time()` interval. +Notice that the returned timestamps are the points' original timestamps; they +are not forced to match the start of the `GROUP BY time()` intervals. + +```sql +> SELECT BOTTOM("water_level",2) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(18m) + +name: h2o_feet +time bottom +---- ------ + __ +2015-08-18T00:00:00Z 2.064 | +2015-08-18T00:12:00Z 2.028 | <------- Smallest points for the first time interval + -- + __ +2015-08-18T00:24:00Z 2.041 | +2015-08-18T00:30:00Z 2.051 | <------- Smallest points for the second time interval -- +``` + +##### BOTTOM() and a tag key with fewer than N tag values + +Queries with the syntax `SELECT BOTTOM(,,)` can return fewer points than expected. +If the tag key has `X` tag values, the query specifies `N` values, and `X` is smaller than `N`, then the query returns `X` points. + +###### Example + +The query below asks for the smallest field values of `water_level` for three tag values of the `location` tag key. +Because the `location` tag key has two tag values (`santa_monica` and `coyote_creek`), the query returns two points instead of three. + +```sql +> SELECT BOTTOM("water_level","location",3) FROM "h2o_feet" + +name: h2o_feet +time bottom location +---- ------ -------- +2015-08-29T10:36:00Z -0.243 santa_monica +2015-08-29T14:30:00Z -0.61 coyote_creek +``` + +##### BOTTOM(), tags, and the INTO clause + +When combined with an [`INTO` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#the-into-clause) and no [`GROUP BY tag` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags), most InfluxQL functions [convert](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#why-are-my-into-queries-missing-data) any tags in the initial data to fields in the newly written data. +This behavior also applies to the `BOTTOM()` function unless `BOTTOM()` includes a tag key as an argument: `BOTTOM(field_key,tag_key(s),N)`. +In those cases, the system preserves the specified tag as a tag in the newly written data. + +###### Example + +The first query in the codeblock below returns the smallest field values in the `water_level` field key for two tag values associated with the `location` tag key. +It also writes those results to the `bottom_water_levels` measurement. + +The second query [shows](/enterprise_influxdb/v1.9/query_language/explore-schema/#show-tag-keys) that InfluxDB preserved the `location` tag as a tag in the `bottom_water_levels` measurement. + +```sql +> SELECT BOTTOM("water_level","location",2) INTO "bottom_water_levels" FROM "h2o_feet" + +name: result +time written +---- ------- +1970-01-01T00:00:00Z 2 + +> SHOW TAG KEYS FROM "bottom_water_levels" + +name: bottom_water_levels +tagKey +------ +location +``` + +### FIRST() + +Returns the [field value ](/enterprise_influxdb/v1.9/concepts/glossary/#field-value) with the oldest timestamp. + +#### Syntax + +``` +SELECT FIRST()[,|] [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`FIRST(field_key)` +Returns the oldest field value (determined by timestamp) associated with the field key. + +`FIRST(/regular_expression/)` +Returns the oldest field value (determined by timestamp) associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +`FIRST(*)` +Returns the oldest field value (determined by timestamp) associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`FIRST(field_key),tag_key(s),field_key(s)` +Returns the oldest field value (determined by timestamp) associated with the field key in the parentheses and the relevant [tag](/enterprise_influxdb/v1.9/concepts/glossary/#tag) and/or [field](/enterprise_influxdb/v1.9/concepts/glossary/#field). + +`FIRST()` supports all field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +#### Examples + +##### Select the first field value associated with a field key + +```sql +> SELECT FIRST("level description") FROM "h2o_feet" + +name: h2o_feet +time first +---- ----- +2015-08-18T00:00:00Z between 6 and 9 feet +``` + +The query returns the oldest field value (determined by timestamp) associated with the `level description` field key and in the `h2o_feet` measurement. + +##### Select the first field value associated with each field key in a measurement + +```sql +> SELECT FIRST(*) FROM "h2o_feet" + +name: h2o_feet +time first_level description first_water_level +---- ----------------------- ----------------- +1970-01-01T00:00:00Z between 6 and 9 feet 8.12 +``` + +The query returns the oldest field value (determined by timestamp) for each field key in the `h2o_feet` measurement. +The `h2o_feet` measurement has two field keys: `level description` and `water_level`. + +##### Select the first field value associated with each field key that matches a regular expression + +```sql +> SELECT FIRST(/level/) FROM "h2o_feet" + +name: h2o_feet +time first_level description first_water_level +---- ----------------------- ----------------- +1970-01-01T00:00:00Z between 6 and 9 feet 8.12 +``` + +The query returns the oldest field value for each field key that includes the word `level` in the `h2o_feet` measurement. + +##### Select the first value associated with a field key and the relevant tags and fields + +```sql +> SELECT FIRST("level description"),"location","water_level" FROM "h2o_feet" + +name: h2o_feet +time first location water_level +---- ----- -------- ----------- +2015-08-18T00:00:00Z between 6 and 9 feet coyote_creek 8.12 +``` + +The query returns the oldest field value (determined by timestamp) in the `level description` field key and the relevant values of the `location` tag key and the `water_level` field key. + +##### Select the first field value associated with a field key and include several clauses + +```sql +> SELECT FIRST("water_level") FROM "h2o_feet" WHERE time >= '2015-08-17T23:48:00Z' AND time <= '2015-08-18T00:54:00Z' GROUP BY time(12m),* fill(9.01) LIMIT 4 SLIMIT 1 + +name: h2o_feet +tags: location=coyote_creek +time first +---- ----- +2015-08-17T23:48:00Z 9.01 +2015-08-18T00:00:00Z 8.12 +2015-08-18T00:12:00Z 7.887 +2015-08-18T00:24:00Z 7.635 +``` + +The query returns the oldest field value (determined by timestamp) in the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-17T23:48:00Z` and `2015-08-18T00:54:00Z` and [groups](/enterprise_influxdb/v1.9/query_language/explore-data/#the-group-by-clause) results into 12-minute time intervals and per tag. +The query [fills](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals-and-fill) empty time intervals with `9.01`, and it [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points and series returned to four and one. + +Notice that the [`GROUP BY time()` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) overrides the points' original timestamps. +The timestamps in the results indicate the the start of each 12-minute time interval; +the first point in the results covers the time interval between `2015-08-17T23:48:00Z` and just before `2015-08-18T00:00:00Z` and the last point in the results covers the time interval between `2015-08-18T00:24:00Z` and just before `2015-08-18T00:36:00Z`. + +### LAST() + +Returns the [field value](/enterprise_influxdb/v1.9/concepts/glossary/#field-value) with the most recent timestamp. + +#### Syntax + +```sql +SELECT LAST()[,|] [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`LAST(field_key)` +Returns the newest field value (determined by timestamp) associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`LAST(/regular_expression/)` +Returns the newest field value (determined by timestamp) associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +`LAST(*)` +Returns the newest field value (determined by timestamp) associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`LAST(field_key),tag_key(s),field_key(s)` +Returns the newest field value (determined by timestamp) associated with the field key in the parentheses and the relevant [tag](/enterprise_influxdb/v1.9/concepts/glossary/#tag) and/or [field](/enterprise_influxdb/v1.9/concepts/glossary/#field). + +`LAST()` supports all field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +#### Examples + +##### Select the last field values associated with a field key + +```sql +> SELECT LAST("level description") FROM "h2o_feet" + +name: h2o_feet +time last +---- ---- +2015-09-18T21:42:00Z between 3 and 6 feet +``` + +The query returns the newest field value (determined by timestamp) associated with the `level description` field key and in the `h2o_feet` measurement. + +##### Select the last field values associated with each field key in a measurement + +```sql +> SELECT LAST(*) FROM "h2o_feet" + +name: h2o_feet +time last_level description last_water_level +---- ----------------------- ----------------- +1970-01-01T00:00:00Z between 3 and 6 feet 4.938 +``` + +The query returns the newest field value (determined by timestamp) for each field key in the `h2o_feet` measurement. +The `h2o_feet` measurement has two field keys: `level description` and `water_level`. + +##### Select the last field value associated with each field key that matches a regular expression + +```sql +> SELECT LAST(/level/) FROM "h2o_feet" + +name: h2o_feet +time last_level description last_water_level +---- ----------------------- ----------------- +1970-01-01T00:00:00Z between 3 and 6 feet 4.938 +``` + +The query returns the newest field value for each field key that includes the word `level` in the `h2o_feet` measurement. + +##### Select the last field value associated with a field key and the relevant tags and fields + +```sql +> SELECT LAST("level description"),"location","water_level" FROM "h2o_feet" + +name: h2o_feet +time last location water_level +---- ---- -------- ----------- +2015-09-18T21:42:00Z between 3 and 6 feet santa_monica 4.938 +``` + +The query returns the newest field value (determined by timestamp) in the `level description` field key and the relevant values of the `location` tag key and the `water_level` field key. + +##### Select the last field value associated with a field key and include several clauses + +```sql +> SELECT LAST("water_level") FROM "h2o_feet" WHERE time >= '2015-08-17T23:48:00Z' AND time <= '2015-08-18T00:54:00Z' GROUP BY time(12m),* fill(9.01) LIMIT 4 SLIMIT 1 + +name: h2o_feet +tags: location=coyote_creek +time last +---- ---- +2015-08-17T23:48:00Z 9.01 +2015-08-18T00:00:00Z 8.005 +2015-08-18T00:12:00Z 7.762 +2015-08-18T00:24:00Z 7.5 +``` + +The query returns the newest field value (determined by timestamp) in the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-17T23:48:00Z` and `2015-08-18T00:54:00Z` and [groups](/enterprise_influxdb/v1.9/query_language/explore-data/#the-group-by-clause) results into 12-minute time intervals and per tag. +The query [fills](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals-and-fill) empty time intervals with `9.01`, and it [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points and series returned to four and one. + +Notice that the [`GROUP BY time()` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) overrides the points' original timestamps. +The timestamps in the results indicate the the start of each 12-minute time interval; +the first point in the results covers the time interval between `2015-08-17T23:48:00Z` and just before `2015-08-18T00:00:00Z` and the last point in the results covers the time interval between `2015-08-18T00:24:00Z` and just before `2015-08-18T00:36:00Z`. + +### MAX() + +Returns the greatest [field value](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). + +#### Syntax + +``` +SELECT MAX()[,|] [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`MAX(field_key)` +Returns the greatest field value associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`MAX(/regular_expression/)` +Returns the greatest field value associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +`MAX(*)` +Returns the greatest field value associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`MAX(field_key),tag_key(s),field_key(s)` +Returns the greatest field value associated with the field key in the parentheses and the relevant [tag](/enterprise_influxdb/v1.9/concepts/glossary/#tag) and/or [field](/enterprise_influxdb/v1.9/concepts/glossary/#field). + +`MAX()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +#### Examples + +##### Select the maximum field value associated with a field key + +```sql +> SELECT MAX("water_level") FROM "h2o_feet" + +name: h2o_feet +time max +---- --- +2015-08-29T07:24:00Z 9.964 +``` + +The query returns the greatest field value in the `water_level` field key and in the `h2o_feet` measurement. + +##### Select the maximum field value associated with each field key in a measurement + +```sql +> SELECT MAX(*) FROM "h2o_feet" + +name: h2o_feet +time max_water_level +---- --------------- +2015-08-29T07:24:00Z 9.964 +``` + +The query returns the greatest field value for each field key that stores numerical values in the `h2o_feet` measurement. +The `h2o_feet` measurement has one numerical field: `water_level`. + +##### Select the maximum field value associated with each field key that matches a regular expression + +```sql +> SELECT MAX(/level/) FROM "h2o_feet" + +name: h2o_feet +time max_water_level +---- --------------- +2015-08-29T07:24:00Z 9.964 +``` + +The query returns the greatest field value for each field key that stores numerical values and includes the word `water` in the `h2o_feet` measurement. + +##### Select the maximum field value associated with a field key and the relevant tags and fields + +```sql +> SELECT MAX("water_level"),"location","level description" FROM "h2o_feet" + +name: h2o_feet +time max location level description +---- --- -------- ----------------- +2015-08-29T07:24:00Z 9.964 coyote_creek at or greater than 9 feet +``` + +The query returns the greatest field value in the `water_level` field key and the relevant values of the `location` tag key and the `level description` field key. + +##### Select the maximum field value associated with a field key and include several clauses + +```sql +> SELECT MAX("water_level") FROM "h2o_feet" WHERE time >= '2015-08-17T23:48:00Z' AND time <= '2015-08-18T00:54:00Z' GROUP BY time(12m),* fill(9.01) LIMIT 4 SLIMIT 1 + +name: h2o_feet +tags: location=coyote_creek +time max +---- --- +2015-08-17T23:48:00Z 9.01 +2015-08-18T00:00:00Z 8.12 +2015-08-18T00:12:00Z 7.887 +2015-08-18T00:24:00Z 7.635 +``` + +The query returns the greatest field value in the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-17T23:48:00Z` and `2015-08-18T00:54:00Z` and [groups](/enterprise_influxdb/v1.9/query_language/explore-data/#the-group-by-clause) results in to 12-minute time intervals and per tag. +The query [fills](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals-and-fill) empty time intervals with `9.01`, and it [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points and series returned to four and one. + +Notice that the [`GROUP BY time()` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) overrides the points’ original timestamps. +The timestamps in the results indicate the the start of each 12-minute time interval; +the first point in the results covers the time interval between `2015-08-17T23:48:00Z` and just before `2015-08-18T00:00:00Z` and the last point in the results covers the time interval between `2015-08-18T00:24:00Z` and just before `2015-08-18T00:36:00Z`. + +### MIN() + +Returns the lowest [field value](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). + +#### Syntax + +``` +SELECT MIN()[,|] [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`MIN(field_key)` +Returns the lowest field value associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`MIN(/regular_expression/)` +Returns the lowest field value associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +`MIN(*)` +Returns the lowest field value associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`MIN(field_key),tag_key(s),field_key(s)` +Returns the lowest field value associated with the field key in the parentheses and the relevant [tag](/enterprise_influxdb/v1.9/concepts/glossary/#tag) and/or [field](/enterprise_influxdb/v1.9/concepts/glossary/#field). + +`MIN()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +#### Examples + +##### Select the minimum field value associated with a field key + +```sql +> SELECT MIN("water_level") FROM "h2o_feet" + +name: h2o_feet +time min +---- --- +2015-08-29T14:30:00Z -0.61 +``` + +The query returns the lowest field value in the `water_level` field key and in the `h2o_feet` measurement. + +##### Select the minimum field value associated with each field key in a measurement + +```sql +> SELECT MIN(*) FROM "h2o_feet" + +name: h2o_feet +time min_water_level +---- --------------- +2015-08-29T14:30:00Z -0.61 +``` + +The query returns the lowest field value for each field key that stores numerical values in the `h2o_feet` measurement. +The `h2o_feet` measurement has one numerical field: `water_level`. + +##### Select the minimum field value associated with each field key that matches a regular expression + +```sql +> SELECT MIN(/level/) FROM "h2o_feet" + +name: h2o_feet +time min_water_level +---- --------------- +2015-08-29T14:30:00Z -0.61 +``` + +The query returns the lowest field value for each field key that stores numerical values and includes the word `water` in the `h2o_feet` measurement. + +##### Select the minimum field value associated with a field key and the relevant tags and fields + +```sql +> SELECT MIN("water_level"),"location","level description" FROM "h2o_feet" + +name: h2o_feet +time min location level description +---- --- -------- ----------------- +2015-08-29T14:30:00Z -0.61 coyote_creek below 3 feet +``` + +The query returns the lowest field value in the `water_level` field key and the relevant values of the `location` tag key and the `level description` field key. + +##### Select the minimum field value associated with a field key and include several clauses + +```sql +> SELECT MIN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-17T23:48:00Z' AND time <= '2015-08-18T00:54:00Z' GROUP BY time(12m),* fill(9.01) LIMIT 4 SLIMIT 1 + +name: h2o_feet +tags: location=coyote_creek +time min +---- --- +2015-08-17T23:48:00Z 9.01 +2015-08-18T00:00:00Z 8.005 +2015-08-18T00:12:00Z 7.762 +2015-08-18T00:24:00Z 7.5 +``` + +The query returns the lowest field value in the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-17T23:48:00Z` and `2015-08-18T00:54:00Z` and [groups](/enterprise_influxdb/v1.9/query_language/explore-data/#the-group-by-clause) results in to 12-minute time intervals and per tag. +The query [fills](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals-and-fill) empty time intervals with `9.01`, and it [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points and series returned to four and one. + +Notice that the [`GROUP BY time()` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) overrides the points’ original timestamps. +The timestamps in the results indicate the the start of each 12-minute time interval; +the first point in the results covers the time interval between `2015-08-17T23:48:00Z` and just before `2015-08-18T00:00:00Z` and the last point in the results covers the time interval between `2015-08-18T00:24:00Z` and just before `2015-08-18T00:36:00Z`. + +### PERCENTILE() + +Returns the `N`th percentile [field value](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). + +#### Syntax + +``` +SELECT PERCENTILE(, )[,|] [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`PERCENTILE(field_key,N)` +Returns the Nth percentile field value associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`PERCENTILE(/regular_expression/,N)` +Returns the Nth percentile field value associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +`PERCENTILE(*,N)` +Returns the Nth percentile field value associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`PERCENTILE(field_key,N),tag_key(s),field_key(s)` +Returns the Nth percentile field value associated with the field key in the parentheses and the relevant [tag](/enterprise_influxdb/v1.9/concepts/glossary/#tag) and/or [field](/enterprise_influxdb/v1.9/concepts/glossary/#field). + +`N` must be an integer or floating point number between `0` and `100`, inclusive. +`PERCENTILE()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +#### Examples + +##### Select the fifth percentile field value associated with a field key + +```sql +> SELECT PERCENTILE("water_level",5) FROM "h2o_feet" + +name: h2o_feet +time percentile +---- ---------- +2015-08-31T03:42:00Z 1.122 +``` + +The query returns the field value that is larger than five percent of the field values in the `water_level` field key and in the `h2o_feet` measurement. + +##### Select the fifth percentile field value associated with each field key in a measurement + +```sql +> SELECT PERCENTILE(*,5) FROM "h2o_feet" + +name: h2o_feet +time percentile_water_level +---- ---------------------- +2015-08-31T03:42:00Z 1.122 +``` + +The query returns the field value that is larger than five percent of the field values in each field key that stores numerical values in the `h2o_feet` measurement. +The `h2o_feet` measurement has one numerical field: `water_level`. + +##### Select fifth percentile field value associated with each field key that matches a regular expression + +```sql +> SELECT PERCENTILE(/level/,5) FROM "h2o_feet" + +name: h2o_feet +time percentile_water_level +---- ---------------------- +2015-08-31T03:42:00Z 1.122 +``` + +The query returns the field value that is larger than five percent of the field values in each field key that stores numerical values and includes the word `water` in the `h2o_feet` measurement. + +##### Select the fifth percentile field values associated with a field key and the relevant tags and fields + +```sql +> SELECT PERCENTILE("water_level",5),"location","level description" FROM "h2o_feet" + +name: h2o_feet +time percentile location level description +---- ---------- -------- ----------------- +2015-08-31T03:42:00Z 1.122 coyote_creek below 3 feet +``` + +The query returns the field value that is larger than five percent of the field values in the `water_level` field key and the relevant values of the `location` tag key and the `level description` field key. + +##### Select the twentieth percentile field value associated with a field key and include several clauses + +```sql +> SELECT PERCENTILE("water_level",20) FROM "h2o_feet" WHERE time >= '2015-08-17T23:48:00Z' AND time <= '2015-08-18T00:54:00Z' GROUP BY time(24m) fill(15) LIMIT 2 + +name: h2o_feet +time percentile +---- ---------- +2015-08-17T23:36:00Z 15 +2015-08-18T00:00:00Z 2.064 +``` + +The query returns the field value that is larger than 20 percent of the values in the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-17T23:48:00Z` and `2015-08-18T00:54:00Z` and [groups](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) results into 24-minute intervals. +It [fills](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals-and-fill) empty time intervals with `15` and it [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to two. + +Notice that the [`GROUP BY time()` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) overrides the points’ original timestamps. +The timestamps in the results indicate the the start of each 24-minute time interval; the first point in the results covers the time interval between `2015-08-17T23:36:00Z` and just before `2015-08-18T00:00:00Z` and the last point in the results covers the time interval between `2015-08-18T00:00:00Z` and just before `2015-08-18T00:24:00Z`. + +#### Common Issues with PERCENTILE() + +##### PERCENTILE() compared to other InfluxQL functions + +* `PERCENTILE(,100)` is equivalent to [`MAX()`](#max). +* `PERCENTILE(, 50)` is nearly equivalent to [`MEDIAN()`](#median), except the `MEDIAN()` function returns the average of the two middle values if the field key contains an even number of field values. +* `PERCENTILE(,0)` is not equivalent to [`MIN()`](#min). This is a known [issue](https://github.com/influxdata/influxdb/issues/4418). + +### SAMPLE() + +Returns a random sample of `N` [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). +`SAMPLE()` uses [reservoir sampling](https://en.wikipedia.org/wiki/Reservoir_sampling) to generate the random points. + +#### Syntax + +``` +SELECT SAMPLE(, )[,|] [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`SAMPLE(field_key,N)` +Returns N randomly selected field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`SAMPLE(/regular_expression/,N)` +Returns N randomly selected field values associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +`SAMPLE(*,N)` +Returns N randomly selected field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`SAMPLE(field_key,N),tag_key(s),field_key(s)` +Returns N randomly selected field values associated with the field key in the parentheses and the relevant [tag](/enterprise_influxdb/v1.9/concepts/glossary/#tag) and/or [field](/enterprise_influxdb/v1.9/concepts/glossary/#field). + +`N` must be an integer. +`SAMPLE()` supports all field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +#### Examples + +##### Select a sample of the field values associated with a field key + +```sql +> SELECT SAMPLE("water_level",2) FROM "h2o_feet" + +name: h2o_feet +time sample +---- ------ +2015-09-09T21:48:00Z 5.659 +2015-09-18T10:00:00Z 6.939 +``` + +The query returns two randomly selected points from the `water_level` field key and in the `h2o_feet` measurement. + +#### Select a sample of the field values associated with each field key in a measurement + +```sql +> SELECT SAMPLE(*,2) FROM "h2o_feet" + +name: h2o_feet +time sample_level description sample_water_level +---- ------------------------ ------------------ +2015-08-25T17:06:00Z 3.284 +2015-09-03T04:30:00Z below 3 feet +2015-09-03T20:06:00Z between 3 and 6 feet +2015-09-08T21:54:00Z 3.412 +``` + +The query returns two randomly selected points for each field key in the `h2o_feet` measurement. +The `h2o_feet` measurement has two field keys: `level description` and `water_level`. + +##### Select a sample of the field values associated with each field key that matches a regular expression + +```sql +> SELECT SAMPLE(/level/,2) FROM "h2o_feet" + +name: h2o_feet +time sample_level description sample_water_level +---- ------------------------ ------------------ +2015-08-30T05:54:00Z between 6 and 9 feet +2015-09-07T01:18:00Z 7.854 +2015-09-09T20:30:00Z 7.32 +2015-09-13T19:18:00Z between 3 and 6 feet +``` + +The query returns two randomly selected points for each field key that includes the word `level` in the `h2o_feet` measurement. + +##### Select a sample of the field values associated with a field key and the relevant tags and fields + +```sql +> SELECT SAMPLE("water_level",2),"location","level description" FROM "h2o_feet" + +name: h2o_feet +time sample location level description +---- ------ -------- ----------------- +2015-08-29T10:54:00Z 5.689 coyote_creek between 3 and 6 feet +2015-09-08T15:48:00Z 6.391 coyote_creek between 6 and 9 feet +``` + +The query returns two randomly selected points from the `water_level` field key and the relevant values of the `location` tag and the `level description` field. + +##### Select a sample of the field values associated with a field key and include several clauses + +```sql +> SELECT SAMPLE("water_level",1) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(18m) + +name: h2o_feet +time sample +---- ------ +2015-08-18T00:12:00Z 2.028 +2015-08-18T00:30:00Z 2.051 +``` + +The query returns one randomly selected point from the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-18T00:00:00Z` and `2015-08-18T00:30:00Z` and [groups](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) results into 18-minute intervals. + +Notice that the [`GROUP BY time()` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) does not override the points' original timestamps. +See [Issue 1](#sample-with-a-group-by-time-clause) in the section below for a more detailed explanation of that behavior. + +#### Common Issues with `SAMPLE()` + +##### `SAMPLE()` with a `GROUP BY time()` clause + +Queries with `SAMPLE()` and a `GROUP BY time()` clause return the specified +number of points (`N`) per `GROUP BY time()` interval. +For +[most `GROUP BY time()` queries](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals), +the returned timestamps mark the start of the `GROUP BY time()` interval. +`GROUP BY time()` queries with the `SAMPLE()` function behave differently; +they maintain the timestamp of the original data point. + +###### Example + +The query below returns two randomly selected points per 18-minute +`GROUP BY time()` interval. +Notice that the returned timestamps are the points' original timestamps; they +are not forced to match the start of the `GROUP BY time()` intervals. + +```sql +> SELECT SAMPLE("water_level",2) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(18m) + +name: h2o_feet +time sample +---- ------ + __ +2015-08-18T00:06:00Z 2.116 | +2015-08-18T00:12:00Z 2.028 | <------- Randomly-selected points for the first time interval + -- + __ +2015-08-18T00:18:00Z 2.126 | +2015-08-18T00:30:00Z 2.051 | <------- Randomly-selected points for the second time interval + -- +``` + +### TOP() + +Returns the greatest `N` [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). + +#### Syntax + +``` +SELECT TOP( [,], )[,|] [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`TOP(field_key,N)` +Returns the greatest N field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`TOP(field_key,tag_key(s),N)` +Returns the greatest field value for N tag values of the [tag key](/enterprise_influxdb/v1.9/concepts/glossary/#tag-key). + +`TOP(field_key,N),tag_key(s),field_key(s)` +Returns the greatest N field values associated with the field key in the parentheses and the relevant [tag](/enterprise_influxdb/v1.9/concepts/glossary/#tag) and/or [field](/enterprise_influxdb/v1.9/concepts/glossary/#field). + +`TOP()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +> **Notes:** +> +* `TOP()` returns the field value with the earliest timestamp if there's a tie between two or more values for the greatest value. +* `TOP()` differs from other InfluxQL functions when combined with an [`INTO` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#the-into-clause). +See the [Common Issues](#common-issues-with-top) section for more information. + +#### Examples + +##### Select the top three field values associated with a field key + +```sql +> SELECT TOP("water_level",3) FROM "h2o_feet" + +name: h2o_feet +time top +---- --- +2015-08-29T07:18:00Z 9.957 +2015-08-29T07:24:00Z 9.964 +2015-08-29T07:30:00Z 9.954 +``` + +The query returns the greatest three field values in the `water_level` field key and in the `h2o_feet` [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +##### Select the top field value associated with a field key for two tags + +```sql +> SELECT TOP("water_level","location",2) FROM "h2o_feet" + +name: h2o_feet +time top location +---- --- -------- +2015-08-29T03:54:00Z 7.205 santa_monica +2015-08-29T07:24:00Z 9.964 coyote_creek +``` + +The query returns the greatest field values in the `water_level` field key for two tag values associated with the `location` tag key. + +##### Select the top four field values associated with a field key and the relevant tags and fields + +```sql +> SELECT TOP("water_level",4),"location","level description" FROM "h2o_feet" + +name: h2o_feet +time top location level description +---- --- -------- ----------------- +2015-08-29T07:18:00Z 9.957 coyote_creek at or greater than 9 feet +2015-08-29T07:24:00Z 9.964 coyote_creek at or greater than 9 feet +2015-08-29T07:30:00Z 9.954 coyote_creek at or greater than 9 feet +2015-08-29T07:36:00Z 9.941 coyote_creek at or greater than 9 feet +``` + +The query returns the greatest four field values in the `water_level` field key and the relevant values of the `location` tag key and the `level description` field key. + +##### Select the top three field values associated with a field key and include several clauses + +```sql +> SELECT TOP("water_level",3),"location" FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:54:00Z' GROUP BY time(24m) ORDER BY time DESC + +name: h2o_feet +time top location +---- --- -------- +2015-08-18T00:48:00Z 7.11 coyote_creek +2015-08-18T00:54:00Z 6.982 coyote_creek +2015-08-18T00:54:00Z 2.054 santa_monica +2015-08-18T00:24:00Z 7.635 coyote_creek +2015-08-18T00:30:00Z 7.5 coyote_creek +2015-08-18T00:36:00Z 7.372 coyote_creek +2015-08-18T00:00:00Z 8.12 coyote_creek +2015-08-18T00:06:00Z 8.005 coyote_creek +2015-08-18T00:12:00Z 7.887 coyote_creek +``` + +The query returns the greatest three values in the `water_level` field key for each 24-minute [interval](/enterprise_influxdb/v1.9/query_language/explore-data/#basic-group-by-time-syntax) between `2015-08-18T00:00:00Z` and `2015-08-18T00:54:00Z`. +It also returns results in [descending timestamp](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc) order. + +Notice that the [GROUP BY time() clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) does not override the points’ original timestamps. +See [Issue 1](#top-with-a-group-by-time-clause) in the section below for a more detailed explanation of that behavior. + +#### Common Issues with `TOP()` + +##### `TOP()` with a `GROUP BY time()` clause + +Queries with `TOP()` and a `GROUP BY time()` clause return the specified +number of points per `GROUP BY time()` interval. +For +[most `GROUP BY time()` queries](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals), +the returned timestamps mark the start of the `GROUP BY time()` interval. +`GROUP BY time()` queries with the `TOP()` function behave differently; +they maintain the timestamp of the original data point. + +###### Example + +The query below returns two points per 18-minute +`GROUP BY time()` interval. +Notice that the returned timestamps are the points' original timestamps; they +are not forced to match the start of the `GROUP BY time()` intervals. + +```sql +> SELECT TOP("water_level",2) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(18m) + +name: h2o_feet +time top +---- ------ + __ +2015-08-18T00:00:00Z 2.064 | +2015-08-18T00:06:00Z 2.116 | <------- Greatest points for the first time interval + -- + __ +2015-08-18T00:18:00Z 2.126 | +2015-08-18T00:30:00Z 2.051 | <------- Greatest points for the second time interval + -- +``` + +##### TOP() and a tag key with fewer than N tag values + +Queries with the syntax `SELECT TOP(,,)` can return fewer points than expected. +If the tag key has `X` tag values, the query specifies `N` values, and `X` is smaller than `N`, then the query returns `X` points. + +###### Example + +The query below asks for the greatest field values of `water_level` for three tag values of the `location` tag key. +Because the `location` tag key has two tag values (`santa_monica` and `coyote_creek`), the query returns two points instead of three. + +```sql +> SELECT TOP("water_level","location",3) FROM "h2o_feet" + +name: h2o_feet +time top location +---- --- -------- +2015-08-29T03:54:00Z 7.205 santa_monica +2015-08-29T07:24:00Z 9.964 coyote_creek +``` + +##### TOP(), tags, and the INTO clause + +When combined with an [`INTO` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#the-into-clause) and no [`GROUP BY tag` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags), most InfluxQL functions [convert](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#why-are-my-into-queries-missing-data) any tags in the initial data to fields in the newly written data. +This behavior also applies to the `TOP()` function unless `TOP()` includes a tag key as an argument: `TOP(field_key,tag_key(s),N)`. +In those cases, the system preserves the specified tag as a tag in the newly written data. + +###### Example + +The first query in the codeblock below returns the greatest field values in the `water_level` field key for two tag values associated with the `location` tag key. +It also writes those results to the `top_water_levels` measurement. + +The second query [shows](/enterprise_influxdb/v1.9/query_language/explore-schema/#show-tag-keys) that InfluxDB preserved the `location` tag as a tag in the `top_water_levels` measurement. + +```sql +> SELECT TOP("water_level","location",2) INTO "top_water_levels" FROM "h2o_feet" + +name: result +time written +---- ------- +1970-01-01T00:00:00Z 2 + +> SHOW TAG KEYS FROM "top_water_levels" + +name: top_water_levels +tagKey +------ +location +``` + +## Transformations + +### ABS() + +Returns the absolute value of the field value. + +#### Basic syntax + +``` +SELECT ABS( [ * | ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`ABS(field_key)` +Returns the absolute values of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + + + +`ABS(*)` +Returns the absolute values of field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`ABS()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `ABS()` with a `GROUP BY time()` clause. + +##### Examples + +The examples below use the following subsample of this [sample data](https://gist.github.com/sanderson/8f8aec94a60b2c31a61f44a37737bfea): + +```sql +> SELECT * FROM "data" WHERE time >= '2018-06-24T12:00:00Z' AND time <= '2018-06-24T12:05:00Z' + +name: data +time a b +---- - - +1529841600000000000 1.33909108671076 -0.163643058925645 +1529841660000000000 -0.774984088561186 0.137034364053949 +1529841720000000000 -0.921037167720451 -0.482943221384294 +1529841780000000000 -1.73880754843378 -0.0729732928756677 +1529841840000000000 -0.905980032168252 1.77857552719844 +1529841900000000000 -0.891164752631417 0.741147445214238 +``` + +###### Calculate the absolute values of field values associated with a field key + +```sql +> SELECT ABS("a") FROM "data" WHERE time >= '2018-06-24T12:00:00Z' AND time <= '2018-06-24T12:05:00Z' + +name: data +time abs +---- --- +1529841600000000000 1.33909108671076 +1529841660000000000 0.774984088561186 +1529841720000000000 0.921037167720451 +1529841780000000000 1.73880754843378 +1529841840000000000 0.905980032168252 +1529841900000000000 0.891164752631417 +``` + +The query returns the absolute values of field values in the `a` field key in the `data` measurement. + +###### Calculate the absolute Values of field values associated with each field key in a measurement + +```sql +> SELECT ABS(*) FROM "data" WHERE time >= '2018-06-24T12:00:00Z' AND time <= '2018-06-24T12:05:00Z' + +name: data +time abs_a abs_b +---- ----- ----- +1529841600000000000 1.33909108671076 0.163643058925645 +1529841660000000000 0.774984088561186 0.137034364053949 +1529841720000000000 0.921037167720451 0.482943221384294 +1529841780000000000 1.73880754843378 0.0729732928756677 +1529841840000000000 0.905980032168252 1.77857552719844 +1529841900000000000 0.891164752631417 0.741147445214238 +``` + +The query returns the absolute values of field values for each field key that stores +numerical values in the `data` measurement. +The `data` measurement has two numerical fields: `a` and `b`. + + + +###### Calculate the absolute values of field values associated with a field key and include several clauses + +```sql +> SELECT ABS("a") FROM "data" WHERE time >= '2018-06-24T12:00:00Z' AND time <= '2018-06-24T12:05:00Z' ORDER BY time DESC LIMIT 4 OFFSET 2 + +name: data +time abs +---- --- +1529841780000000000 1.73880754843378 +1529841720000000000 0.921037167720451 +1529841660000000000 0.774984088561186 +1529841600000000000 1.33909108671076 +``` + +The query returns the absolute values of field values associated with the `a` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2018-06-24T12:00:00Z` and `2018-06-24T12:05:00Z` and returns results in [descending timestamp order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to four and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by two points. + +#### Advanced syntax + +``` +SELECT ABS(( [ * | ] )) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `ABS()` function to those results. + +`ABS()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples + +###### Calculate the absolute values of mean values + +```sql +> SELECT ABS(MEAN("a")) FROM "data" WHERE time >= '2018-06-24T12:00:00Z' AND time <= '2018-06-24T13:00:00Z' GROUP BY time(12m) + +name: data +time abs +---- --- +1529841600000000000 0.3960977256302787 +1529842320000000000 0.0010541018316373302 +1529843040000000000 0.04494733240283668 +1529843760000000000 0.2553594777104415 +1529844480000000000 0.20382988543108413 +1529845200000000000 0.790836070736962 +``` + +The query returns the absolute values of [average](#mean) `a`s that are calculated at 12-minute intervals. + +To get those results, InfluxDB first calculates the average `a`s at 12-minute intervals. +This step is the same as using the `MEAN()` function with the `GROUP BY time()` clause and without `ABS()`: + +```sql +> SELECT MEAN("a") FROM "data" WHERE time >= '2018-06-24T12:00:00Z' AND time <= '2018-06-24T13:00:00Z' GROUP BY time(12m) + +name: data +time mean +---- ---- +1529841600000000000 -0.3960977256302787 +1529842320000000000 0.0010541018316373302 +1529843040000000000 0.04494733240283668 +1529843760000000000 0.2553594777104415 +1529844480000000000 0.20382988543108413 +1529845200000000000 -0.790836070736962 +``` + +InfluxDB then calculates absolute values of those averages. + +### ACOS() + +Returns the arccosine (in radians) of the field value. Field values must be between -1 and 1. + +#### Basic syntax + +``` +SELECT ACOS( [ * | ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`ACOS(field_key)` +Returns the arccosine of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + + + +`ACOS(*)` +Returns the arccosine of field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`ACOS()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types) with values between -1 and 1. + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `ACOS()` with a `GROUP BY time()` clause. + +##### Examples + +The examples below use the following data sample of simulated park occupancy relative to total capacity. The important thing to note is that all field values fall within the calculable range (-1 to 1) of the `ACOS()` function: + +```sql +> SELECT "of_capacity" FROM "park_occupancy" WHERE time >= '2017-05-01T00:00:00Z' AND time <= '2017-05-09T00:00:00Z' + +name: park_occupancy +time capacity +---- -------- +2017-05-01T00:00:00Z 0.83 +2017-05-02T00:00:00Z 0.3 +2017-05-03T00:00:00Z 0.84 +2017-05-04T00:00:00Z 0.22 +2017-05-05T00:00:00Z 0.17 +2017-05-06T00:00:00Z 0.77 +2017-05-07T00:00:00Z 0.64 +2017-05-08T00:00:00Z 0.72 +2017-05-09T00:00:00Z 0.16 +``` + +###### Calculate the arccosine of field values associated with a field key + +```sql +> SELECT ACOS("of_capacity") FROM "park_occupancy" WHERE time >= '2017-05-01T00:00:00Z' AND time <= '2017-05-09T00:00:00Z' + +name: park_occupancy +time acos +---- ---- +2017-05-01T00:00:00Z 0.591688642426544 +2017-05-02T00:00:00Z 1.266103672779499 +2017-05-03T00:00:00Z 0.5735131044230969 +2017-05-04T00:00:00Z 1.3489818562981022 +2017-05-05T00:00:00Z 1.399966657665792 +2017-05-06T00:00:00Z 0.6919551751263169 +2017-05-07T00:00:00Z 0.8762980611683406 +2017-05-08T00:00:00Z 0.7669940078618667 +2017-05-09T00:00:00Z 1.410105673842986 +``` + +The query returns arccosine of field values in the `of_capacity` field key in the `park_occupancy` measurement. + +###### Calculate the arccosine of field values associated with each field key in a measurement + +```sql +> SELECT ACOS(*) FROM "park_occupancy" WHERE time >= '2017-05-01T00:00:00Z' AND time <= '2017-05-09T00:00:00Z' + +name: park_occupancy +time acos_of_capacity +---- ------------- +2017-05-01T00:00:00Z 0.591688642426544 +2017-05-02T00:00:00Z 1.266103672779499 +2017-05-03T00:00:00Z 0.5735131044230969 +2017-05-04T00:00:00Z 1.3489818562981022 +2017-05-05T00:00:00Z 1.399966657665792 +2017-05-06T00:00:00Z 0.6919551751263169 +2017-05-07T00:00:00Z 0.8762980611683406 +2017-05-08T00:00:00Z 0.7669940078618667 +2017-05-09T00:00:00Z 1.410105673842986 +``` + +The query returns arccosine of field values for each field key that stores numerical values in the `park_occupancy` measurement. +The `park_occupancy` measurement has one numerical field: `of_capacity`. + + + +###### Calculate the arccosine of field values associated with a field key and include several clauses + +```sql +> SELECT ACOS("of_capacity") FROM "park_occupancy" WHERE time >= '2017-05-01T00:00:00Z' AND time <= '2017-05-09T00:00:00Z' ORDER BY time DESC LIMIT 4 OFFSET 2 + +name: park_occupancy +time acos +---- ---- +2017-05-07T00:00:00Z 0.8762980611683406 +2017-05-06T00:00:00Z 0.6919551751263169 +2017-05-05T00:00:00Z 1.399966657665792 +2017-05-04T00:00:00Z 1.3489818562981022 +``` + +The query returns arccosine of field values associated with the `of_capacity` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2017-05-01T00:00:00Z` and `2017-05-09T00:00:00Z` and returns results in [descending timestamp order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to four and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by two points. + +#### Advanced syntax + +``` +SELECT ACOS(( [ * | ] )) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `ACOS()` function to those results. + +`ACOS()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples + +###### Calculate the arccosine of mean values + +```sql +> SELECT ACOS(MEAN("of_capacity")) FROM "park_occupancy" WHERE time >= '2017-05-01T00:00:00Z' AND time <= '2017-05-09T00:00:00Z' GROUP BY time(3d) + +name: park_occupancy +time acos +---- ---- +2017-04-30T00:00:00Z 0.9703630732143733 +2017-05-03T00:00:00Z 1.1483422646081407 +2017-05-06T00:00:00Z 0.7812981174487247 +2017-05-09T00:00:00Z 1.410105673842986 +``` + +The query returns arccosine of [average](#mean) `of_capacity`s that are calculated at 3-day intervals. + +To get those results, InfluxDB first calculates the average `of_capacity`s at 3-day intervals. +This step is the same as using the `MEAN()` function with the `GROUP BY time()` clause and without `ACOS()`: + +```sql +> SELECT MEAN("of_capacity") FROM "park_occupancy" WHERE time >= '2017-05-01T00:00:00Z' AND time <= '2017-05-09T00:00:00Z' GROUP BY time(3d) + +name: park_occupancy +time mean +---- ---- +2017-04-30T00:00:00Z 0.565 +2017-05-03T00:00:00Z 0.41 +2017-05-06T00:00:00Z 0.71 +2017-05-09T00:00:00Z 0.16 +``` + +InfluxDB then calculates arccosine of those averages. + +### ASIN() + +Returns the arcsine (in radians) of the field value. Field values must be between -1 and 1. + +#### Basic syntax + +``` +SELECT ASIN( [ * | ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`ASIN(field_key)` +Returns the arcsine of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + + + +`ASIN(*)` +Returns the arcsine of field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`ASIN()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types) with values between -1 and 1. + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `ASIN()` with a `GROUP BY time()` clause. + +##### Examples + +The examples below use the following data sample of simulated park occupancy relative to total capacity. The important thing to note is that all field values fall within the calculable range (-1 to 1) of the `ASIN()` function: + +```sql +> SELECT "of_capacity" FROM "park_occupancy" WHERE time >= '2017-05-01T00:00:00Z' AND time <= '2017-05-09T00:00:00Z' + +name: park_occupancy +time capacity +---- -------- +2017-05-01T00:00:00Z 0.83 +2017-05-02T00:00:00Z 0.3 +2017-05-03T00:00:00Z 0.84 +2017-05-04T00:00:00Z 0.22 +2017-05-05T00:00:00Z 0.17 +2017-05-06T00:00:00Z 0.77 +2017-05-07T00:00:00Z 0.64 +2017-05-08T00:00:00Z 0.72 +2017-05-09T00:00:00Z 0.16 +``` + +###### Calculate the arcsine of field values associated with a field key + +```sql +> SELECT ASIN("of_capacity") FROM "park_occupancy" WHERE time >= '2017-05-01T00:00:00Z' AND time <= '2017-05-09T00:00:00Z' + +name: park_occupancy +time asin +---- ---- +2017-05-01T00:00:00Z 0.9791076843683526 +2017-05-02T00:00:00Z 0.3046926540153975 +2017-05-03T00:00:00Z 0.9972832223717997 +2017-05-04T00:00:00Z 0.22181447049679442 +2017-05-05T00:00:00Z 0.1708296691291045 +2017-05-06T00:00:00Z 0.8788411516685797 +2017-05-07T00:00:00Z 0.6944982656265559 +2017-05-08T00:00:00Z 0.8038023189330299 +2017-05-09T00:00:00Z 0.1606906529519106 +``` + +The query returns arcsine of field values in the `of_capacity` field key in the `park_capacity` measurement. + +###### Calculate the arcsine of field values associated with each field key in a measurement + +```sql +> SELECT ASIN(*) FROM "park_occupancy" WHERE time >= '2017-05-01T00:00:00Z' AND time <= '2017-05-09T00:00:00Z' + +name: park_occupancy +time asin_of_capacity +---- ------------- +2017-05-01T00:00:00Z 0.9791076843683526 +2017-05-02T00:00:00Z 0.3046926540153975 +2017-05-03T00:00:00Z 0.9972832223717997 +2017-05-04T00:00:00Z 0.22181447049679442 +2017-05-05T00:00:00Z 0.1708296691291045 +2017-05-06T00:00:00Z 0.8788411516685797 +2017-05-07T00:00:00Z 0.6944982656265559 +2017-05-08T00:00:00Z 0.8038023189330299 +2017-05-09T00:00:00Z 0.1606906529519106 +``` + +The query returns arcsine of field values for each field key that stores numerical values in the `park_capacity` measurement. +The `h2o_feet` measurement has one numerical field: `of_capacity`. + + + +###### Calculate the arcsine of field values associated with a field key and include several clauses + +```sql +> SELECT ASIN("of_capacity") FROM "park_occupancy" WHERE time >= '2017-05-01T00:00:00Z' AND time <= '2017-05-09T00:00:00Z' ORDER BY time DESC LIMIT 4 OFFSET 2 + +name: park_occupancy +time asin +---- ---- +2017-05-07T00:00:00Z 0.6944982656265559 +2017-05-06T00:00:00Z 0.8788411516685797 +2017-05-05T00:00:00Z 0.1708296691291045 +2017-05-04T00:00:00Z 0.22181447049679442 +``` + +The query returns arcsine of field values associated with the `of_capacity` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2017-05-01T00:00:00Z` and `2017-05-09T00:00:00Z` and returns results in [descending timestamp order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to four and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by two points. + +#### Advanced syntax + +``` +SELECT ASIN(( [ * | ] )) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `ASIN()` function to those results. + +`ASIN()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples + +###### Calculate the arcsine of mean values. + +```sql +> SELECT ASIN(MEAN("of_capacity")) FROM "park_occupancy" WHERE time >= '2017-05-01T00:00:00Z' AND time <= '2017-05-09T00:00:00Z' GROUP BY time(3d) + +name: park_occupancy +time asin +---- ---- +2017-04-30T00:00:00Z 0.6004332535805232 +2017-05-03T00:00:00Z 0.42245406218675574 +2017-05-06T00:00:00Z 0.7894982093461719 +2017-05-09T00:00:00Z 0.1606906529519106 +``` + +The query returns arcsine of [average](#mean) `of_capacity`s that are calculated at 3-day intervals. + +To get those results, InfluxDB first calculates the average `of_capacity`s at 3-day intervals. +This step is the same as using the `MEAN()` function with the `GROUP BY time()` clause and without `ASIN()`: + +```sql +> SELECT MEAN("of_capacity") FROM "park_occupancy" WHERE time >= '2017-05-01T00:00:00Z' AND time <= '2017-05-09T00:00:00Z' GROUP BY time(3d) + +name: park_occupancy +time mean +---- ---- +2017-04-30T00:00:00Z 0.565 +2017-05-03T00:00:00Z 0.41 +2017-05-06T00:00:00Z 0.71 +2017-05-09T00:00:00Z 0.16 +``` + +InfluxDB then calculates arcsine of those averages. + +### ATAN() + +Returns the arctangent (in radians) of the field value. Field values must be between -1 and 1. + +#### Basic syntax + +```sql +SELECT ATAN( [ * | ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`ATAN(field_key)` +Returns the arctangent of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + + + +`ATAN(*)` +Returns the arctangent of field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`ATAN()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types) with values between -1 and 1. + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `ATAN()` with a `GROUP BY time()` clause. + +##### Examples + +The examples below use the following data sample of simulated park occupancy relative to total capacity. The important thing to note is that all field values fall within the calculable range (-1 to 1) of the `ATAN()` function: + +```sql +> SELECT "of_capacity" FROM "park_occupancy" WHERE time >= '2017-05-01T00:00:00Z' AND time <= '2017-05-09T00:00:00Z' + +name: park_occupancy +time capacity +---- -------- +2017-05-01T00:00:00Z 0.83 +2017-05-02T00:00:00Z 0.3 +2017-05-03T00:00:00Z 0.84 +2017-05-04T00:00:00Z 0.22 +2017-05-05T00:00:00Z 0.17 +2017-05-06T00:00:00Z 0.77 +2017-05-07T00:00:00Z 0.64 +2017-05-08T00:00:00Z 0.72 +2017-05-09T00:00:00Z 0.16 +``` + +###### Calculate the arctangent of field values associated with a field key + +```sql +> SELECT ATAN("of_capacity") FROM "park_occupancy" WHERE time >= '2017-05-01T00:00:00Z' AND time <= '2017-05-09T00:00:00Z' + +name: park_occupancy +time atan +---- ---- +2017-05-01T00:00:00Z 0.6927678353971222 +2017-05-02T00:00:00Z 0.2914567944778671 +2017-05-03T00:00:00Z 0.6986598247214632 +2017-05-04T00:00:00Z 0.2165503049760893 +2017-05-05T00:00:00Z 0.16839015714752992 +2017-05-06T00:00:00Z 0.6561787179913948 +2017-05-07T00:00:00Z 0.5693131911006619 +2017-05-08T00:00:00Z 0.6240230529767568 +2017-05-09T00:00:00Z 0.1586552621864014 +``` + +The query returns arctangent of field values in the `of_capacity` field key in the `park_occupancy` measurement. + +###### Calculate the arctangent of field values associated with each field key in a measurement + +```sql +> SELECT ATAN(*) FROM "park_occupancy" WHERE time >= '2017-05-01T00:00:00Z' AND time <= '2017-05-09T00:00:00Z' + +name: park_occupancy +time atan_of_capacity +---- ------------- +2017-05-01T00:00:00Z 0.6927678353971222 +2017-05-02T00:00:00Z 0.2914567944778671 +2017-05-03T00:00:00Z 0.6986598247214632 +2017-05-04T00:00:00Z 0.2165503049760893 +2017-05-05T00:00:00Z 0.16839015714752992 +2017-05-06T00:00:00Z 0.6561787179913948 +2017-05-07T00:00:00Z 0.5693131911006619 +2017-05-08T00:00:00Z 0.6240230529767568 +2017-05-09T00:00:00Z 0.1586552621864014 +``` + +The query returns arctangent of field values for each field key that stores numerical values in the `park_occupancy` measurement. +The `park_occupancy` measurement has one numerical field: `of_capacity`. + + + +###### Calculate the arctangent of field values associated with a field key and include several clauses + +```sql +> SELECT ATAN("of_capacity") FROM "park_occupancy" WHERE time >= '2017-05-01T00:00:00Z' AND time <= '2017-05-09T00:00:00Z' ORDER BY time DESC LIMIT 4 OFFSET 2 + +name: park_occupancy +time atan +---- ---- +2017-05-07T00:00:00Z 0.5693131911006619 +2017-05-06T00:00:00Z 0.6561787179913948 +2017-05-05T00:00:00Z 0.16839015714752992 +2017-05-04T00:00:00Z 0.2165503049760893 +``` + +The query returns arctangent of field values associated with the `of_capacity` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2017-05-01T00:00:00Z` and `2017-05-09T00:00:00Z` and returns results in [descending timestamp order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to four and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by two points. + +#### Advanced syntax + +``` +SELECT ATAN(( [ * | ] )) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `ATAN()` function to those results. + +`ATAN()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples of advanced syntax + +###### Calculate the arctangent of mean values. + +```sql +> SELECT ATAN(MEAN("of_capacity")) FROM "park_occupancy" WHERE time >= '2017-05-01T00:00:00Z' AND time <= '2017-05-09T00:00:00Z' GROUP BY time(3d) + +name: park_occupancy +time atan +---- ---- +2017-04-30T00:00:00Z 0.5142865412694495 +2017-05-03T00:00:00Z 0.3890972310552784 +2017-05-06T00:00:00Z 0.6174058917515726 +2017-05-09T00:00:00Z 0.1586552621864014 +``` + +The query returns arctangent of [average](#mean) `of_capacity`s that are calculated at 3-day intervals. + +To get those results, InfluxDB first calculates the average `of_capacity`s at 3-day intervals. +This step is the same as using the `MEAN()` function with the `GROUP BY time()` clause and without `ATAN()`: + +```sql +> SELECT MEAN("of_capacity") FROM "park_occupancy" WHERE time >= '2017-05-01T00:00:00Z' AND time <= '2017-05-09T00:00:00Z' GROUP BY time(3d) + +name: park_occupancy +time mean +---- ---- +2017-04-30T00:00:00Z 0.565 +2017-05-03T00:00:00Z 0.41 +2017-05-06T00:00:00Z 0.71 +2017-05-09T00:00:00Z 0.16 +``` + +InfluxDB then calculates arctangent of those averages. + +### ATAN2() + +Returns the the arctangent of `y/x` in radians. + +#### Basic syntax + +``` +SELECT ATAN2( [ * | | num ], [ | num ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`ATAN2(field_key_y, field_key_x)` +Returns the arctangent of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key), `field_key_y`, divided by field values associated with `field_key_x`. + + + +`ATAN2(*, field_key_x)` +Returns the field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) +divided by field values associated with `field_key_x`. + +`ATAN2()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `ATAN2()` with a `GROUP BY time()` clause. + +##### Examples + +The examples below use the following sample of simulated flight data: + +```sql +> SELECT "altitude_ft", "distance_ft" FROM "flight_data" WHERE time >= '2018-05-16T12:01:00Z' AND time <= '2018-05-16T12:10:00Z' + +name: flight_data +time altitude_ft distance_ft +---- ----------- ----------- +2018-05-16T12:01:00Z 1026 50094 +2018-05-16T12:02:00Z 2549 53576 +2018-05-16T12:03:00Z 4033 55208 +2018-05-16T12:04:00Z 5579 58579 +2018-05-16T12:05:00Z 7065 61213 +2018-05-16T12:06:00Z 8589 64807 +2018-05-16T12:07:00Z 10180 67707 +2018-05-16T12:08:00Z 11777 69819 +2018-05-16T12:09:00Z 13321 72452 +2018-05-16T12:10:00Z 14885 75881 +``` + +###### Calculate the arctangent of field_key_y over field_key_x + +```sql +> SELECT ATAN2("altitude_ft", "distance_ft") FROM "flight_data" WHERE time >= '2018-05-16T12:01:00Z' AND time <= '2018-05-16T12:10:00Z' + +name: flight_data +time atan2 +---- ----- +2018-05-16T12:01:00Z 0.020478631571881498 +2018-05-16T12:02:00Z 0.04754142349303296 +2018-05-16T12:03:00Z 0.07292147724575364 +2018-05-16T12:04:00Z 0.09495251193874832 +2018-05-16T12:05:00Z 0.11490822875441563 +2018-05-16T12:06:00Z 0.13176409347584003 +2018-05-16T12:07:00Z 0.14923587589682233 +2018-05-16T12:08:00Z 0.1671059946640312 +2018-05-16T12:09:00Z 0.18182893717409565 +2018-05-16T12:10:00Z 0.1937028631495223 +``` + +The query returns the arctangents of field values in the `altitude_ft` field key divided by values in the `distance_ft` field key. Both are part of the `flight_data` measurement. + +###### Calculate the arctangent of values associated with each field key in a measurement divided by field_key_x + +```sql +> SELECT ATAN2(*, "distance_ft") FROM "flight_data" WHERE time >= '2018-05-16T12:01:00Z' AND time <= '2018-05-16T12:10:00Z' + +name: flight_data +time atan2_altitude_ft atan2_distance_ft +---- ----------------- ----------------- +2018-05-16T12:01:00Z 0.020478631571881498 0.7853981633974483 +2018-05-16T12:02:00Z 0.04754142349303296 0.7853981633974483 +2018-05-16T12:03:00Z 0.07292147724575364 0.7853981633974483 +2018-05-16T12:04:00Z 0.09495251193874832 0.7853981633974483 +2018-05-16T12:05:00Z 0.11490822875441563 0.7853981633974483 +2018-05-16T12:06:00Z 0.13176409347584003 0.7853981633974483 +2018-05-16T12:07:00Z 0.14923587589682233 0.7853981633974483 +2018-05-16T12:08:00Z 0.1671059946640312 0.7853981633974483 +2018-05-16T12:09:00Z 0.18182893717409565 0.7853981633974483 +2018-05-16T12:10:00Z 0.19370286314952234 0.7853981633974483 +``` + +The query returns the arctangents of all numeric field values in the `flight_data` measurement divided by values in the `distance_ft` field key. +The `flight_data` measurement has two numeric fields: `altitude_ft` and `distance_ft`. + + + +###### Calculate the arctangents of field values and include several clauses + +```sql +> SELECT ATAN2("altitude_ft", "distance_ft") FROM "flight_data" WHERE time >= '2018-05-16T12:01:00Z' AND time <= '2018-05-16T12:10:00Z' ORDER BY time DESC LIMIT 4 OFFSET 2 + +name: flight_data +time atan2 +---- ----- +2018-05-16T12:08:00Z 0.1671059946640312 +2018-05-16T12:07:00Z 0.14923587589682233 +2018-05-16T12:06:00Z 0.13176409347584003 +2018-05-16T12:05:00Z 0.11490822875441563 +``` + +The query returns the arctangent of field values associated with the `altitude_ft` field key divided by the `distance_ft` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2018-05-16T12:10:00Z` and `2018-05-16T12:10:00Z` and returns results in [descending timestamp order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to four and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by two points. + +#### Advanced syntax + +``` +SELECT ATAN2(, ) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `ATAN2()` function to those results. + +`ATAN2()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples + +###### Calculate arctangents of mean values + +```sql +> SELECT ATAN2(MEAN("altitude_ft"), MEAN("distance_ft")) FROM "flight_data" WHERE time >= '2018-05-16T12:01:00Z' AND time <= '2018-05-16T13:01:00Z' GROUP BY time(12m) + +name: flight_data +time atan2 +---- ----- +2018-05-16T12:00:00Z 0.133815587896842 +2018-05-16T12:12:00Z 0.2662716308351908 +2018-05-16T12:24:00Z 0.2958845306108965 +2018-05-16T12:36:00Z 0.23783439588429497 +2018-05-16T12:48:00Z 0.1906803720242831 +2018-05-16T13:00:00Z 0.17291511946158172 +``` + +The query returns the argtangents of [average](#mean) `altitude_ft`s divided by average `distance_ft`s. Averages are calculated at 12-minute intervals. + +To get those results, InfluxDB first calculates the average `altitude_ft`s and `distance_ft` at 12-minute intervals. +This step is the same as using the `MEAN()` function with the `GROUP BY time()` clause and without `ATAN2()`: +^ +```sql +> SELECT MEAN("altitude_ft"), MEAN("distance_ft") FROM "flight_data" WHERE time >= '2018-05-16T12:01:00Z' AND time <= '2018-05-16T13:01:00Z' GROUP BY time(12m) + +name: flight_data +time mean mean_1 +---- ---- ------ +2018-05-16T12:00:00Z 8674 64433.181818181816 +2018-05-16T12:12:00Z 26419.833333333332 96865.25 +2018-05-16T12:24:00Z 40337.416666666664 132326.41666666666 +2018-05-16T12:36:00Z 41149.583333333336 169743.16666666666 +2018-05-16T12:48:00Z 41230.416666666664 213600.91666666666 +2018-05-16T13:00:00Z 41184.5 235799 +``` + +InfluxDB then calculates the arctangents of those averages. + +### CEIL() + +Returns the subsequent value rounded up to the nearest integer. + +#### Basic syntax + +``` +SELECT CEIL( [ * | ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`CEIL(field_key)` +Returns the field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) rounded up to the nearest integer. + + + +`CEIL(*)` +Returns the field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) rounded up to the nearest integer. + +`CEIL()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `CEIL()` with a `GROUP BY time()` clause. + +##### Examples + +The examples below use the following subsample of the [`NOAA_water_database` data](/enterprise_influxdb/v1.9/query_language/data_download/): + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +2015-08-18T00:18:00Z 2.126 +2015-08-18T00:24:00Z 2.041 +2015-08-18T00:30:00Z 2.051 +``` + +###### Calculate the ceiling of field values associated with a field key + +```sql +> SELECT CEIL("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time ceil +---- ---- +2015-08-18T00:00:00Z 3 +2015-08-18T00:06:00Z 3 +2015-08-18T00:12:00Z 3 +2015-08-18T00:18:00Z 3 +2015-08-18T00:24:00Z 3 +2015-08-18T00:30:00Z 3 +``` + +The query returns field values in the `water_level` field key in the `h2o_feet` measurement rounded up to the nearest integer. + +###### Calculate the ceiling of field values associated with each field key in a measurement + +```sql +> SELECT CEIL(*) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time ceil_water_level +---- ---------------- +2015-08-18T00:00:00Z 3 +2015-08-18T00:06:00Z 3 +2015-08-18T00:12:00Z 3 +2015-08-18T00:18:00Z 3 +2015-08-18T00:24:00Z 3 +2015-08-18T00:30:00Z 3 +``` + +The query returns field values for each field key that stores numerical values in the `h2o_feet` measurement rounded up to the nearest integer. +The `h2o_feet` measurement has one numerical field: `water_level`. + + + +###### Calculate the ceiling of field values associated with a field key and include several clauses + +```sql +> SELECT CEIL("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' ORDER BY time DESC LIMIT 4 OFFSET 2 + +name: h2o_feet +time ceil +---- ---- +2015-08-18T00:18:00Z 3 +2015-08-18T00:12:00Z 3 +2015-08-18T00:06:00Z 3 +2015-08-18T00:00:00Z 3 +``` + +The query returns field values associated with the `water_level` field key rounded up to the nearest integer. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-18T00:00:00Z` and `2015-08-18T00:30:00Z` and returns results in [descending timestamp order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to four and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by two points. + +#### Advanced syntax + +``` +SELECT CEIL(( [ * | | // ] )) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `CEIL()` function to those results. + +`CEIL()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples + +###### Calculate mean values rounded up to the nearest integer + +```sql +> SELECT CEIL(MEAN("water_level")) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time ceil +---- ---- +2015-08-18T00:00:00Z 3 +2015-08-18T00:12:00Z 3 +2015-08-18T00:24:00Z 3 +``` + +The query returns the [average](#mean) `water_level`s that are calculated at 12-minute intervals and rounds them up to the nearest integer. + +To get those results, InfluxDB first calculates the average `water_level`s at 12-minute intervals. +This step is the same as using the `MEAN()` function with the `GROUP BY time()` clause and without `CEIL()`: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time mean +---- ---- +2015-08-18T00:00:00Z 2.09 +2015-08-18T00:12:00Z 2.077 +2015-08-18T00:24:00Z 2.0460000000000003 +``` + +InfluxDB then rounds those averages up to the nearest integer. + +### COS() + +Returns the cosine of the field value. + +#### Basic syntax + +``` +SELECT COS( [ * | ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`COS(field_key)` +Returns the cosine of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + + + +`COS(*)` +Returns the cosine of field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`COS()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `COS()` with a `GROUP BY time()` clause. + +##### Examples + +The examples below use the following subsample of the [`NOAA_water_database` data](/enterprise_influxdb/v1.9/query_language/data_download/): +^ +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +2015-08-18T00:18:00Z 2.126 +2015-08-18T00:24:00Z 2.041 +2015-08-18T00:30:00Z 2.051 +``` + +###### Calculate the cosine of field values associated with a field key + +```sql +> SELECT COS("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time cos +---- --- +2015-08-18T00:00:00Z -0.47345017433543124 +2015-08-18T00:06:00Z -0.5185922462666872 +2015-08-18T00:12:00Z -0.4414407189100776 +2015-08-18T00:18:00Z -0.5271163912192579 +2015-08-18T00:24:00Z -0.45306786455514825 +2015-08-18T00:30:00Z -0.4619598230611262 +``` + +The query returns cosine of field values in the `water_level` field key in the `h2o_feet` measurement. + +###### Calculate the cosine of field values associated with each field key in a measurement + +```sql +> SELECT COS(*) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time cos_water_level +---- --------------- +2015-08-18T00:00:00Z -0.47345017433543124 +2015-08-18T00:06:00Z -0.5185922462666872 +2015-08-18T00:12:00Z -0.4414407189100776 +2015-08-18T00:18:00Z -0.5271163912192579 +2015-08-18T00:24:00Z -0.45306786455514825 +2015-08-18T00:30:00Z -0.4619598230611262 +``` + +The query returns cosine of field values for each field key that stores numerical values in the `h2o_feet` measurement. +The `h2o_feet` measurement has one numerical field: `water_level`. + + + +###### Calculate the cosine of field values associated with a field key and include several clauses + +```sql +> SELECT COS("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' ORDER BY time DESC LIMIT 4 OFFSET 2 + +name: h2o_feet +time cos +---- --- +2015-08-18T00:18:00Z -0.5271163912192579 +2015-08-18T00:12:00Z -0.4414407189100776 +2015-08-18T00:06:00Z -0.5185922462666872 +2015-08-18T00:00:00Z -0.47345017433543124 +``` + +The query returns cosine of field values associated with the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-18T00:00:00Z` and `2015-08-18T00:30:00Z` and returns results in [descending timestamp order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to four and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by two points. + +#### Advanced syntax + +``` +SELECT COS(( [ * | ] )) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `COS()` function to those results. + +`COS()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +#### Examples + +###### Calculate the cosine of mean values + +```sql +> SELECT COS(MEAN("water_level")) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time cos +---- --- +2015-08-18T00:00:00Z -0.49618891270599885 +2015-08-18T00:12:00Z -0.4848605136571181 +2015-08-18T00:24:00Z -0.4575195627907578 +``` + +The query returns cosine of [average](#mean) `water_level`s that are calculated at 12-minute intervals. + +To get those results, InfluxDB first calculates the average `water_level`s at 12-minute intervals. +This step is the same as using the `MEAN()` function with the `GROUP BY time()` clause and without `COS()`: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time mean +---- ---- +2015-08-18T00:00:00Z 2.09 +2015-08-18T00:12:00Z 2.077 +2015-08-18T00:24:00Z 2.0460000000000003 +``` + +InfluxDB then calculates cosine of those averages. + +### CUMULATIVE_SUM() + +Returns the running total of subsequent [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). + +#### Basic syntax + +``` +SELECT CUMULATIVE_SUM( [ * | | // ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`CUMULATIVE_SUM(field_key)` +Returns the running total of subsequent field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`CUMULATIVE_SUM(/regular_expression/)` +Returns the running total of subsequent field values associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +`CUMULATIVE_SUM(*)` +Returns the running total of subsequent field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`CUMULATIVE_SUM()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `CUMULATIVE_SUM()` with a `GROUP BY time()` clause. + +##### Examples + +The examples below use the following subsample of the [`NOAA_water_database` data](/enterprise_influxdb/v1.9/query_language/data_download/): + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +2015-08-18T00:18:00Z 2.126 +2015-08-18T00:24:00Z 2.041 +2015-08-18T00:30:00Z 2.051 +``` + +###### Calculate the cumulative sum of the field values associated with a field key + +```sql +> SELECT CUMULATIVE_SUM("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time cumulative_sum +---- -------------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 4.18 +2015-08-18T00:12:00Z 6.208 +2015-08-18T00:18:00Z 8.334 +2015-08-18T00:24:00Z 10.375 +2015-08-18T00:30:00Z 12.426 +``` + +The query returns the running total of the field values in the `water_level` field key and in the `h2o_feet` measurement. + +###### Calculate the cumulative sum of the field values associated with each field key in a measurement + +```sql +> SELECT CUMULATIVE_SUM(*) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time cumulative_sum_water_level +---- -------------------------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 4.18 +2015-08-18T00:12:00Z 6.208 +2015-08-18T00:18:00Z 8.334 +2015-08-18T00:24:00Z 10.375 +2015-08-18T00:30:00Z 12.426 +``` + +The query returns the running total of the field values for each field key that stores numerical values in the `h2o_feet` measurement. +The `h2o_feet` measurement has one numerical field: `water_level`. + +###### Calculate the cumulative sum of the field values associated with each field key that matches a regular expression + +```sql +> SELECT CUMULATIVE_SUM(/water/) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time cumulative_sum_water_level +---- -------------------------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 4.18 +2015-08-18T00:12:00Z 6.208 +2015-08-18T00:18:00Z 8.334 +2015-08-18T00:24:00Z 10.375 +2015-08-18T00:30:00Z 12.426 +``` + +The query returns the running total of the field values for each field key that stores numerical values and includes the word `water` in the `h2o_feet` measurement. + +###### Calculate the cumulative sum of the field values associated with a field key and include several clauses + +```sql +> SELECT CUMULATIVE_SUM("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' ORDER BY time DESC LIMIT 4 OFFSET 2 + +name: h2o_feet +time cumulative_sum +---- -------------- +2015-08-18T00:18:00Z 6.218 +2015-08-18T00:12:00Z 8.246 +2015-08-18T00:06:00Z 10.362 +2015-08-18T00:00:00Z 12.426 +``` + +The query returns the running total of the field values associated with the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-18T00:00:00Z` and `2015-08-18T00:30:00Z` and returns results in [descending timestamp order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to four and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by two points. + +#### Advanced syntax + +``` +SELECT CUMULATIVE_SUM(( [ * | | // ] )) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `CUMULATIVE_SUM()` function to those results. + +`CUMULATIVE_SUM()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples + +###### Calculate the cumulative sum of mean values + +```sql +> SELECT CUMULATIVE_SUM(MEAN("water_level")) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time cumulative_sum +---- -------------- +2015-08-18T00:00:00Z 2.09 +2015-08-18T00:12:00Z 4.167 +2015-08-18T00:24:00Z 6.213 +``` + +The query returns the running total of [average](#mean) `water_level`s that are calculated at 12-minute intervals. + +To get those results, InfluxDB first calculates the average `water_level`s at 12-minute intervals. +This step is the same as using the `MEAN()` function with the `GROUP BY time()` clause and without `CUMULATIVE_SUM()`: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time mean +---- ---- +2015-08-18T00:00:00Z 2.09 +2015-08-18T00:12:00Z 2.077 +2015-08-18T00:24:00Z 2.0460000000000003 +``` + +Next, InfluxDB calculates the running total of those averages. +The second point in the final results (`4.167`) is the sum of `2.09` and `2.077` +and the third point (`6.213`) is the sum of `2.09`, `2.077`, and `2.0460000000000003`. + +### DERIVATIVE() + +Returns the rate of change between subsequent [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). + +#### Basic syntax + +``` +SELECT DERIVATIVE( [ * | | // ] [ , ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +InfluxDB calculates the difference between subsequent field values and converts those results into the rate of change per `unit`. +The `unit` argument is an integer followed by a [duration literal](/enterprise_influxdb/v1.9/query_language/spec/#literals) and it is optional. +If the query does not specify the `unit` the unit defaults to one second (`1s`). + +`DERIVATIVE(field_key)` +Returns the rate of change between subsequent field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`DERIVATIVE(/regular_expression/)` +Returns the rate of change between subsequent field values associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +`DERIVATIVE(*)` +Returns the rate of change between subsequent field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`DERIVATIVE()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax-1) section for how to use `DERIVATIVE()` with a `GROUP BY time()` clause. + +##### Examples + +Examples 1-5 use the following subsample of the [`NOAA_water_database` data](/enterprise_influxdb/v1.9/query_language/data_download/): + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +2015-08-18T00:18:00Z 2.126 +2015-08-18T00:24:00Z 2.041 +2015-08-18T00:30:00Z 2.051 +``` + +###### Calculate the derivative between the field values associated with a field key + +```sql +> SELECT DERIVATIVE("water_level") FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' + +name: h2o_feet +time derivative +---- ---------- +2015-08-18T00:06:00Z 0.00014444444444444457 +2015-08-18T00:12:00Z -0.00024444444444444465 +2015-08-18T00:18:00Z 0.0002722222222222218 +2015-08-18T00:24:00Z -0.000236111111111111 +2015-08-18T00:30:00Z 2.777777777777842e-05 +``` + +The query returns the one-second rate of change between the field values associated with the `water_level` field key and in the `h2o_feet` measurement. + +The first result (`0.00014444444444444457`) is the one-second rate of change between the first two subsequent field values in the raw data. +InfluxDB calculates the difference between the field values and normalizes that value to the one-second rate of change: + +``` +(2.116 - 2.064) / (360s / 1s) +-------------- ---------- + | | + | the difference between the field values' timestamps / the default unit +second field value - first field value +``` + +###### Calculate the derivative between the field values associated with a field key and specify the unit option + +```sql +> SELECT DERIVATIVE("water_level",6m) FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' + +name: h2o_feet +time derivative +---- ---------- +2015-08-18T00:06:00Z 0.052000000000000046 +2015-08-18T00:12:00Z -0.08800000000000008 +2015-08-18T00:18:00Z 0.09799999999999986 +2015-08-18T00:24:00Z -0.08499999999999996 +2015-08-18T00:30:00Z 0.010000000000000231 +``` + +The query returns the six-minute rate of change between the field values associated with the `water_level` field key and in the `h2o_feet` measurement. + +The first result (`0.052000000000000046`) is the six-minute rate of change between the first two subsequent field values in the raw data. +InfluxDB calculates the difference between the field values and normalizes that value to the six-minute rate of change: + +``` +(2.116 - 2.064) / (6m / 6m) +-------------- ---------- + | | + | the difference between the field values' timestamps / the specified unit +second field value - first field value +``` + +###### Calculate the derivative between the field values associated with each field key in a measurement and specify the unit option + +```sql +> SELECT DERIVATIVE(*,3m) FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' + + +name: h2o_feet +time derivative_water_level +---- ---------------------- +2015-08-18T00:06:00Z 0.026000000000000023 +2015-08-18T00:12:00Z -0.04400000000000004 +2015-08-18T00:18:00Z 0.04899999999999993 +2015-08-18T00:24:00Z -0.04249999999999998 +2015-08-18T00:30:00Z 0.0050000000000001155 +``` + +The query returns the three-minute rate of change between the field values associated with each field key that stores numerical values in the `h2o_feet` measurement. +The `h2o_feet` measurement has one numerical field: `water_level`. + +The first result (`0.026000000000000023`) is the three-minute rate of change between the first two subsequent field values in the raw data. +InfluxDB calculates the difference between the field values and normalizes that value to the three-minute rate of change: + +``` +(2.116 - 2.064) / (6m / 3m) +-------------- ---------- + | | + | the difference between the field values' timestamps / the specified unit +second field value - first field value +``` + +###### Calculate the derivative between the field values associated with each field key that matches a regular expression and specify the unit option + +```sql +> SELECT DERIVATIVE(/water/,2m) FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' + +name: h2o_feet +time derivative_water_level +---- ---------------------- +2015-08-18T00:06:00Z 0.01733333333333335 +2015-08-18T00:12:00Z -0.02933333333333336 +2015-08-18T00:18:00Z 0.03266666666666662 +2015-08-18T00:24:00Z -0.02833333333333332 +2015-08-18T00:30:00Z 0.0033333333333334103 +``` + +The query returns the two-minute rate of change between the field values associated with each field key that stores numerical values and includes the word `water` in the `h2o_feet` measurement. + +The first result (`0.01733333333333335`) is the two-minute rate of change between the first two subsequent field values in the raw data. +InfluxDB calculates the difference between the field values and normalizes that value to the two-minute rate of change: + +``` +(2.116 - 2.064) / (6m / 2m) +-------------- ---------- + | | + | the difference between the field values' timestamps / the specified unit +second field value - first field value +``` + +###### Calculate the derivative between the field values associated with a field key and include several clauses + +```sql +> SELECT DERIVATIVE("water_level") FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' ORDER BY time DESC LIMIT 1 OFFSET 2 + +name: h2o_feet +time derivative +---- ---------- +2015-08-18T00:12:00Z -0.0002722222222222218 +``` + +The query returns the one-second rate of change between the field values associated with the `water_level` field key and in the `h2o_feet` measurement. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-18T00:00:00Z` and `2015-08-18T00:30:00Z` and returns results in [descending timestamp order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to one and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by two points. + +The only result (`-0.0002722222222222218`) is the one-second rate of change between the relevant subsequent field values in the raw data. +InfluxDB calculates the difference between the field values and normalizes that value to the one-second rate of change: + +``` +(2.126 - 2.028) / (360s / 1s) +-------------- ---------- + | | + | the difference between the field values' timestamps / the default unit +second field value - first field value +``` + +#### Advanced syntax + +``` +SELECT DERIVATIVE( ([ * | | // ]) [ , ] ) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `DERIVATIVE()` function to those results. + +The `unit` argument is an integer followed by a [duration literal](/enterprise_influxdb/v1.9/query_language/spec/#literals) and it is optional. +If the query does not specify the `unit` the `unit` defaults to the `GROUP BY time()` interval. +Note that this behavior is different from the [basic syntax's](#basic-syntax-1) default behavior. + +`DERIVATIVE()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples + +###### Calculate the derivative of mean values + +```sql +> SELECT DERIVATIVE(MEAN("water_level")) FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' GROUP BY time(12m) + +name: h2o_feet +time derivative +---- ---------- +2015-08-18T00:12:00Z -0.0129999999999999 +2015-08-18T00:24:00Z -0.030999999999999694 +``` + +The query returns the 12-minute rate of change between [average](#mean) `water_level`s that are calculated at 12-minute intervals. + +To get those results, InfluxDB first calculates the average `water_level`s at 12-minute intervals. +This step is the same as using the `MEAN()` function with the `GROUP BY time()` clause and without `DERIVATIVE()`: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' GROUP BY time(12m) + +name: h2o_feet +time mean +---- ---- +2015-08-18T00:00:00Z 2.09 +2015-08-18T00:12:00Z 2.077 +2015-08-18T00:24:00Z 2.0460000000000003 +``` + +Next, InfluxDB calculates the 12-minute rate of change between those averages. +The first result (`-0.0129999999999999`) is the 12-minute rate of change between the first two averages. +InfluxDB calculates the difference between the field values and normalizes that value to the 12-minute rate of change. + +``` +(2.077 - 2.09) / (12m / 12m) +------------- ---------- + | | + | the difference between the field values' timestamps / the default unit +second field value - first field value +``` + +###### Calculate the derivative of mean values and specify the unit option + +```sql +> SELECT DERIVATIVE(MEAN("water_level"),6m) FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' GROUP BY time(12m) + +name: h2o_feet +time derivative +---- ---------- +2015-08-18T00:12:00Z -0.00649999999999995 +2015-08-18T00:24:00Z -0.015499999999999847 +``` + +The query returns the six-minute rate of change between average `water_level`s that are calculated at 12-minute intervals. + +To get those results, InfluxDB first calculates the average `water_level`s at 12-minute intervals. +This step is the same as using the `MEAN()` function with the `GROUP BY time()` clause and without `DERIVATIVE()`: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' GROUP BY time(12m) + +name: h2o_feet +time mean +---- ---- +2015-08-18T00:00:00Z 2.09 +2015-08-18T00:12:00Z 2.077 +2015-08-18T00:24:00Z 2.0460000000000003 +``` + +Next, InfluxDB calculates the six-minute rate of change between those averages. +The first result (`-0.00649999999999995`) is the six-minute rate of change between the first two averages. +InfluxDB calculates the difference between the field values and normalizes that value to the six-minute rate of change. + +``` +(2.077 - 2.09) / (12m / 6m) +------------- ---------- + | | + | the difference between the field values' timestamps / the specified unit +second field value - first field value +``` + +### DIFFERENCE() + +Returns the result of subtraction between subsequent [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). + +#### Basic syntax + +``` +SELECT DIFFERENCE( [ * | | // ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`DIFFERENCE(field_key)` +Returns the difference between subsequent field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`DIFFERENCE(/regular_expression/)` +Returns the difference between subsequent field values associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +`DIFFERENCE(*)` +Returns the difference between subsequent field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`DIFFERENCE()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax-2) section for how to use `DIFFERENCE()` with a `GROUP BY time()` clause. + +##### Examples + +The examples below use the following subsample of the [`NOAA_water_database` data](/enterprise_influxdb/v1.9/query_language/data_download/): + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +2015-08-18T00:18:00Z 2.126 +2015-08-18T00:24:00Z 2.041 +2015-08-18T00:30:00Z 2.051 +``` + +###### Calculate the difference between the field values associated with a field key + +```sql +> SELECT DIFFERENCE("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time difference +---- ---------- +2015-08-18T00:06:00Z 0.052000000000000046 +2015-08-18T00:12:00Z -0.08800000000000008 +2015-08-18T00:18:00Z 0.09799999999999986 +2015-08-18T00:24:00Z -0.08499999999999996 +2015-08-18T00:30:00Z 0.010000000000000231 +``` + +The query returns the difference between the subsequent field values in the `water_level` field key and in the `h2o_feet` measurement. + +###### Calculate the difference between the field values associated with each field key in a measurement + +```sql +> SELECT DIFFERENCE(*) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time difference_water_level +---- ---------------------- +2015-08-18T00:06:00Z 0.052000000000000046 +2015-08-18T00:12:00Z -0.08800000000000008 +2015-08-18T00:18:00Z 0.09799999999999986 +2015-08-18T00:24:00Z -0.08499999999999996 +2015-08-18T00:30:00Z 0.010000000000000231 +``` + +The query returns the difference between the subsequent field values for each field key that stores numerical values in the `h2o_feet` measurement. +The `h2o_feet` measurement has one numerical field: `water_level`. + +###### Calculate the difference between the field values associated with each field key that matches a regular expression + +```sql +> SELECT DIFFERENCE(/water/) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time difference_water_level +---- ---------------------- +2015-08-18T00:06:00Z 0.052000000000000046 +2015-08-18T00:12:00Z -0.08800000000000008 +2015-08-18T00:18:00Z 0.09799999999999986 +2015-08-18T00:24:00Z -0.08499999999999996 +2015-08-18T00:30:00Z 0.010000000000000231 +``` + +The query returns the difference between the subsequent field values for each field key that stores numerical values and includes the word `water` in the `h2o_feet` measurement. + +###### Calculate the difference between the field values associated with a field key and include several clauses + +```sql +> SELECT DIFFERENCE("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' ORDER BY time DESC LIMIT 2 OFFSET 2 + +name: h2o_feet +time difference +---- ---------- +2015-08-18T00:12:00Z -0.09799999999999986 +2015-08-18T00:06:00Z 0.08800000000000008 +``` + +The query returns the difference between the subsequent field values in the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-18T00:00:00Z` and `2015-08-18T00:30:00Z` and returns results in [descending timestamp order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +They query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to two and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by two points. + +#### Advanced syntax + +``` +SELECT DIFFERENCE(( [ * | | // ] )) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `DIFFERENCE()` function to those results. + +`DIFFERENCE()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples + +###### Calculate the difference between maximum values + +```sql +> SELECT DIFFERENCE(MAX("water_level")) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time difference +---- ---------- +2015-08-18T00:12:00Z 0.009999999999999787 +2015-08-18T00:24:00Z -0.07499999999999973 +``` + +The query returns the difference between [maximum](#max) `water_level`s that are calculated at 12-minute intervals. + +To get those results, InfluxDB first calculates the maximum `water_level`s at 12-minute intervals. +This step is the same as using the `MAX()` function with the `GROUP BY time()` clause and without `DIFFERENCE()`: + +```sql +> SELECT MAX("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time max +---- --- +2015-08-18T00:00:00Z 2.116 +2015-08-18T00:12:00Z 2.126 +2015-08-18T00:24:00Z 2.051 +``` + +Next, InfluxDB calculates the difference between those maximum values. +The first point in the final results (`0.009999999999999787`) is the difference between `2.126` and `2.116`, and the second point in the final results (`-0.07499999999999973`) is the difference between `2.051` and `2.126`. + +### ELAPSED() + +Returns the difference between subsequent [field value's](/enterprise_influxdb/v1.9/concepts/glossary/#field-value) timestamps. + +#### Syntax + +``` +SELECT ELAPSED( [ * | | // ] [ , ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +InfluxDB calculates the difference between subsequent timestamps. +The `unit` option is an integer followed by a [duration literal](/enterprise_influxdb/v1.9/query_language/spec/#literals) and it determines the unit of the returned difference. +If the query does not specify the `unit` option the query returns the difference between timestamps in nanoseconds. + +`ELAPSED(field_key)` +Returns the difference between subsequent timestamps associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`ELAPSED(/regular_expression/)` +Returns the difference between subsequent timestamps associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +`ELAPSED(*)` +Returns the difference between subsequent timestamps associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`ELAPSED()` supports all field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +#### Examples + +The examples use the following subsample of the [`NOAA_water_database` data](/enterprise_influxdb/v1.9/query_language/data_download/): + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:12:00Z' + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +``` + +##### Calculate the elapsed time between field values associated with a field key + +```sql +> SELECT ELAPSED("water_level") FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:12:00Z' + +name: h2o_feet +time elapsed +---- ------- +2015-08-18T00:06:00Z 360000000000 +2015-08-18T00:12:00Z 360000000000 +``` + +The query returns the difference (in nanoseconds) between subsequent timestamps in the `water_level` field key and in the `h2o_feet` measurement. + +##### Calculate the elapsed time between field values associated with a field key and specify the unit option + +```sql +> SELECT ELAPSED("water_level",1m) FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:12:00Z' + +name: h2o_feet +time elapsed +---- ------- +2015-08-18T00:06:00Z 6 +2015-08-18T00:12:00Z 6 +``` + +The query returns the difference (in minutes) between subsequent timestamps in the `water_level` field key and in the `h2o_feet` measurement. + +##### Calculate the elapsed time between field values associated with each field key in a measurement and specify the unit option + +```sql +> SELECT ELAPSED(*,1m) FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:12:00Z' + +name: h2o_feet +time elapsed_level description elapsed_water_level +---- ------------------------- ------------------- +2015-08-18T00:06:00Z 6 6 +2015-08-18T00:12:00Z 6 6 +``` + +The query returns the difference (in minutes) between subsequent timestamps associated with each field key in the `h2o_feet` +measurement. +The `h2o_feet` measurement has two field keys: `level description` and `water_level`. + +##### Calculate the elapsed time between field values associated with each field key that matches a regular expression and specify the unit option + +```sql +> SELECT ELAPSED(/level/,1s) FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:12:00Z' + +name: h2o_feet +time elapsed_level description elapsed_water_level +---- ------------------------- ------------------- +2015-08-18T00:06:00Z 360 360 +2015-08-18T00:12:00Z 360 360 +``` + +The query returns the difference (in seconds) between subsequent timestamps associated with each field key that includes the word `level` in the `h2o_feet` measurement. + +##### Calculate the elapsed time between field values associated with a field key and include several clauses + +```sql +> SELECT ELAPSED("water_level",1ms) FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:12:00Z' ORDER BY time DESC LIMIT 1 OFFSET 1 + +name: h2o_feet +time elapsed +---- ------- +2015-08-18T00:00:00Z -360000 +``` + +The query returns the difference (in milliseconds) between subsequent timestamps in the `water_level` field key and in the `h2o_feet` measurement. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-18T00:00:00Z` and `2015-08-18T00:12:00Z` and sorts timestamps in [descending order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to one and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by one point. + +Notice that the result is negative; the [`ORDER BY time DESC` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc) sorts timestamps in descending order so `ELAPSED()` calculates the difference between timestamps in reverse order. + +### Common Issues with ELAPSED() + +#### ELAPSED() and units greater than the elapsed time + +InfluxDB returns `0` if the `unit` option is greater than the difference between the timestamps. + +##### Example + +The timestamps in the `h2o_feet` measurement occur at six-minute intervals. +If the query sets the `unit` option to one hour, InfluxDB returns `0`: + +```sql +> SELECT ELAPSED("water_level",1h) FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:12:00Z' + +name: h2o_feet +time elapsed +---- ------- +2015-08-18T00:06:00Z 0 +2015-08-18T00:12:00Z 0 +``` + +#### ELAPSED() with GROUP BY time() clauses + +The `ELAPSED()` function supports the [`GROUP BY time()` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) but the query results aren't particularly useful. +Currently, an `ELAPSED()` query with a nested function and a `GROUP BY time()` clause simply returns the interval specified in the `GROUP BY time()` clause. + +The `GROUP BY time()` clause determines the timestamps in the results; each timestamp marks the start of a time interval. +That behavior also applies to nested selector functions (like [`FIRST()`](#first) or [`MAX()`](#max)) which would, in all other cases, return a specific timestamp from the raw data. +Because the `GROUP BY time()` clause overrides the original timestamps, the `ELAPSED()` calculation always returns the same value as the `GROUP BY time()` interval. + +##### Example + +In the codeblock below, the first query attempts to use the `ELAPSED()` function with a `GROUP BY time()` clause to find the time elapsed (in minutes) between [minimum](#min) `water_level`s. +The query returns 12 minutes for both time intervals. + +To get those results, InfluxDB first calculates the minimum `water_level`s at 12-minute intervals. +The second query in the codeblock shows the results of that step. +The step is the same as using the `MIN()` function with the `GROUP BY time()` clause and without the `ELAPSED()` function. +Notice that the timestamps returned by the second query are 12 minutes apart. +In the raw data, the first result (`2.057`) occurs at `2015-08-18T00:42:00Z` but the `GROUP BY time()` clause overrides that original timestamp. +Because the timestamps are determined by the `GROUP BY time()` interval and not by the original data, the `ELAPSED()` calculation always returns the same value as the `GROUP BY time()` interval. + +```sql +> SELECT ELAPSED(MIN("water_level"),1m) FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:36:00Z' AND time <= '2015-08-18T00:54:00Z' GROUP BY time(12m) + +name: h2o_feet +time elapsed +---- ------- +2015-08-18T00:36:00Z 12 +2015-08-18T00:48:00Z 12 + +> SELECT MIN("water_level") FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:36:00Z' AND time <= '2015-08-18T00:54:00Z' GROUP BY time(12m) + +name: h2o_feet +time min +---- --- +2015-08-18T00:36:00Z 2.057 <--- Actually occurs at 2015-08-18T00:42:00Z +2015-08-18T00:48:00Z 1.991 +``` + +### EXP() + +Returns the exponential of the field value. + +#### Basic syntax + +``` +SELECT EXP( [ * | ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`EXP(field_key)` +Returns the exponential of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + + + +`EXP(*)` +Returns the exponential of field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`EXP()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `EXP()` with a `GROUP BY time()` clause. + +##### Examples + +The examples below use the following subsample of the [`NOAA_water_database` data](/enterprise_influxdb/v1.9/query_language/data_download/): + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +2015-08-18T00:18:00Z 2.126 +2015-08-18T00:24:00Z 2.041 +2015-08-18T00:30:00Z 2.051 +``` + +###### Calculate the exponential of field values associated with a field key + +```sql +> SELECT EXP("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time exp +---- --- +2015-08-18T00:00:00Z 7.877416541092307 +2015-08-18T00:06:00Z 8.297879498060171 +2015-08-18T00:12:00Z 7.598873404088091 +2015-08-18T00:18:00Z 8.381274573459967 +2015-08-18T00:24:00Z 7.6983036546645645 +2015-08-18T00:30:00Z 7.775672892658607 +``` + +The query returns the exponential of field values in the `water_level` field key in the `h2o_feet` measurement. + +###### Calculate the exponential of field values associated with each field key in a measurement + +```sql +> SELECT EXP(*) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time exp_water_level +---- --------------- +2015-08-18T00:00:00Z 7.877416541092307 +2015-08-18T00:06:00Z 8.297879498060171 +2015-08-18T00:12:00Z 7.598873404088091 +2015-08-18T00:18:00Z 8.381274573459967 +2015-08-18T00:24:00Z 7.6983036546645645 +2015-08-18T00:30:00Z 7.775672892658607 +``` + +The query returns the exponential of field values for each field key that stores numerical values in the `h2o_feet` measurement. +The `h2o_feet` measurement has one numerical field: `water_level`. + + + +###### Calculate the exponential of field values associated with a field key and include several clauses + +```sql +> SELECT EXP("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' ORDER BY time DESC LIMIT 4 OFFSET 2 + +name: h2o_feet +time exp +---- --- +2015-08-18T00:18:00Z 8.381274573459967 +2015-08-18T00:12:00Z 7.598873404088091 +2015-08-18T00:06:00Z 8.297879498060171 +2015-08-18T00:00:00Z 7.877416541092307 +``` + +The query returns the exponentials of field values associated with the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-18T00:00:00Z` and `2015-08-18T00:30:00Z` and returns results in [descending timestamp order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to four and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by two points. + +#### Advanced syntax + +``` +SELECT EXP(( [ * | ] )) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `EXP()` function to those results. + +`EXP()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples + +###### Calculate the exponential of mean values. + +```sql +> SELECT EXP(MEAN("water_level")) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time exp +---- --- +2015-08-18T00:00:00Z 8.084915164305059 +2015-08-18T00:12:00Z 7.980491491670466 +2015-08-18T00:24:00Z 7.736891562315577 +``` + +The query returns the exponential of [average](#mean) `water_level`s that are calculated at 12-minute intervals. + +To get those results, InfluxDB first calculates the average `water_level`s at 12-minute intervals. +This step is the same as using the `MEAN()` function with the `GROUP BY time()` clause and without `EXP()`: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time mean +---- ---- +2015-08-18T00:00:00Z 2.09 +2015-08-18T00:12:00Z 2.077 +2015-08-18T00:24:00Z 2.0460000000000003 +``` + +InfluxDB then calculates the exponentials of those averages. + +### FLOOR() + +Returns the subsequent value rounded down to the nearest integer. + +#### Basic syntax + +``` +SELECT FLOOR( [ * | ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`FLOOR(field_key)` +Returns the field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) rounded down to the nearest integer. + + + +`FLOOR(*)` +Returns the field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) rounded down to the nearest integer. + +`FLOOR()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `FLOOR()` with a `GROUP BY time()` clause. + +##### Examples + +The examples below use the following subsample of the [`NOAA_water_database` data](/enterprise_influxdb/v1.9/query_language/data_download/): + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +2015-08-18T00:18:00Z 2.126 +2015-08-18T00:24:00Z 2.041 +2015-08-18T00:30:00Z 2.051 +``` + +###### Calculate the floor of field values associated with a field key + +```sql +> SELECT FLOOR("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time floor +---- ----- +2015-08-18T00:00:00Z 2 +2015-08-18T00:06:00Z 2 +2015-08-18T00:12:00Z 2 +2015-08-18T00:18:00Z 2 +2015-08-18T00:24:00Z 2 +2015-08-18T00:30:00Z 2 +``` + +The query returns field values in the `water_level` field key in the `h2o_feet` measurement rounded down to the nearest integer. + +###### Calculate the floor of field values associated with each field key in a measurement + +```sql +> SELECT FLOOR(*) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time floor_water_level +---- ----------------- +2015-08-18T00:00:00Z 2 +2015-08-18T00:06:00Z 2 +2015-08-18T00:12:00Z 2 +2015-08-18T00:18:00Z 2 +2015-08-18T00:24:00Z 2 +2015-08-18T00:30:00Z 2 +``` + +The query returns field values for each field key that stores numerical values in the `h2o_feet` measurement rounded down to the nearest integer. +The `h2o_feet` measurement has one numerical field: `water_level`. + + + +###### Calculate the floor of field values associated with a field key and include several clauses + +```sql +> SELECT FLOOR("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' ORDER BY time DESC LIMIT 4 OFFSET 2 + +name: h2o_feet +time floor +---- ----- +2015-08-18T00:18:00Z 2 +2015-08-18T00:12:00Z 2 +2015-08-18T00:06:00Z 2 +2015-08-18T00:00:00Z 2 +``` + +The query returns field values associated with the `water_level` field key rounded down to the nearest integer. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-18T00:00:00Z` and `2015-08-18T00:30:00Z` and returns results in [descending timestamp order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to four and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by two points. + +#### Advanced syntax + +``` +SELECT FLOOR(( [ * | ] )) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `FLOOR()` function to those results. + +`FLOOR()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples + +###### Calculate mean values rounded down to the nearest integer. + +```sql +> SELECT FLOOR(MEAN("water_level")) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time floor +---- ----- +2015-08-18T00:00:00Z 2 +2015-08-18T00:12:00Z 2 +2015-08-18T00:24:00Z 2 +``` + +The query returns the [average](#mean) `water_level`s that are calculated at 12-minute intervals and rounds them up to the nearest integer. + +To get those results, InfluxDB first calculates the average `water_level`s at 12-minute intervals. +This step is the same as using the `MEAN()` function with the `GROUP BY time()` clause and without `FLOOR()`: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time mean +---- ---- +2015-08-18T00:00:00Z 2.09 +2015-08-18T00:12:00Z 2.077 +2015-08-18T00:24:00Z 2.0460000000000003 +``` + +InfluxDB then rounds those averages down to the nearest integer. + +### HISTOGRAM() + +_InfluxQL does not currently support histogram generation. +For information about creating histograms with data stored in InfluxDB, see +[Flux's `histogram()` function](/{{< latest "influxdb" "v2" >}}/reference/flux/stdlib/built-in/transformations/histogram)._ + +### LN() + +Returns the natural logarithm of the field value. + +#### Basic syntax + +``` +SELECT LN( [ * | ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`LN(field_key)` +Returns the natural logarithm of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + + + +`LN(*)` +Returns the natural logarithm of field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`LN()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `LN()` with a `GROUP BY time()` clause. + +##### Examples + +The examples below use the following subsample of the [`NOAA_water_database` data](/enterprise_influxdb/v1.9/query_language/data_download/): + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +2015-08-18T00:18:00Z 2.126 +2015-08-18T00:24:00Z 2.041 +2015-08-18T00:30:00Z 2.051 +``` + +###### Calculate the natural logarithm of field values associated with a field key + +```sql +> SELECT LN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time ln +---- -- +2015-08-18T00:00:00Z 0.7246458476193163 +2015-08-18T00:06:00Z 0.749527513996053 +2015-08-18T00:12:00Z 0.7070500857289368 +2015-08-18T00:18:00Z 0.7542422799197561 +2015-08-18T00:24:00Z 0.7134398838277077 +2015-08-18T00:30:00Z 0.7183274790902436 +``` + +The query returns the natural logarithm of field values in the `water_level` field key in the `h2o_feet` measurement. + +###### Calculate the natural logarithm of field values associated with each field key in a measurement + +```sql +> SELECT LN(*) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time ln_water_level +---- -------------- +2015-08-18T00:00:00Z 0.7246458476193163 +2015-08-18T00:06:00Z 0.749527513996053 +2015-08-18T00:12:00Z 0.7070500857289368 +2015-08-18T00:18:00Z 0.7542422799197561 +2015-08-18T00:24:00Z 0.7134398838277077 +2015-08-18T00:30:00Z 0.7183274790902436 +``` + +The query returns the natural logarithm of field values for each field key that stores numerical values in the `h2o_feet` measurement. +The `h2o_feet` measurement has one numerical field: `water_level`. + + + +###### Calculate the natural logarithm of field values associated with a field key and include several clauses + +```sql +> SELECT LN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' ORDER BY time DESC LIMIT 4 OFFSET 2 + +name: h2o_feet +time ln +---- -- +2015-08-18T00:18:00Z 0.7542422799197561 +2015-08-18T00:12:00Z 0.7070500857289368 +2015-08-18T00:06:00Z 0.749527513996053 +2015-08-18T00:00:00Z 0.7246458476193163 +``` + +The query returns the natural logarithms of field values associated with the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-18T00:00:00Z` and `2015-08-18T00:30:00Z` and returns results in [descending timestamp order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to four and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by two points. + +#### Advanced syntax + +``` +SELECT LN(( [ * | ] )) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `LN()` function to those results. + +`LN()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples + +###### Calculate the natural logarithm of mean values. + +```sql +> SELECT LN(MEAN("water_level")) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time ln +---- -- +2015-08-18T00:00:00Z 0.7371640659767196 +2015-08-18T00:12:00Z 0.7309245448939752 +2015-08-18T00:24:00Z 0.7158866675294349 +``` + +The query returns the natural logarithm of [average](#mean) `water_level`s that are calculated at 12-minute intervals. + +To get those results, InfluxDB first calculates the average `water_level`s at 12-minute intervals. +This step is the same as using the `MEAN()` function with the `GROUP BY time()` clause and without `LN()`: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time mean +---- ---- +2015-08-18T00:00:00Z 2.09 +2015-08-18T00:12:00Z 2.077 +2015-08-18T00:24:00Z 2.0460000000000003 +``` + +InfluxDB then calculates the natural logarithms of those averages. + +### LOG() + +Returns the logarithm of the field value with base `b`. + +#### Basic syntax + +``` +SELECT LOG( [ * | ], ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`LOG(field_key, b)` +Returns the logarithm of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) with base `b`. + + + +`LOG(*, b)` +Returns the logarithm of field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) with base `b`. + +`LOG()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `LOG()` with a `GROUP BY time()` clause. + +##### Examples + +The examples below use the following subsample of the [`NOAA_water_database` data](/enterprise_influxdb/v1.9/query_language/data_download/): + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +2015-08-18T00:18:00Z 2.126 +2015-08-18T00:24:00Z 2.041 +2015-08-18T00:30:00Z 2.051 +``` + +###### Calculate the logarithm base 4 of field values associated with a field key + +```sql +> SELECT LOG("water_level", 4) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time log +---- --- +2015-08-18T00:00:00Z 0.5227214853805835 +2015-08-18T00:06:00Z 0.5406698137259695 +2015-08-18T00:12:00Z 0.5100288261706268 +2015-08-18T00:18:00Z 0.5440707984345088 +2015-08-18T00:24:00Z 0.5146380911853161 +2015-08-18T00:30:00Z 0.5181637459088826 +``` + +The query returns the logarithm base 4 of field values in the `water_level` field key in the `h2o_feet` measurement. + +###### Calculate the logarithm base 4 of field values associated with each field key in a measurement + +```sql +> SELECT LOG(*, 4) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time log_water_level +---- --------------- +2015-08-18T00:00:00Z 0.5227214853805835 +2015-08-18T00:06:00Z 0.5406698137259695 +2015-08-18T00:12:00Z 0.5100288261706268 +2015-08-18T00:18:00Z 0.5440707984345088 +2015-08-18T00:24:00Z 0.5146380911853161 +2015-08-18T00:30:00Z 0.5181637459088826 +``` + +The query returns the logarithm base 4 of field values for each field key that stores numerical values in the `h2o_feet` measurement. +The `h2o_feet` measurement has one numerical field: `water_level`. + + + +###### Calculate the logarithm base 4 of field values associated with a field key and include several clauses + +```sql +> SELECT LOG("water_level", 4) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' ORDER BY time DESC LIMIT 4 OFFSET 2 + +name: h2o_feet +time log +---- --- +2015-08-18T00:18:00Z 0.5440707984345088 +2015-08-18T00:12:00Z 0.5100288261706268 +2015-08-18T00:06:00Z 0.5406698137259695 +2015-08-18T00:00:00Z 0.5227214853805835 +``` + +The query returns the logarithm base 4 of field values associated with the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-18T00:00:00Z` and `2015-08-18T00:30:00Z` and returns results in [descending timestamp order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to four and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by two points. + +#### Advanced syntax + +``` +SELECT LOG(( [ * | ] ), ) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `LOG()` function to those results. + +`LOG()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples + +###### Calculate the logarithm base 4 of mean values + +```sql +> SELECT LOG(MEAN("water_level"), 4) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time log +---- --- +2015-08-18T00:00:00Z 0.531751471153079 +2015-08-18T00:12:00Z 0.5272506080912802 +2015-08-18T00:24:00Z 0.5164030725416209 +``` + +The query returns the logarithm base 4 of [average](#mean) `water_level`s that are calculated at 12-minute intervals. + +To get those results, InfluxDB first calculates the average `water_level`s at 12-minute intervals. +This step is the same as using the `MEAN()` function with the `GROUP BY time()` clause and without `LOG()`: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time mean +---- ---- +2015-08-18T00:00:00Z 2.09 +2015-08-18T00:12:00Z 2.077 +2015-08-18T00:24:00Z 2.0460000000000003 +``` + +InfluxDB then calculates the logarithm base 4 of those averages. + +### LOG2() + +Returns the logarithm of the field value to the base 2. + +#### Basic syntax + +``` +SELECT LOG2( [ * | ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`LOG2(field_key)` +Returns the logarithm of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) to the base 2. + + + +`LOG2(*)` +Returns the logarithm of field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) to the base 2. + +`LOG2()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced syntax](#advanced-syntax) section for how to use `LOG2()` with a `GROUP BY time()` clause. + +##### Examples + +The examples below use the following subsample of the [`NOAA_water_database` data](/enterprise_influxdb/v1.9/query_language/data_download/): + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +2015-08-18T00:18:00Z 2.126 +2015-08-18T00:24:00Z 2.041 +2015-08-18T00:30:00Z 2.051 +``` + +###### Calculate the logarithm base 2 of field values associated with a field key + +```sql +> SELECT LOG2("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time log2 +---- ---- +2015-08-18T00:00:00Z 1.045442970761167 +2015-08-18T00:06:00Z 1.081339627451939 +2015-08-18T00:12:00Z 1.0200576523412537 +2015-08-18T00:18:00Z 1.0881415968690176 +2015-08-18T00:24:00Z 1.0292761823706322 +2015-08-18T00:30:00Z 1.0363274918177652 +``` + +The query returns the logarithm base 2 of field values in the `water_level` field key in the `h2o_feet` measurement. + +###### Calculate the logarithm base 2 of field values associated with each field key in a measurement + +```sql +> SELECT LOG2(*) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time log2_water_level +---- ---------------- +2015-08-18T00:00:00Z 1.045442970761167 +2015-08-18T00:06:00Z 1.081339627451939 +2015-08-18T00:12:00Z 1.0200576523412537 +2015-08-18T00:18:00Z 1.0881415968690176 +2015-08-18T00:24:00Z 1.0292761823706322 +2015-08-18T00:30:00Z 1.0363274918177652 +``` + +The query returns the logarithm base 2 of field values for each field key that stores numerical values in the `h2o_feet` measurement. +The `h2o_feet` measurement has one numerical field: `water_level`. + + + +###### Calculate the logarithm base 2 of field values associated with a field key and include several clauses + +```sql +> SELECT LOG2("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' ORDER BY time DESC LIMIT 4 OFFSET 2 + +name: h2o_feet +time log2 +---- ---- +2015-08-18T00:18:00Z 1.0881415968690176 +2015-08-18T00:12:00Z 1.0200576523412537 +2015-08-18T00:06:00Z 1.081339627451939 +2015-08-18T00:00:00Z 1.045442970761167 +``` + +The query returns the logarithm base 2 of field values associated with the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-18T00:00:00Z` and `2015-08-18T00:30:00Z` and returns results in [descending timestamp order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to four and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by two points. + +#### Advanced syntax + +```sql +SELECT LOG2(( [ * | ] )) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `LOG2()` function to those results. + +`LOG2()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples + +###### Calculate the logarithm base 2 of mean values + +```sql +> SELECT LOG2(MEAN("water_level")) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time log2 +---- ---- +2015-08-18T00:00:00Z 1.063502942306158 +2015-08-18T00:12:00Z 1.0545012161825604 +2015-08-18T00:24:00Z 1.0328061450832418 +``` + +The query returns the logarithm base 2 of [average](#mean) `water_level`s that are calculated at 12-minute intervals. + +To get those results, InfluxDB first calculates the average `water_level`s at 12-minute intervals. +This step is the same as using the `MEAN()` function with the `GROUP BY time()` clause and without `LOG2()`: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time mean +---- ---- +2015-08-18T00:00:00Z 2.09 +2015-08-18T00:12:00Z 2.077 +2015-08-18T00:24:00Z 2.0460000000000003 +``` + +InfluxDB then calculates the logarithm base 2 of those averages. + +### LOG10() + +Returns the logarithm of the field value to the base 10. + +#### Basic syntax + +``` +SELECT LOG10( [ * | ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`LOG10(field_key)` +Returns the logarithm of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) to the base 10. + + + +`LOG10(*)` +Returns the logarithm of field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) to the base 10. + +`LOG10()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `LOG10()` with a `GROUP BY time()` clause. + +##### Examples + +The examples below use the following subsample of the [`NOAA_water_database` data](/enterprise_influxdb/v1.9/query_language/data_download/): + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +2015-08-18T00:18:00Z 2.126 +2015-08-18T00:24:00Z 2.041 +2015-08-18T00:30:00Z 2.051 +``` + +###### Calculate the logarithm base 10 of field values associated with a field key + +```sql +> SELECT LOG10("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time log10 +---- ----- +2015-08-18T00:00:00Z 0.3147096929551737 +2015-08-18T00:06:00Z 0.32551566336314813 +2015-08-18T00:12:00Z 0.3070679506612984 +2015-08-18T00:18:00Z 0.32756326018727794 +2015-08-18T00:24:00Z 0.3098430047160705 +2015-08-18T00:30:00Z 0.3119656603683663 +``` + +The query returns the logarithm base 10 of field values in the `water_level` field key in the `h2o_feet` measurement. + +###### Calculate the logarithm base 10 of field values associated with each field key in a measurement + +```sql +> SELECT LOG10(*) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time log10_water_level +---- ----------------- +2015-08-18T00:00:00Z 0.3147096929551737 +2015-08-18T00:06:00Z 0.32551566336314813 +2015-08-18T00:12:00Z 0.3070679506612984 +2015-08-18T00:18:00Z 0.32756326018727794 +2015-08-18T00:24:00Z 0.3098430047160705 +2015-08-18T00:30:00Z 0.3119656603683663 +``` + +The query returns the logarithm base 10 of field values for each field key that stores numerical values in the `h2o_feet` measurement. +The `h2o_feet` measurement has one numerical field: `water_level`. + + + +###### Calculate the logarithm base 10 of field values associated with a field key and include several clauses + +```sql +> SELECT LOG10("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' ORDER BY time DESC LIMIT 4 OFFSET 2 + +name: h2o_feet +time log10 +---- ----- +2015-08-18T00:18:00Z 0.32756326018727794 +2015-08-18T00:12:00Z 0.3070679506612984 +2015-08-18T00:06:00Z 0.32551566336314813 +2015-08-18T00:00:00Z 0.3147096929551737 +``` + +The query returns the logarithm base 10 of field values associated with the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-18T00:00:00Z` and `2015-08-18T00:30:00Z` and returns results in [descending timestamp order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to four and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by two points. + +#### Advanced syntax + +``` +SELECT LOG10(( [ * | ] )) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `LOG10()` function to those results. + +`LOG10()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples + +###### Calculate the logarithm base 10 of mean values + +```sql +> SELECT LOG10(MEAN("water_level")) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time log10 +---- ----- +2015-08-18T00:00:00Z 0.32014628611105395 +2015-08-18T00:12:00Z 0.3174364965350991 +2015-08-18T00:24:00Z 0.3109056293761414 +``` + +The query returns the logarithm base 10 of [average](#mean) `water_level`s that are calculated at 12-minute intervals. + +To get those results, InfluxDB first calculates the average `water_level`s at 12-minute intervals. +This step is the same as using the `MEAN()` function with the `GROUP BY time()` clause and without `LOG10()`: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time mean +---- ---- +2015-08-18T00:00:00Z 2.09 +2015-08-18T00:12:00Z 2.077 +2015-08-18T00:24:00Z 2.0460000000000003 +``` + +InfluxDB then calculates the logarithm base 10 of those averages. + +### MOVING_AVERAGE() + +Returns the rolling average across a window of subsequent [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). + +#### Basic syntax + +``` +SELECT MOVING_AVERAGE( [ * | | // ] , ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`MOVING_AVERAGE()` calculates the rolling average across a window of `N` subsequent field values. +The `N` argument is an integer and it is required. + +`MOVING_AVERAGE(field_key,N)` +Returns the rolling average across `N` field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`MOVING_AVERAGE(/regular_expression/,N)` +Returns the rolling average across `N` field values associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +`MOVING_AVERAGE(*,N)` +Returns the rolling average across `N` field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`MOVING_AVERAGE()` int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax-3) section for how to use `MOVING_AVERAGE()` with a `GROUP BY time()` clause. + +##### Examples + +The examples below use the following subsample of the [`NOAA_water_database` data](/enterprise_influxdb/v1.9/query_language/data_download/): + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +2015-08-18T00:18:00Z 2.126 +2015-08-18T00:24:00Z 2.041 +2015-08-18T00:30:00Z 2.051 +``` + +###### Calculate the moving average of the field values associated with a field key + +```sql +> SELECT MOVING_AVERAGE("water_level",2) FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' + +name: h2o_feet +time moving_average +---- -------------- +2015-08-18T00:06:00Z 2.09 +2015-08-18T00:12:00Z 2.072 +2015-08-18T00:18:00Z 2.077 +2015-08-18T00:24:00Z 2.0835 +2015-08-18T00:30:00Z 2.0460000000000003 +``` + +The query returns the rolling average across a two-field-value window for the `water_level` field key and the `h2o_feet` measurement. +The first result (`2.09`) is the average of the first two points in the raw data: (`2.064 + 2.116) / 2`). +The second result (`2.072`) is the average of the second two points in the raw data: (`2.116 + 2.028) / 2`). + +###### Calculate the moving average of the field values associated with each field key in a measurement + +```sql +> SELECT MOVING_AVERAGE(*,3) FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' + +name: h2o_feet +time moving_average_water_level +---- -------------------------- +2015-08-18T00:12:00Z 2.0693333333333332 +2015-08-18T00:18:00Z 2.09 +2015-08-18T00:24:00Z 2.065 +2015-08-18T00:30:00Z 2.0726666666666667 +``` + +The query returns the rolling average across a three-field-value window for each field key that stores numerical values in the `h2o_feet` measurement. +The `h2o_feet` measurement has one numerical field: `water_level`. + +###### Calculate the moving average of the field values associated with each field key that matches a regular expression + +```sql +> SELECT MOVING_AVERAGE(/level/,4) FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' + +name: h2o_feet +time moving_average_water_level +---- -------------------------- +2015-08-18T00:18:00Z 2.0835 +2015-08-18T00:24:00Z 2.07775 +2015-08-18T00:30:00Z 2.0615 +``` + +The query returns the rolling average across a four-field-value window for each field key that stores numerical values and includes the word `level` in the `h2o_feet` measurement. + +###### Calculate the moving average of the field values associated with a field key and include several clauses + +```sql +> SELECT MOVING_AVERAGE("water_level",2) FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' ORDER BY time DESC LIMIT 2 OFFSET 3 + +name: h2o_feet +time moving_average +---- -------------- +2015-08-18T00:06:00Z 2.072 +2015-08-18T00:00:00Z 2.09 +``` + +The query returns the rolling average across a two-field-value window for the `water_level` field key in the `h2o_feet` measurement. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-18T00:00:00Z` and `2015-08-18T00:30:00Z` and returns results in [descending timestamp order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to two and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by three points. + +#### Advanced syntax + +``` +SELECT MOVING_AVERAGE( ([ * | | // ]) , N ) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `MOVING_AVERAGE()` function to those results. + +`MOVING_AVERAGE()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples + +###### Calculate the moving average of maximum values + +```sql +> SELECT MOVING_AVERAGE(MAX("water_level"),2) FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' GROUP BY time(12m) + +name: h2o_feet +time moving_average +---- -------------- +2015-08-18T00:12:00Z 2.121 +2015-08-18T00:24:00Z 2.0885 +``` + +The query returns the rolling average across a two-value window of [maximum](#max) `water_level`s that are calculated at 12-minute intervals. + +To get those results, InfluxDB first calculates the maximum `water_level`s at 12-minute intervals. +This step is the same as using the `MAX()` function with the `GROUP BY time()` clause and without `MOVING_AVERAGE()`: + +```sql +> SELECT MAX("water_level") FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' GROUP BY time(12m) + +name: h2o_feet +time max +---- --- +2015-08-18T00:00:00Z 2.116 +2015-08-18T00:12:00Z 2.126 +2015-08-18T00:24:00Z 2.051 +``` + +Next, InfluxDB calculates the rolling average across a two-value window using those maximum values. +The first final result (`2.121`) is the average of the first two maximum values (`(2.116 + 2.126) / 2`). + +### NON_NEGATIVE_DERIVATIVE() + +Returns the non-negative rate of change between subsequent [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). +Non-negative rates of change include positive rates of change and rates of change that equal zero. + +#### Basic syntax + +``` +SELECT NON_NEGATIVE_DERIVATIVE( [ * | | // ] [ , ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +InfluxDB calculates the difference between subsequent field values and converts those results into the rate of change per `unit`. +The `unit` argument is an integer followed by a [duration literal](/enterprise_influxdb/v1.9/query_language/spec/#literals) and it is optional. +If the query does not specify the `unit`, the unit defaults to one second (`1s`). +`NON_NEGATIVE_DERIVATIVE()` returns only positive rates of change or rates of change that equal zero. + +`NON_NEGATIVE_DERIVATIVE(field_key)` +Returns the non-negative rate of change between subsequent field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`NON_NEGATIVE_DERIVATIVE(/regular_expression/)` +Returns the non-negative rate of change between subsequent field values associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +`NON_NEGATIVE_DERIVATIVE(*)` +Returns the non-negative rate of change between subsequent field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`NON_NEGATIVE_DERIVATIVE()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax-4) section for how to use `NON_NEGATIVE_DERIVATIVE()` with a `GROUP BY time()` clause. + +##### Examples + +See the examples in the [`DERIVATIVE()` documentation](#basic-syntax-8). +`NON_NEGATIVE_DERIVATIVE()` behaves the same as the `DERIVATIVE()` function but `NON_NEGATIVE_DERIVATIVE()` returns only positive rates of change or rates of change that equal zero. + +#### Advanced syntax + +``` +SELECT NON_NEGATIVE_DERIVATIVE( ([ * | | // ]) [ , ] ) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `NON_NEGATIVE_DERIVATIVE()` function to those results. + +The `unit` argument is an integer followed by a [duration literal](/enterprise_influxdb/v1.9/query_language/spec/#literals) and it is optional. +If the query does not specify the `unit`, the `unit` defaults to the `GROUP BY time()` interval. +Note that this behavior is different from the [basic syntax's](#basic-syntax-4) default behavior. +`NON_NEGATIVE_DERIVATIVE()` returns only positive rates of change or rates of change that equal zero. + +`NON_NEGATIVE_DERIVATIVE()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples + +See the examples in the [`DERIVATIVE()` documentation](#advanced-syntax-8). +`NON_NEGATIVE_DERIVATIVE()` behaves the same as the `DERIVATIVE()` function but `NON_NEGATIVE_DERIVATIVE()` returns only positive rates of change or rates of change that equal zero. + +### NON_NEGATIVE_DIFFERENCE() + +Returns the non-negative result of subtraction between subsequent [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value). +Non-negative results of subtraction include positive differences and differences that equal zero. + +#### Basic syntax + +``` +SELECT NON_NEGATIVE_DIFFERENCE( [ * | | // ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`NON_NEGATIVE_DIFFERENCE(field_key)` +Returns the non-negative difference between subsequent field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +`NON_NEGATIVE_DIFFERENCE(/regular_expression/)` +Returns the non-negative difference between subsequent field values associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions). + +`NON_NEGATIVE_DIFFERENCE(*)` +Returns the non-negative difference between subsequent field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`NON_NEGATIVE_DIFFERENCE()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax-5) section for how to use `NON_NEGATIVE_DIFFERENCE()` with a `GROUP BY time()` clause. + +##### Examples + +See the examples in the [`DIFFERENCE()` documentation](#basic-syntax-9). +`NON_NEGATIVE_DIFFERENCE()` behaves the same as the `DIFFERENCE()` function but `NON_NEGATIVE_DIFFERENCE()` returns only positive differences or differences that equal zero. + +#### Advanced syntax + +``` +SELECT NON_NEGATIVE_DIFFERENCE(( [ * | | // ] )) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `NON_NEGATIVE_DIFFERENCE()` function to those results. + +`NON_NEGATIVE_DIFFERENCE()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples + +See the examples in the [`DIFFERENCE()` documentation](#advanced-syntax-9). +`NON_NEGATIVE_DIFFERENCE()` behaves the same as the `DIFFERENCE()` function but `NON_NEGATIVE_DIFFERENCE()` returns only positive differences or differences that equal zero. + +### POW() + +Returns the field value to the power of `x`. + +#### Basic syntax + +``` +SELECT POW( [ * | ], ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`POW(field_key, x)` +Returns the field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) to the power of `x`. + + + +`POW(*, x)` +Returns the field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) to the power of `x`. + +`POW()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `POW()` with a `GROUP BY time()` clause. + +##### Examples + +The examples below use the following subsample of the [`NOAA_water_database` data](/enterprise_influxdb/v1.9/query_language/data_download/): + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +2015-08-18T00:18:00Z 2.126 +2015-08-18T00:24:00Z 2.041 +2015-08-18T00:30:00Z 2.051 +``` + +###### Calculate field values associated with a field key to the power of 4 + +```sql +> SELECT POW("water_level", 4) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time pow +---- --- +2015-08-18T00:00:00Z 18.148417929216 +2015-08-18T00:06:00Z 20.047612231936 +2015-08-18T00:12:00Z 16.914992230656004 +2015-08-18T00:18:00Z 20.429279055375993 +2015-08-18T00:24:00Z 17.352898193760993 +2015-08-18T00:30:00Z 17.69549197320101 +``` + +The query returns field values in the `water_level` field key in the `h2o_feet` measurement multiplied to a power of 4. + +###### Calculate field values associated with each field key in a measurement to the power of 4 + +```sql +> SELECT POW(*, 4) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time pow_water_level +---- --------------- +2015-08-18T00:00:00Z 18.148417929216 +2015-08-18T00:06:00Z 20.047612231936 +2015-08-18T00:12:00Z 16.914992230656004 +2015-08-18T00:18:00Z 20.429279055375993 +2015-08-18T00:24:00Z 17.352898193760993 +2015-08-18T00:30:00Z 17.69549197320101 +``` + +The query returns field values for each field key that stores numerical values in the `h2o_feet` measurement multiplied to the power of 4. +The `h2o_feet` measurement has one numerical field: `water_level`. + + + +###### Calculate field values associated with a field key to the power of 4 and include several clauses + +```sql +> SELECT POW("water_level", 4) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' ORDER BY time DESC LIMIT 4 OFFSET 2 + +name: h2o_feet +time pow +---- --- +2015-08-18T00:18:00Z 20.429279055375993 +2015-08-18T00:12:00Z 16.914992230656004 +2015-08-18T00:06:00Z 20.047612231936 +2015-08-18T00:00:00Z 18.148417929216 +``` + +The query returns field values associated with the `water_level` field key multiplied to the power of 4. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-18T00:00:00Z` and `2015-08-18T00:30:00Z` and returns results in [descending timestamp order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to four and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by two points. + +#### Advanced syntax + +``` +SELECT POW(( [ * | ] ), ) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `POW()` function to those results. + +`POW()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples + +###### Calculate mean values to the power of 4 + +```sql +> SELECT POW(MEAN("water_level"), 4) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time pow +---- --- +2015-08-18T00:00:00Z 19.08029760999999 +2015-08-18T00:12:00Z 18.609983417041 +2015-08-18T00:24:00Z 17.523567165456008 +``` + +The query returns [average](#mean) `water_level`s that are calculated at 12-minute intervals multiplied to the power of 4. + +To get those results, InfluxDB first calculates the average `water_level`s at 12-minute intervals. +This step is the same as using the `MEAN()` function with the `GROUP BY time()` clause and without `POW()`: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time mean +---- ---- +2015-08-18T00:00:00Z 2.09 +2015-08-18T00:12:00Z 2.077 +2015-08-18T00:24:00Z 2.0460000000000003 +``` + +InfluxDB then calculates those averages multiplied to the power of 4. + +### ROUND() + +Returns the subsequent value rounded to the nearest integer. + +#### Basic syntax + +``` +SELECT ROUND( [ * | ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`ROUND(field_key)` +Returns the field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) rounded to the nearest integer. + + + +`ROUND(*)` +Returns the field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) rounded to the nearest integer. + +`ROUND()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `ROUND()` with a `GROUP BY time()` clause. + +##### Examples + +The examples below use the following subsample of the [`NOAA_water_database` data](/enterprise_influxdb/v1.9/query_language/data_download/): + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +2015-08-18T00:18:00Z 2.126 +2015-08-18T00:24:00Z 2.041 +2015-08-18T00:30:00Z 2.051 +``` + +###### Round field values associated with a field key + +```sql +> SELECT ROUND("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time round +---- ----- +2015-08-18T00:00:00Z 2 +2015-08-18T00:06:00Z 2 +2015-08-18T00:12:00Z 2 +2015-08-18T00:18:00Z 2 +2015-08-18T00:24:00Z 2 +2015-08-18T00:30:00Z 2 +``` + +The query returns field values in the `water_level` field key in the `h2o_feet` measurement rounded to the nearest integer. + +###### Round field values associated with each field key in a measurement + +```sql +> SELECT ROUND(*) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time round_water_level +---- ----------------- +2015-08-18T00:00:00Z 2 +2015-08-18T00:06:00Z 2 +2015-08-18T00:12:00Z 2 +2015-08-18T00:18:00Z 2 +2015-08-18T00:24:00Z 2 +2015-08-18T00:30:00Z 2 +``` + +The query returns field values for each field key that stores numerical values in the `h2o_feet` measurement rounded to the nearest integer. +The `h2o_feet` measurement has one numerical field: `water_level`. + + + +###### Round field values associated with a field key and include several clauses + +```sql +> SELECT ROUND("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' ORDER BY time DESC LIMIT 4 OFFSET 2 + +name: h2o_feet +time round +---- ----- +2015-08-18T00:18:00Z 2 +2015-08-18T00:12:00Z 2 +2015-08-18T00:06:00Z 2 +2015-08-18T00:00:00Z 2 +``` + +The query returns field values associated with the `water_level` field key rounded to the nearest integer. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-18T00:00:00Z` and `2015-08-18T00:30:00Z` and returns results in [descending timestamp order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to four and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by two points. + +#### Advanced syntax + +``` +SELECT ROUND(( [ * | ] )) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `ROUND()` function to those results. + +`ROUND()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples + +###### Calculate mean values rounded to the nearest integer + +```sql +> SELECT ROUND(MEAN("water_level")) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time round +---- ----- +2015-08-18T00:00:00Z 2 +2015-08-18T00:12:00Z 2 +2015-08-18T00:24:00Z 2 +``` + +The query returns the [average](#mean) `water_level`s that are calculated at 12-minute intervals and rounds to the nearest integer. + +To get those results, InfluxDB first calculates the average `water_level`s at 12-minute intervals. +This step is the same as using the `MEAN()` function with the `GROUP BY time()` clause and without `ROUND()`: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time mean +---- ---- +2015-08-18T00:00:00Z 2.09 +2015-08-18T00:12:00Z 2.077 +2015-08-18T00:24:00Z 2.0460000000000003 +``` + +InfluxDB then rounds those averages to the nearest integer. + +### SIN() + +Returns the sine of the field value. + +#### Basic syntax + +``` +SELECT SIN( [ * | ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`SIN(field_key)` +Returns the sine of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + + + +`SIN(*)` +Returns the sine of field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`SIN()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `SIN()` with a `GROUP BY time()` clause. + +##### Examples + +The examples below use the following subsample of the [`NOAA_water_database` data](/enterprise_influxdb/v1.9/query_language/data_download/): + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +2015-08-18T00:18:00Z 2.126 +2015-08-18T00:24:00Z 2.041 +2015-08-18T00:30:00Z 2.051 +``` + +###### Calculate the sine of field values associated with a field key + +```sql +> SELECT SIN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time sin +---- --- +2015-08-18T00:00:00Z 0.8808206017241819 +2015-08-18T00:06:00Z 0.8550216851706579 +2015-08-18T00:12:00Z 0.8972904165810275 +2015-08-18T00:18:00Z 0.8497930984115993 +2015-08-18T00:24:00Z 0.8914760289023131 +2015-08-18T00:30:00Z 0.8869008523376968 +``` + +The query returns sine of field values in the `water_level` field key in the `h2o_feet` measurement. + +###### Calculate the sine of field values associated with each field key in a measurement + +```sql +> SELECT SIN(*) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time sin_water_level +---- --------------- +2015-08-18T00:00:00Z 0.8808206017241819 +2015-08-18T00:06:00Z 0.8550216851706579 +2015-08-18T00:12:00Z 0.8972904165810275 +2015-08-18T00:18:00Z 0.8497930984115993 +2015-08-18T00:24:00Z 0.8914760289023131 +2015-08-18T00:30:00Z 0.8869008523376968 +``` + +The query returns sine of field values for each field key that stores numerical values in the `h2o_feet` measurement. +The `h2o_feet` measurement has one numerical field: `water_level`. + + + +###### Calculate the sine of field values associated with a field key and include several clauses + +```sql +> SELECT SIN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' ORDER BY time DESC LIMIT 4 OFFSET 2 + +name: h2o_feet +time sin +---- --- +2015-08-18T00:18:00Z 0.8497930984115993 +2015-08-18T00:12:00Z 0.8972904165810275 +2015-08-18T00:06:00Z 0.8550216851706579 +2015-08-18T00:00:00Z 0.8808206017241819 +``` + +The query returns sine of field values associated with the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-18T00:00:00Z` and `2015-08-18T00:30:00Z` and returns results in [descending timestamp order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to four and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by two points. + +#### Advanced syntax + +``` +SELECT SIN(( [ * | ] )) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `SIN()` function to those results. + +`SIN()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples + +###### Calculate the sine of mean values + +```sql +> SELECT SIN(MEAN("water_level")) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time sin +---- --- +2015-08-18T00:00:00Z 0.8682145834456126 +2015-08-18T00:12:00Z 0.8745914945253902 +2015-08-18T00:24:00Z 0.8891995555912935 +``` + +The query returns the sine of [average](#mean) `water_level`s that are calculated at 12-minute intervals. + +To get those results, InfluxDB first calculates the average `water_level`s at 12-minute intervals. +This step is the same as using the `MEAN()` function with the `GROUP BY time()` clause and without `SIN()`: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time mean +---- ---- +2015-08-18T00:00:00Z 2.09 +2015-08-18T00:12:00Z 2.077 +2015-08-18T00:24:00Z 2.0460000000000003 +``` + +InfluxDB then calculates sine of those averages. + +### SQRT() + +Returns the square root of field value. + +#### Basic syntax + +``` +SELECT SQRT( [ * | ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`SQRT(field_key)` +Returns the square root of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + + + +`SQRT(*)` +Returns the square root field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`SQRT()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `SQRT()` with a `GROUP BY time()` clause. + +##### Examples + +The examples below use the following subsample of the [`NOAA_water_database` data](/enterprise_influxdb/v1.9/query_language/data_download/): + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +2015-08-18T00:18:00Z 2.126 +2015-08-18T00:24:00Z 2.041 +2015-08-18T00:30:00Z 2.051 +``` + +###### Calculate the square root of field values associated with a field key + +```sql +> SELECT SQRT("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time sqrt +---- ---- +2015-08-18T00:00:00Z 1.4366627996854378 +2015-08-18T00:06:00Z 1.4546477236774544 +2015-08-18T00:12:00Z 1.4240786495134319 +2015-08-18T00:18:00Z 1.4580809305384939 +2015-08-18T00:24:00Z 1.4286357128393508 +2015-08-18T00:30:00Z 1.4321312788986909 +``` + +The query returns the square roots of field values in the `water_level` field key in the `h2o_feet` measurement. + +###### Calculate the square root of field values associated with each field key in a measurement + +```sql +> SELECT SQRT(*) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time sqrt_water_level +---- ---------------- +2015-08-18T00:00:00Z 1.4366627996854378 +2015-08-18T00:06:00Z 1.4546477236774544 +2015-08-18T00:12:00Z 1.4240786495134319 +2015-08-18T00:18:00Z 1.4580809305384939 +2015-08-18T00:24:00Z 1.4286357128393508 +2015-08-18T00:30:00Z 1.4321312788986909 +``` + +The query returns the square roots of field values for each field key that stores numerical values in the `h2o_feet` measurement. +The `h2o_feet` measurement has one numerical field: `water_level`. + + + +###### Calculate the square root of field values associated with a field key and include several clauses + +```sql +> SELECT SQRT("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' ORDER BY time DESC LIMIT 4 OFFSET 2 + +name: h2o_feet +time sqrt +---- ---- +2015-08-18T00:18:00Z 1.4580809305384939 +2015-08-18T00:12:00Z 1.4240786495134319 +2015-08-18T00:06:00Z 1.4546477236774544 +2015-08-18T00:00:00Z 1.4366627996854378 +``` + +The query returns the square roots of field values associated with the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-18T00:00:00Z` and `2015-08-18T00:30:00Z` and returns results in [descending timestamp order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to four and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by two points. + +#### Advanced syntax + +``` +SELECT SQRT(( [ * | ] )) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `SQRT()` function to those results. + +`SQRT()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples + +###### Calculate the square root of mean values + +```sql +> SELECT SQRT(MEAN("water_level")) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time sqrt +---- ---- +2015-08-18T00:00:00Z 1.445683229480096 +2015-08-18T00:12:00Z 1.4411800720243115 +2015-08-18T00:24:00Z 1.430384563675098 +``` + +The query returns the square roots of [average](#mean) `water_level`s that are calculated at 12-minute intervals. + +To get those results, InfluxDB first calculates the average `water_level`s at 12-minute intervals. +This step is the same as using the `MEAN()` function with the `GROUP BY time()` clause and without `SQRT()`: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time mean +---- ---- +2015-08-18T00:00:00Z 2.09 +2015-08-18T00:12:00Z 2.077 +2015-08-18T00:24:00Z 2.0460000000000003 +``` + +InfluxDB then calculates the square roots of those averages. + +### TAN() + +Returns the tangent of the field value. + +#### Basic syntax + +``` +SELECT TAN( [ * | ] ) [INTO_clause] FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`TAN(field_key)` +Returns the tangent of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + + + +`TAN(*)` +Returns the tangent of field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement). + +`TAN()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `TAN()` with a `GROUP BY time()` clause. + +##### Examples + +The examples below use the following subsample of the [`NOAA_water_database` data](/enterprise_influxdb/v1.9/query_language/data_download/): + +```sql +> SELECT "water_level" FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time water_level +---- ----------- +2015-08-18T00:00:00Z 2.064 +2015-08-18T00:06:00Z 2.116 +2015-08-18T00:12:00Z 2.028 +2015-08-18T00:18:00Z 2.126 +2015-08-18T00:24:00Z 2.041 +2015-08-18T00:30:00Z 2.051 +``` + +###### Calculate the tangent of field values associated with a field key + +```sql +> SELECT TAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time tan +---- --- +2015-08-18T00:00:00Z -1.8604293534384375 +2015-08-18T00:06:00Z -1.6487359603347427 +2015-08-18T00:12:00Z -2.0326408012302273 +2015-08-18T00:18:00Z -1.6121545688343464 +2015-08-18T00:24:00Z -1.9676434782626282 +2015-08-18T00:30:00Z -1.9198657720074992 +``` + +The query returns tangent of field values in the `water_level` field key in the `h2o_feet` measurement. + +###### Calculate the tangent of field values associated with each field key in a measurement + +```sql +> SELECT TAN(*) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' + +name: h2o_feet +time tan_water_level +---- --------------- +2015-08-18T00:00:00Z -1.8604293534384375 +2015-08-18T00:06:00Z -1.6487359603347427 +2015-08-18T00:12:00Z -2.0326408012302273 +2015-08-18T00:18:00Z -1.6121545688343464 +2015-08-18T00:24:00Z -1.9676434782626282 +2015-08-18T00:30:00Z -1.9198657720074992 +``` + +The query returns tangent of field values for each field key that stores numerical values in the `h2o_feet` measurement. +The `h2o_feet` measurement has one numerical field: `water_level`. + + + +###### Calculate the tangent of field values associated with a field key and include several clauses + +```sql +> SELECT TAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' ORDER BY time DESC LIMIT 4 OFFSET 2 + +name: h2o_feet +time tan +---- --- +2015-08-18T00:18:00Z -1.6121545688343464 +2015-08-18T00:12:00Z -2.0326408012302273 +2015-08-18T00:06:00Z -1.6487359603347427 +2015-08-18T00:00:00Z -1.8604293534384375 +``` + +The query returns tangent of field values associated with the `water_level` field key. +It covers the [time range](/enterprise_influxdb/v1.9/query_language/explore-data/#time-syntax) between `2015-08-18T00:00:00Z` and `2015-08-18T00:30:00Z` and returns results in [descending timestamp order](/enterprise_influxdb/v1.9/query_language/explore-data/#order-by-time-desc). +The query also [limits](/enterprise_influxdb/v1.9/query_language/explore-data/#the-limit-and-slimit-clauses) the number of points returned to four and [offsets](/enterprise_influxdb/v1.9/query_language/explore-data/#the-offset-and-soffset-clauses) results by two points. + +#### Advanced syntax + +``` +SELECT TAN(( [ * | ] )) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +The advanced syntax requires a [`GROUP BY time() ` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) and a nested InfluxQL function. +The query first calculates the results for the nested function at the specified `GROUP BY time()` interval and then applies the `TAN()` function to those results. + +`TAN()` supports the following nested functions: +[`COUNT()`](#count), +[`MEAN()`](#mean), +[`MEDIAN()`](#median), +[`MODE()`](#mode), +[`SUM()`](#sum), +[`FIRST()`](#first), +[`LAST()`](#last), +[`MIN()`](#min), +[`MAX()`](#max), and +[`PERCENTILE()`](#percentile). + +##### Examples + +###### Calculate the tangent of mean values + +```sql +> SELECT TAN(MEAN("water_level")) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time tan +---- --- +2015-08-18T00:00:00Z -1.7497661902817365 +2015-08-18T00:12:00Z -1.8038002062256624 +2015-08-18T00:24:00Z -1.9435224805850773 +``` + +The query returns tangent of [average](#mean) `water_level`s that are calculated at 12-minute intervals. + +To get those results, InfluxDB first calculates the average `water_level`s at 12-minute intervals. +This step is the same as using the `MEAN()` function with the `GROUP BY time()` clause and without `TAN()`: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(12m) + +name: h2o_feet +time mean +---- ---- +2015-08-18T00:00:00Z 2.09 +2015-08-18T00:12:00Z 2.077 +2015-08-18T00:24:00Z 2.0460000000000003 +``` + +InfluxDB then calculates tangent of those averages. + +## Predictors + +### HOLT_WINTERS() + +Returns N number of predicted [field values](/enterprise_influxdb/v1.9/concepts/glossary/#field-value) using the +[Holt-Winters](https://www.otexts.org/fpp/7/5) seasonal method. + +Use `HOLT_WINTERS()` to: + +* Predict when data values will cross a given threshold +* Compare predicted values with actual values to detect anomalies in your data + +#### Syntax + +``` +SELECT HOLT_WINTERS[_WITH-FIT]((),,) [INTO_clause] FROM_clause [WHERE_clause] GROUP_BY_clause [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] +``` + +`HOLT_WINTERS(function(field_key),N,S)` returns `N` seasonally adjusted +predicted field values for the specified [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key). + +The `N` predicted values occur at the same interval as the [`GROUP BY time()` interval](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +If your `GROUP BY time()` interval is `6m` and `N` is `3` you'll +receive three predicted values that are each six minutes apart. + +`S` is the seasonal pattern parameter and delimits the length of a seasonal +pattern according to the `GROUP BY time()` interval. +If your `GROUP BY time()` interval is `2m` and `S` is `3`, then the +seasonal pattern occurs every six minutes, that is, every three data points. +If you do not want to seasonally adjust your predicted values, set `S` to `0` +or `1.` + +`HOLT_WINTERS_WITH_FIT(function(field_key),N,S)` returns the fitted values in +addition to `N` seasonally adjusted predicted field values for the specified field key. + +`HOLT_WINTERS()` and `HOLT_WINTERS_WITH_FIT()` work with data that occur at +consistent time intervals; the nested InfluxQL function and the +`GROUP BY time()` clause ensure that the Holt-Winters functions operate on regular data. + +`HOLT_WINTERS()` and `HOLT_WINTERS_WITH_FIT()` support int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +#### Examples + +##### Predict field values associated with a field key + +###### Raw Data + +The example uses [Chronograf](https://github.com/influxdata/chronograf) to visualize the data. +The example focuses on the following subsample of the [`NOAA_water_database` data](/enterprise_influxdb/v1.9/query_language/data_download/): + +```sql +SELECT "water_level" FROM "NOAA_water_database"."autogen"."h2o_feet" WHERE "location"='santa_monica' AND time >= '2015-08-22 22:12:00' AND time <= '2015-08-28 03:00:00' +``` + +![Raw Data](/img/influxdb/1-3-hw-raw-data-1-2.png) + +###### Step 1: Match the Trends of the Raw Data + +Write a `GROUP BY time()` query that matches the general trends of the raw `water_level` data. +Here, we use the [`FIRST()`](#first) function: + +```sql +SELECT FIRST("water_level") FROM "NOAA_water_database"."autogen"."h2o_feet" WHERE "location"='santa_monica' and time >= '2015-08-22 22:12:00' and time <= '2015-08-28 03:00:00' GROUP BY time(379m,348m) +``` + +In the `GROUP BY time()` clause, the first argument (`379m`) matches +the length of time that occurs between each peak and trough in the `water_level` data. +The second argument (`348m`) is the +[offset interval](/enterprise_influxdb/v1.9/query_language/explore-data/#advanced-group-by-time-syntax). +The offset interval alters the default `GROUP BY time()` boundaries to +match the time range of the raw data. + +The blue line shows the results of the query: + +![First step](/img/influxdb/1-3-hw-first-step-1-2.png) + +###### Step 2: Determine the Seasonal Pattern + +Identify the seasonal pattern in the data using the information from the +query in step 1. + +Focusing on the blue line in the graph below, the pattern in the `water_level` data repeats about every 25 hours and 15 minutes. +There are four data points per season, so `4` is the seasonal pattern argument. + +![Second step](/img/influxdb/1-3-hw-second-step-1-2.png) + +###### Step 3: Apply the HOLT_WINTERS() function + +Add the Holt-Winters function to the query. +Here, we use `HOLT_WINTERS_WITH_FIT()` to view both the fitted values and the predicted values: + +```sql +SELECT HOLT_WINTERS_WITH_FIT(FIRST("water_level"),10,4) FROM "NOAA_water_database"."autogen"."h2o_feet" WHERE "location"='santa_monica' AND time >= '2015-08-22 22:12:00' AND time <= '2015-08-28 03:00:00' GROUP BY time(379m,348m) +``` + +In the `HOLT_WINTERS_WITH_FIT()` function, the first argument (`10`) requests 10 predicted field values. +Each predicted point is `379m` apart, the same interval as the first argument in the `GROUP BY time()` clause. +The second argument in the `HOLT_WINTERS_WITH_FIT()` function (`4`) is the seasonal pattern that we determined in the previous step. + +The blue line shows the results of the query: + +![Third step](/img/influxdb/1-3-hw-third-step-1-2.png) + +#### Common Issues with `HOLT_WINTERS()` + +##### `HOLT_WINTERS()` and receiving fewer than `N` points + +In some cases, users may receive fewer predicted points than +requested by the `N` parameter. +That behavior occurs when the math becomes unstable and cannot forecast more +points. +It implies that either `HOLT_WINTERS()` is not suited for the dataset or that +the seasonal adjustment parameter is invalid and is confusing the algorithm. + +## Technical Analysis + +The following technical analysis functions apply widely used algorithms to your data. +While they are primarily used in the world of finance and investing, they have +application in other industries and use cases as well. + +[CHANDE_MOMENTUM_OSCILLATOR()](#chande-momentum-oscillator) +[EXPONENTIAL_MOVING_AVERAGE()](#exponential-moving-average) +[DOUBLE_EXPONENTIAL_MOVING_AVERAGE()](#double-exponential-moving-average) +[KAUFMANS_EFFICIENCY_RATIO()](#kaufmans-efficiency-ratio) +[KAUFMANS_ADAPTIVE_MOVING_AVERAGE()](#kaufmans-adaptive-moving-average) +[TRIPLE_EXPONENTIAL_MOVING_AVERAGE()](#triple-exponential-moving-average) +[TRIPLE_EXPONENTIAL_DERIVATIVE()](#triple-exponential-derivative) +[RELATIVE_STRENGTH_INDEX()](#relative-strength-index) + +### Arguments + +Along with a [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key), +technical analysis function accept the following arguments: + +#### `PERIOD` + +**Required, integer, min=1** + +The sample size of the algorithm. +This is essentially the number of historical samples which have any significant +effect on the output of the algorithm. +E.G. `2` means the current point and the point before it. +The algorithm uses an exponential decay rate to determine the weight of a historical point, +generally known as the alpha (α). The `PERIOD` controls the decay rate. + +> NOTE: Older points can still have an impact. + +#### `HOLD_PERIOD` + +**integer, min=-1** + +How many samples the algorithm needs before it will start emitting results. +The default of `-1` means the value is based on the algorithm, the `PERIOD`, +and the `WARMUP_TYPE`, but is a value in which the algorithm can emit meaningful results. + +_**Default Hold Periods:**_ + +For most of the available technical analysis, the default `HOLD_PERIOD` is +determined by which technical analysis algorithm you're using and the [`WARMUP_TYPE`](#warmup-type) + +| Algorithm \ Warmup Type | simple | exponential | none | +| --------------------------------- | ---------------------- | ----------- |:----------: | +| [EXPONENTIAL_MOVING_AVERAGE](#exponential-moving-average) | PERIOD - 1 | PERIOD - 1 | n/a | +| [DOUBLE_EXPONENTIAL_MOVING_AVERAGE](#double-exponential-moving-average) | ( PERIOD - 1 ) * 2 | PERIOD - 1 | n/a | +| [TRIPLE_EXPONENTIAL_MOVING_AVERAGE](#triple-exponential-moving-average) | ( PERIOD - 1 ) * 3 | PERIOD - 1 | n/a | +| [TRIPLE_EXPONENTIAL_DERIVATIVE](#triple-exponential-derivative) | ( PERIOD - 1 ) * 3 + 1 | PERIOD | n/a | +| [RELATIVE_STRENGTH_INDEX](#relative-strength-index) | PERIOD | PERIOD | n/a | +| [CHANDE_MOMENTUM_OSCILLATOR](#chande-momentum-oscillator) | PERIOD | PERIOD | PERIOD - 1 | + +_**Kaufman Algorithm Default Hold Periods:**_ + +| Algorithm | Default Hold Period | +| --------- | ------------------- | +| [KAUFMANS_EFFICIENCY_RATIO()](#kaufmans-efficiency-ratio) | PERIOD | +| [KAUFMANS_ADAPTIVE_MOVING_AVERAGE()](#kaufmans-adaptive-moving-average) | PERIOD | + +#### `WARMUP_TYPE` + +**default='exponential'** + +This controls how the algorithm initializes itself for the first `PERIOD` samples. +It is essentially the duration for which it has an incomplete sample set. + +`simple` +Simple moving average (SMA) of the first `PERIOD` samples. +This is the method used by [ta-lib](https://www.ta-lib.org/). + +`exponential` +Exponential moving average (EMA) with scaling alpha (α). +This basically uses an EMA with `PERIOD=1` for the first point, `PERIOD=2` +for the second point, etc., until algorithm has consumed `PERIOD` number of points. +As the algorithm immediately starts using an EMA, when this method is used and +`HOLD_PERIOD` is unspecified or `-1`, the algorithm may start emitting points +after a much smaller sample size than with `simple`. + +`none` +The algorithm does not perform any smoothing at all. +This is the method used by [ta-lib](https://www.ta-lib.org/). +When this method is used and `HOLD_PERIOD` is unspecified, `HOLD_PERIOD` +defaults to `PERIOD - 1`. + +> The `none` warmup type is only available with the +> [`CHANDE_MOMENTUM_OSCILLATOR()`](#chande-momentum-oscillator) function. + +### CHANDE_MOMENTUM_OSCILLATOR() + +The Chande Momentum Oscillator (CMO) is a technical momentum indicator developed by Tushar Chande. +The CMO indicator is created by calculating the difference between the sum of all +recent higher data points and the sum of all recent lower data points, +then dividing the result by the sum of all data movement over a given time period. +The result is multiplied by 100 to give the -100 to +100 range. +Source + +#### Basic syntax + +``` +CHANDE_MOMENTUM_OSCILLATOR([ * | | /regular_expression/ ], [, , [warmup_type]]) +``` + +**Available Arguments:** + +[period](#period) +[hold_period](#hold-period) (Optional) +[warmup_type](#warmup-type) (Optional) + +`CHANDE_MOMENTUM_OSCILLATOR(field_key, 2)` +Returns the field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Chande Momentum Oscillator algorithm with a 2-value period +and the default hold period and warmup type. + +`CHANDE_MOMENTUM_OSCILLATOR(field_key, 10, 9, 'none')` +Returns the field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Chande Momentum Oscillator algorithm with a 10-value period +a 9-value hold period, and the `none` warmup type. + +`CHANDE_MOMENTUM_OSCILLATOR(MEAN(), 2) ... GROUP BY time(1d)` +Returns the mean of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Chande Momentum Oscillator algorithm with a 2-value period +and the default hold period and warmup type. + +> **Note:** When aggregating data with a `GROUP BY` clause, you must include an +> [aggregate function](#aggregations) in your call to the `CHANDE_MOMENTUM_OSCILLATOR()` function. + +`CHANDE_MOMENTUM_OSCILLATOR(/regular_expression/, 2)` +Returns the field values associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions) +processed using the Chande Momentum Oscillator algorithm with a 2-value period +and the default hold period and warmup type. + +`CHANDE_MOMENTUM_OSCILLATOR(*, 2)` +Returns the field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) +processed using the Chande Momentum Oscillator algorithm with a 2-value period +and the default hold period and warmup type. + +`CHANDE_MOMENTUM_OSCILLATOR()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `CHANDE_MOMENTUM_OSCILLATOR()` with a `GROUP BY time()` clause. + +### EXPONENTIAL_MOVING_AVERAGE() + +An exponential moving average (EMA) is a type of moving average that is similar +to a [simple moving average](#moving-average), except that more weight is given to the latest data. +It's also known as the "exponentially weighted moving average." +This type of moving average reacts faster to recent data changes than a simple moving average. + +Source + +#### Basic syntax + +``` +EXPONENTIAL_MOVING_AVERAGE([ * | | /regular_expression/ ], [, (Optional) +[warmup_type](#warmup-type) (Optional) + +`EXPONENTIAL_MOVING_AVERAGE(field_key, 2)` +Returns the field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Exponential Moving Average algorithm with a 2-value period +and the default hold period and warmup type. + +`EXPONENTIAL_MOVING_AVERAGE(field_key, 10, 9, 'exponential')` +Returns the field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Exponential Moving Average algorithm with a 10-value period +a 9-value hold period, and the `exponential` warmup type. + +`EXPONENTIAL_MOVING_AVERAGE(MEAN(), 2) ... GROUP BY time(1d)` +Returns the mean of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Exponential Moving Average algorithm with a 2-value period +and the default hold period and warmup type. + +> **Note:** When aggregating data with a `GROUP BY` clause, you must include an +> [aggregate function](#aggregations) in your call to the `EXPONENTIAL_MOVING_AVERAGE()` function. + +`EXPONENTIAL_MOVING_AVERAGE(/regular_expression/, 2)` +Returns the field values associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions) +processed using the Exponential Moving Average algorithm with a 2-value period +and the default hold period and warmup type. + +`EXPONENTIAL_MOVING_AVERAGE(*, 2)` +Returns the field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) +processed using the Exponential Moving Average algorithm with a 2-value period +and the default hold period and warmup type. + +`EXPONENTIAL_MOVING_AVERAGE()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `EXPONENTIAL_MOVING_AVERAGE()` with a `GROUP BY time()` clause. + +### DOUBLE_EXPONENTIAL_MOVING_AVERAGE() + +The Double Exponential Moving Average (DEMA) attempts to remove the inherent lag +associated to Moving Averages by placing more weight on recent values. +The name suggests this is achieved by applying a double exponential smoothing which is not the case. +The name double comes from the fact that the value of an [EMA](#exponential-moving-average) is doubled. +To keep it in line with the actual data and to remove the lag, the value "EMA of EMA" +is subtracted from the previously doubled EMA. + +Source + +#### Basic syntax + +``` +DOUBLE_EXPONENTIAL_MOVING_AVERAGE([ * | | /regular_expression/ ], [, (Optional) +[warmup_type](#warmup-type) (Optional) + +`DOUBLE_EXPONENTIAL_MOVING_AVERAGE(field_key, 2)` +Returns the field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Double Exponential Moving Average algorithm with a 2-value period +and the default hold period and warmup type. + +`DOUBLE_EXPONENTIAL_MOVING_AVERAGE(field_key, 10, 9, 'exponential')` +Returns the field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Double Exponential Moving Average algorithm with a 10-value period +a 9-value hold period, and the `exponential` warmup type. + +`DOUBLE_EXPONENTIAL_MOVING_AVERAGE(MEAN(), 2) ... GROUP BY time(1d)` +Returns the mean of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Double Exponential Moving Average algorithm with a 2-value period +and the default hold period and warmup type. + +> **Note:** When aggregating data with a `GROUP BY` clause, you must include an +> [aggregate function](#aggregations) in your call to the `DOUBLE_EXPONENTIAL_MOVING_AVERAGE()` function. + +`DOUBLE_EXPONENTIAL_MOVING_AVERAGE(/regular_expression/, 2)` +Returns the field values associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions) +processed using the Double Exponential Moving Average algorithm with a 2-value period +and the default hold period and warmup type. + +`DOUBLE_EXPONENTIAL_MOVING_AVERAGE(*, 2)` +Returns the field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) +processed using the Double Exponential Moving Average algorithm with a 2-value period +and the default hold period and warmup type. + +`DOUBLE_EXPONENTIAL_MOVING_AVERAGE()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `DOUBLE_EXPONENTIAL_MOVING_AVERAGE()` with a `GROUP BY time()` clause. + +### KAUFMANS_EFFICIENCY_RATIO() + +Kaufman's Efficiency Ration, or simply "Efficiency Ratio" (ER), is calculated by +dividing the data change over a period by the absolute sum of the data movements +that occurred to achieve that change. +The resulting ratio ranges between 0 and 1 with higher values representing a +more efficient or trending market. + +The ER is very similar to the [Chande Momentum Oscillator](#chande-momentum-oscillator) (CMO). +The difference is that the CMO takes market direction into account, but if you take the absolute CMO and divide by 100, you you get the Efficiency Ratio. + +Source + +#### Basic syntax + +``` +KAUFMANS_EFFICIENCY_RATIO([ * | | /regular_expression/ ], [, ]) +``` + +**Available Arguments:** + +[period](#period) +[hold_period](#hold-period) (Optional) + +`KAUFMANS_EFFICIENCY_RATIO(field_key, 2)` +Returns the field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Efficiency Index algorithm with a 2-value period +and the default hold period and warmup type. + +`KAUFMANS_EFFICIENCY_RATIO(field_key, 10, 10)` +Returns the field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Efficiency Index algorithm with a 10-value period and +a 10-value hold period. + +`KAUFMANS_EFFICIENCY_RATIO(MEAN(), 2) ... GROUP BY time(1d)` +Returns the mean of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Efficiency Index algorithm with a 2-value period +and the default hold period. + +> **Note:** When aggregating data with a `GROUP BY` clause, you must include an +> [aggregate function](#aggregations) in your call to the `KAUFMANS_EFFICIENCY_RATIO()` function. + +`KAUFMANS_EFFICIENCY_RATIO(/regular_expression/, 2)` +Returns the field values associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions) +processed using the Efficiency Index algorithm with a 2-value period +and the default hold period and warmup type. + +`KAUFMANS_EFFICIENCY_RATIO(*, 2)` +Returns the field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) +processed using the Efficiency Index algorithm with a 2-value period +and the default hold period and warmup type. + +`KAUFMANS_EFFICIENCY_RATIO()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `KAUFMANS_EFFICIENCY_RATIO()` with a `GROUP BY time()` clause. + +### KAUFMANS_ADAPTIVE_MOVING_AVERAGE() + +Kaufman's Adaptive Moving Average (KAMA) is a moving average designed to +account for sample noise or volatility. +KAMA will closely follow data points when the data swings are relatively small and noise is low. +KAMA will adjust when the data swings widen and follow data from a greater distance. +This trend-following indicator can be used to identify the overall trend, +time turning points and filter data movements. + +Source + +#### Basic syntax + +``` +KAUFMANS_ADAPTIVE_MOVING_AVERAGE([ * | | /regular_expression/ ], [, ]) +``` + +**Available Arguments:** +[period](#period) +[hold_period](#hold-period) (Optional) + +`KAUFMANS_ADAPTIVE_MOVING_AVERAGE(field_key, 2)` +Returns the field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Kaufman Adaptive Moving Average algorithm with a 2-value period +and the default hold period and warmup type. + +`KAUFMANS_ADAPTIVE_MOVING_AVERAGE(field_key, 10, 10)` +Returns the field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Kaufman Adaptive Moving Average algorithm with a 10-value period +and a 10-value hold period. + +`KAUFMANS_ADAPTIVE_MOVING_AVERAGE(MEAN(), 2) ... GROUP BY time(1d)` +Returns the mean of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Kaufman Adaptive Moving Average algorithm with a 2-value period +and the default hold period. + +> **Note:** When aggregating data with a `GROUP BY` clause, you must include an +> [aggregate function](#aggregations) in your call to the `KAUFMANS_ADAPTIVE_MOVING_AVERAGE()` function. + +`KAUFMANS_ADAPTIVE_MOVING_AVERAGE(/regular_expression/, 2)` +Returns the field values associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions) +processed using the Kaufman Adaptive Moving Average algorithm with a 2-value period +and the default hold period and warmup type. + +`KAUFMANS_ADAPTIVE_MOVING_AVERAGE(*, 2)` +Returns the field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) +processed using the Kaufman Adaptive Moving Average algorithm with a 2-value period +and the default hold period and warmup type. + +`KAUFMANS_ADAPTIVE_MOVING_AVERAGE()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `KAUFMANS_ADAPTIVE_MOVING_AVERAGE()` with a `GROUP BY time()` clause. + +### TRIPLE_EXPONENTIAL_MOVING_AVERAGE() + +The triple exponential moving average (TEMA) was developed to filter out +volatility from conventional moving averages. +While the name implies that it's a triple exponential smoothing, it's actually a +composite of a [single exponential moving average](#exponential-moving-average), +a [double exponential moving average](#double-exponential-moving-average), +and a triple exponential moving average. + +Source + +#### Basic syntax + +``` +TRIPLE_EXPONENTIAL_MOVING_AVERAGE([ * | | /regular_expression/ ], [, (Optional) +[warmup_type](#warmup-type) (Optional) + +`TRIPLE_EXPONENTIAL_MOVING_AVERAGE(field_key, 2)` +Returns the field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Triple Exponential Moving Average algorithm with a 2-value period +and the default hold period and warmup type. + +`TRIPLE_EXPONENTIAL_MOVING_AVERAGE(field_key, 10, 9, 'exponential')` +Returns the field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Triple Exponential Moving Average algorithm with a 10-value period +a 9-value hold period, and the `exponential` warmup type. + +`TRIPLE_EXPONENTIAL_MOVING_AVERAGE(MEAN(), 2) ... GROUP BY time(1d)` +Returns the mean of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Triple Exponential Moving Average algorithm with a 2-value period +and the default hold period and warmup type. + +> **Note:** When aggregating data with a `GROUP BY` clause, you must include an +> [aggregate function](#aggregations) in your call to the `TRIPLE_EXPONENTIAL_MOVING_AVERAGE()` function. + +`TRIPLE_EXPONENTIAL_MOVING_AVERAGE(/regular_expression/, 2)` +Returns the field values associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions) +processed using the Triple Exponential Moving Average algorithm with a 2-value period +and the default hold period and warmup type. + +`TRIPLE_EXPONENTIAL_MOVING_AVERAGE(*, 2)` +Returns the field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) +processed using the Triple Exponential Moving Average algorithm with a 2-value period +and the default hold period and warmup type. + +`TRIPLE_EXPONENTIAL_MOVING_AVERAGE()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `TRIPLE_EXPONENTIAL_MOVING_AVERAGE()` with a `GROUP BY time()` clause. + +### TRIPLE_EXPONENTIAL_DERIVATIVE() + +The triple exponential derivative indicator, commonly referred to as "TRIX," is +an oscillator used to identify oversold and overbought markets, and can also be +used as a momentum indicator. +TRIX calculates a [triple exponential moving average](#triple-exponential-moving-average) +of the [log](#log) of the data input over the period of time. +The previous value is subtracted from the previous value. +This prevents cycles that are shorter than the defined period from being considered by the indicator. + +Like many oscillators, TRIX oscillates around a zero line. When used as an oscillator, +a positive value indicates an overbought market while a negative value indicates an oversold market. +When used as a momentum indicator, a positive value suggests momentum is increasing +while a negative value suggests momentum is decreasing. +Many analysts believe that when the TRIX crosses above the zero line it gives a +buy signal, and when it closes below the zero line, it gives a sell signal. + +Source + +#### Basic syntax + +``` +TRIPLE_EXPONENTIAL_DERIVATIVE([ * | | /regular_expression/ ], [, (Optional) +[warmup_type](#warmup-type) (Optional) + +`TRIPLE_EXPONENTIAL_DERIVATIVE(field_key, 2)` +Returns the field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Triple Exponential Derivative algorithm with a 2-value period +and the default hold period and warmup type. + +`TRIPLE_EXPONENTIAL_DERIVATIVE(field_key, 10, 10, 'exponential')` +Returns the field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Triple Exponential Derivative algorithm with a 10-value period, +a 10-value hold period, and the `exponential` warmup type. + +`TRIPLE_EXPONENTIAL_DERIVATIVE(MEAN(), 2) ... GROUP BY time(1d)` +Returns the mean of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Triple Exponential Derivative algorithm with a 2-value period +and the default hold period and warmup type. + +> **Note:** When aggregating data with a `GROUP BY` clause, you must include an +> [aggregate function](#aggregations) in your call to the `TRIPLE_EXPONENTIAL_DERIVATIVE()` function. + +`TRIPLE_EXPONENTIAL_DERIVATIVE(/regular_expression/, 2)` +Returns the field values associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions) +processed using the Triple Exponential Derivative algorithm with a 2-value period +and the default hold period and warmup type. + +`TRIPLE_EXPONENTIAL_DERIVATIVE(*, 2)` +Returns the field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) +processed using the Triple Exponential Derivative algorithm with a 2-value period +and the default hold period and warmup type. + +`TRIPLE_EXPONENTIAL_DERIVATIVE()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `TRIPLE_EXPONENTIAL_DERIVATIVE()` with a `GROUP BY time()` clause. + +### RELATIVE_STRENGTH_INDEX() + +The relative strength index (RSI) is a momentum indicator that compares the magnitude of recent increases and decreases over a specified time period to measure speed and change of data movements. + +Source + +#### Basic syntax + +``` +RELATIVE_STRENGTH_INDEX([ * | | /regular_expression/ ], [, (Optional) +[warmup_type](#warmup-type) (Optional) + +`RELATIVE_STRENGTH_INDEX(field_key, 2)` +Returns the field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Relative Strength Index algorithm with a 2-value period +and the default hold period and warmup type. + +`RELATIVE_STRENGTH_INDEX(field_key, 10, 10, 'exponential')` +Returns the field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Relative Strength Index algorithm with a 10-value period, +a 10-value hold period, and the `exponential` warmup type. + +`RELATIVE_STRENGTH_INDEX(MEAN(), 2) ... GROUP BY time(1d)` +Returns the mean of field values associated with the [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) +processed using the Relative Strength Index algorithm with a 2-value period +and the default hold period and warmup type. + +> **Note:** When aggregating data with a `GROUP BY` clause, you must include an +> [aggregate function](#aggregations) in your call to the `RELATIVE_STRENGTH_INDEX()` function. + +`RELATIVE_STRENGTH_INDEX(/regular_expression/, 2)` +Returns the field values associated with each field key that matches the [regular expression](/enterprise_influxdb/v1.9/query_language/explore-data/#regular-expressions) +processed using the Relative Strength Index algorithm with a 2-value period +and the default hold period and warmup type. + +`RELATIVE_STRENGTH_INDEX(*, 2)` +Returns the field values associated with each field key in the [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement) +processed using the Relative Strength Index algorithm with a 2-value period +and the default hold period and warmup type. + +`RELATIVE_STRENGTH_INDEX()` supports int64 and float64 field value [data types](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types). + +The basic syntax supports `GROUP BY` clauses that [group by tags](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-tags) but not `GROUP BY` clauses that [group by time](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals). +See the [Advanced Syntax](#advanced-syntax) section for how to use `RELATIVE_STRENGTH_INDEX()` with a `GROUP BY time()` clause. + +## Other + +### Sample Data + +The data used in this document are available for download on the [Sample Data](/enterprise_influxdb/v1.9/query_language/data_download/) page. + +### General Syntax for Functions + +#### Specify Multiple Functions in the SELECT Clause + +##### Syntax + +``` +SELECT (),() FROM_clause [...] +``` + +Separate multiple functions in one `SELECT` statement with a comma (`,`). +The syntax applies to all InfluxQL functions except [`TOP()`](#top) and [`BOTTOM()`](#bottom). +The `SELECT` clause does not support specifying `TOP()` or `BOTTOM()` with another function. + +##### Examples + +###### Calculate the mean and median field values in one query + +```sql +> SELECT MEAN("water_level"),MEDIAN("water_level") FROM "h2o_feet" + +name: h2o_feet +time mean median +---- ---- ------ +1970-01-01T00:00:00Z 4.442107025822522 4.124 +``` + +The query returns the [average](#mean) and [median](#median) field values in the `water_level` field key. + +###### Calculate the mode of two fields in one query + +```sql +> SELECT MODE("water_level"),MODE("level description") FROM "h2o_feet" + +name: h2o_feet +time mode mode_1 +---- ---- ------ +1970-01-01T00:00:00Z 2.69 between 3 and 6 feet +``` + +The query returns the [mode](#mode) field values for the `water_level` field key and for the `level description` field key. +The `water_level` mode is in the `mode` column and the `level description` mode is in the `mode_1` column. +The system can't return more than one column with the same name so it renames the second `mode` column to `mode_1`. + +See [Rename the Output Field Key](#rename-the-output-field-key) for how to configure the output column headers. + +###### Calculate the minimum and maximum field values in one query + +```sql +> SELECT MIN("water_level"), MAX("water_level") [...] + +name: h2o_feet +time min max +---- --- --- +1970-01-01T00:00:00Z -0.61 9.964 +``` + +The query returns the [minimum](#min) and [maximum](#max) field values in the `water_level` field key. + +Notice that the query returns `1970-01-01T00:00:00Z`, the InfluxDB equivalent to a null timestamp, as the timestamp value. +`MIN()` and `MAX()` are [selector](#selectors) functions; when a selector function is the only function in the `SELECT` clause, it returns a specific timestamp. +Because `MIN()` and `MAX()` return two different timestamps (see below), the system overrides those timestamps with the null timestamp equivalent. + +```sql +> SELECT MIN("water_level") FROM "h2o_feet" + +name: h2o_feet +time min +---- --- +2015-08-29T14:30:00Z -0.61 <--- Timestamp 1 + +> SELECT MAX("water_level") FROM "h2o_feet" + +name: h2o_feet +time max +---- --- +2015-08-29T07:24:00Z 9.964 <--- Timestamp 2 +``` + +#### Rename the Output Field Key + +##### Syntax + +``` +SELECT () AS [...] +``` + +By default, functions return results under a field key that matches the function name. +Include an `AS` clause to specify the name of the output field key. + +##### Examples + +###### Specify the output field key + +```sql +> SELECT MEAN("water_level") AS "dream_name" FROM "h2o_feet" + +name: h2o_feet +time dream_name +---- ---------- +1970-01-01T00:00:00Z 4.442107025822522 +``` + +The query returns the [average](#mean) field value of the `water_level` field key and renames the output field key to `dream_name`. +Without the `AS` clause, the query returns `mean` as the output field key: + +```sql +> SELECT MEAN("water_level") FROM "h2o_feet" + +name: h2o_feet +time mean +---- ---- +1970-01-01T00:00:00Z 4.442107025822522 +``` + +###### Specify the output field key for multiple functions + +```sql +> SELECT MEDIAN("water_level") AS "med_wat",MODE("water_level") AS "mode_wat" FROM "h2o_feet" + +name: h2o_feet +time med_wat mode_wat +---- ------- -------- +1970-01-01T00:00:00Z 4.124 2.69 +``` + +The query returns the [median](#median) and [mode](#mode) field values for the `water_level` field key and renames the output field keys to `med_wat` and `mode_wat`. +Without the `AS` clauses, the query returns `median` and `mode` as the output field keys: + +```sql +> SELECT MEDIAN("water_level"),MODE("water_level") FROM "h2o_feet" + +name: h2o_feet +time median mode +---- ------ ---- +1970-01-01T00:00:00Z 4.124 2.69 +``` + +#### Change the Values Reported for Intervals with no Data + +By default, queries with an InfluxQL function and a [`GROUP BY time()` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) report null values for intervals with no data. +Include `fill()` at the end of the `GROUP BY` clause to change that value. +See [Data Exploration](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals-and-fill) for a complete discussion of `fill()`. + +### Common Issues with Functions + +The following sections describe frequent sources of confusion with all functions, aggregation functions, and selector functions. +See the function-specific documentation for common issues with individual functions: + +* [DISTINCT()](#common-issues-with-distinct) +* [BOTTOM()](#common-issues-with-bottom) +* [PERCENTILE()](#common-issues-with-percentile) +* [SAMPLE()](#common-issues-with-sample) +* [TOP()](#common-issues-with-top) +* [ELAPSED()](#common-issues-with-elapsed) +* [HOLT_WINTERS()](#common-issues-with-holt-winters) + +#### All Functions + +##### Nesting functions + +Some InfluxQL functions support nesting in the [`SELECT` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#select-clause): + +* [`COUNT()`](#count) with [`DISTINCT()`](#distinct) +* [`CUMULATIVE_SUM()`](#cumulative-sum) +* [`DERIVATIVE()`](#derivative) +* [`DIFFERENCE()`](#difference) +* [`ELAPSED()`](#elapsed) +* [`MOVING_AVERAGE()`](#moving-average) +* [`NON_NEGATIVE_DERIVATIVE()`](#non-negative-derivative) +* [`HOLT_WINTERS()`](#holt-winters) and [`HOLT_WINTERS_WITH_FIT()`](#holt-winters) + +For other functions, use InfluxQL's [subqueries](/enterprise_influxdb/v1.9/query_language/explore-data/#subqueries) to nest functions in the [`FROM` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#from-clause). +See the [Data Exploration](/enterprise_influxdb/v1.9/query_language/explore-data/#subqueries) page more on using subqueries. + +##### Querying time ranges after now() + +Most `SELECT` statements have a default time range between [`1677-09-21 00:12:43.145224194` and `2262-04-11T23:47:16.854775806Z` UTC](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#what-are-the-minimum-and-maximum-timestamps-that-influxdb-can-store). +For `SELECT` statements with an InfluxQL function and a [`GROUP BY time()` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals), the default time +range is between `1677-09-21 00:12:43.145224194` UTC and [`now()`](/enterprise_influxdb/v1.9/concepts/glossary/#now). + +To query data with timestamps that occur after `now()`, `SELECT` statements with +an InfluxQL function and a `GROUP BY time()` clause must provide an alternative upper bound in the +[`WHERE` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#the-where-clause). +See the [Frequently Asked Questions](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#why-don-t-my-group-by-time-queries-return-timestamps-that-occur-after-now) page for an example. + +#### Aggregation Functions + +##### Understanding the returned timestamp + +A query with an [aggregation function](#aggregations) and no time range in the [`WHERE` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#the-where-clause) returns epoch 0 (`1970-01-01T00:00:00Z`) as the timestamp. +InfluxDB uses epoch 0 as the null timestamp equivalent. +A query with an aggregate function that includes a time range in the `WHERE` clause returns the lower time bound as the timestamp. + +##### Examples + +###### Use an aggregate function without a specified time range + +```sql +> SELECT SUM("water_level") FROM "h2o_feet" + +name: h2o_feet +time sum +---- --- +1970-01-01T00:00:00Z 67777.66900000004 +``` + +The query returns the InfluxDB equivalent of a null timestamp (epoch 0: `1970-01-01T00:00:00Z`) as the timestamp. +[`SUM()`](#sum) aggregates points across several timestamps and has no single timestamp to return. + +###### Use an aggregate function with a specified time range + +```sql +> SELECT SUM("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' + +name: h2o_feet +time sum +---- --- +2015-08-18T00:00:00Z 67777.66900000004 +``` + +The query returns the lower time bound (`WHERE time >= '2015-08-18T00:00:00Z'`) as the timestamp. + +###### Use an aggregate function with a specified time range and a GROUP BY time() clause + +```sql +> SELECT SUM("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:18:00Z' GROUP BY time(12m) + +name: h2o_feet +time sum +---- --- +2015-08-18T00:00:00Z 20.305 +2015-08-18T00:12:00Z 19.802999999999997 +``` + +The query returns the lower time bound for each [`GROUP BY time()`](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) interval as the timestamps. + +###### Mixing aggregation functions with non-aggregates + +Aggregation functions do not support specifying standalone [field keys](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) or [tag keys](/enterprise_influxdb/v1.9/concepts/glossary/#tag-key) in the [`SELECT` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#select-clause). +Aggregation functions return a single calculated value and there is no obvious single value to return for any unaggregated fields or tags. +Including a standalone field key or tag key with an aggregation function in the `SELECT` clause returns an error: + +```sql +> SELECT SUM("water_level"),"location" FROM "h2o_feet" + +ERR: error parsing query: mixing aggregate and non-aggregate queries is not supported +``` + +##### Getting slightly different results + +For some aggregation functions, executing the same function on the same set of [float64](/enterprise_influxdb/v1.9/write_protocols/line_protocol_reference/#data-types) points may yield slightly different results. +InfluxDB does not sort points before it applies the aggregation function; that behavior can cause small discrepancies in the query results. + +#### Selector Functions + +##### Understanding the returned timestamp + +The timestamps returned by [selector functions](#selectors) depend on the number of functions in the query and on the other clauses in the query: + +A query with a single selector function, a single [field key](/enterprise_influxdb/v1.9/concepts/glossary/#field-key) argument, and no [`GROUP BY time()` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) returns the timestamp for the point that appears in the raw data. +A query with a single selector function, multiple field key arguments, and no [`GROUP BY time()` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#group-by-time-intervals) returns the timestamp for the point that appears in the raw data or the InfluxDB equivalent of a null timestamp (epoch 0: `1970-01-01T00:00:00Z`). + +A query with more than one function and no time range in the [`WHERE` clause](/enterprise_influxdb/v1.9/query_language/explore-data/#the-where-clause) returns the InfluxDB equivalent of a null timestamp (epoch 0: `1970-01-01T00:00:00Z`). +A query with more than one function and a time range in the `WHERE` clause returns the lower time bound as the timestamp. + +A query with a selector function and a `GROUP BY time()` clause returns the lower time bound for each `GROUP BY time()` interval. +Note that the `SAMPLE()` function behaves differently from other selector functions when paired with the `GROUP BY time()` clause. +See [Common Issues with `SAMPLE()`](#common-issues-with-sample) for more information. + +##### Examples + +###### Use a single selector function with a single field key and without a specified time range + +```sql +> SELECT MAX("water_level") FROM "h2o_feet" + +name: h2o_feet +time max +---- --- +2015-08-29T07:24:00Z 9.964 + +> SELECT MAX("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' + +name: h2o_feet +time max +---- --- +2015-08-29T07:24:00Z 9.964 +``` + +The queries return the timestamp for the [maximum](#max) point that appears in the raw data. + +###### Use a single selector function with multiple field keys and without a specified time range + +```sql +> SELECT FIRST(*) FROM "h2o_feet" + +name: h2o_feet +time first_level description first_water_level +---- ----------------------- ----------------- +1970-01-01T00:00:00Z between 6 and 9 feet 8.12 + +> SELECT MAX(*) FROM "h2o_feet" + +name: h2o_feet +time max_water_level +---- --------------- +2015-08-29T07:24:00Z 9.964 +``` + +The first query returns the InfluxDB equivalent of a null timestamp (epoch 0: `1970-01-01T00:00:00Z`) as the timestamp. +`FIRST(*)` returns two timestamps (one for each field key in the `h2o_feet` [measurement](/enterprise_influxdb/v1.9/concepts/glossary/#measurement)) so the system overrides those timestamps with the null timestamp equivalent. + +The second query returns the timestamp for the maximum point that appears in the raw data. +`MAX(*)` returns one timestamp (the `h2o-feet` measurement has only one numerical field) so the system does not overwrite the original timestamp. + +###### Use a selector function with another function and without a specified time range + +```sql +> SELECT MAX("water_level"),MIN("water_level") FROM "h2o_feet" + +name: h2o_feet +time max min +---- --- --- +1970-01-01T00:00:00Z 9.964 -0.61 +``` + +The query returns the InfluxDB equivalent of a null timestamp (epoch 0: `1970-01-01T00:00:00Z`) as the timestamp. +The `MAX()` and [`MIN()`](#min) functions return different timestamps so the system has no single timestamp to return. + +###### Use a selector function with another function and with a specified time range + +```sql +> SELECT MAX("water_level"),MIN("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' + +name: h2o_feet +time max min +---- --- --- +2015-08-18T00:00:00Z 9.964 -0.61 +``` + +The query returns the lower time bound (`WHERE time >= '2015-08-18T00:00:00Z'`) as the timestamp. + +###### Use a selector function with a GROUP BY time() clause + +```sql +> SELECT MAX("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:18:00Z' GROUP BY time(12m) + +name: h2o_feet +time max +---- --- +2015-08-18T00:00:00Z 8.12 +2015-08-18T00:12:00Z 7.887 +``` + +The query returns the lower time bound for each `GROUP BY time()` interval as the timestamp. diff --git a/content/enterprise_influxdb/v1.9/query_language/manage-database.md b/content/enterprise_influxdb/v1.9/query_language/manage-database.md new file mode 100644 index 000000000..5736ac3b7 --- /dev/null +++ b/content/enterprise_influxdb/v1.9/query_language/manage-database.md @@ -0,0 +1,363 @@ +--- +title: Manage your database using InfluxQL +description: > + Use InfluxQL to administer your InfluxDB server and work with InfluxDB databases, retention policies, series, measurements, and shards. +menu: + enterprise_influxdb_1_9: + name: Manage your database + weight: 40 + parent: InfluxQL +aliases: + - /enterprise_influxdb/v1.9/query_language/database_management/ +--- + +InfluxQL offers a full suite of administrative commands. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
        Data Management:Retention Policy Management:
        CREATE DATABASECREATE RETENTION POLICY
        DROP DATABASEALTER RETENTION POLICY
        DROP SERIESDROP RETENTION POLICY
        DELETE
        DROP MEASUREMENT
        DROP SHARD
        + +If you're looking for `SHOW` queries (for example, `SHOW DATABASES` or `SHOW RETENTION POLICIES`), see [Schema Exploration](/enterprise_influxdb/v1.9/query_language/explore-schema). + +The examples in the sections below use the InfluxDB [Command Line Interface (CLI)](/enterprise_influxdb/v1.9/introduction/getting-started/). +You can also execute the commands using the InfluxDB API; simply send a `GET` request to the `/query` endpoint and include the command in the URL parameter `q`. +For more on using the InfluxDB API, see [Querying data](/enterprise_influxdb/v1.9/guides/querying_data/). + +> **Note:** When authentication is enabled, only admin users can execute most of the commands listed on this page. +> See the documentation on [authentication and authorization](/enterprise_influxdb/v1.9/administration/authentication_and_authorization/) for more information. + +## Data management + +### CREATE DATABASE + +Creates a new database. + +#### Syntax + +```sql +CREATE DATABASE [WITH [DURATION ] [REPLICATION ] [SHARD DURATION ] [NAME ]] +``` + +#### Description of syntax + +`CREATE DATABASE` requires a database [name](/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions/#what-words-and-characters-should-i-avoid-when-writing-data-to-influxdb). + +The `WITH`, `DURATION`, `REPLICATION`, `SHARD DURATION`, and `NAME` clauses are optional and create a single [retention policy](/enterprise_influxdb/v1.9/concepts/glossary/#retention-policy-rp) associated with the created database. +If you do not specify one of the clauses after `WITH`, the relevant behavior defaults to the `autogen` retention policy settings. +The created retention policy automatically serves as the database's default retention policy. +For more information about those clauses, see [Retention Policy Management](/enterprise_influxdb/v1.9/query_language/manage-database/#retention-policy-management). + +A successful `CREATE DATABASE` query returns an empty result. +If you attempt to create a database that already exists, InfluxDB does nothing and does not return an error. + +#### Examples + +##### Create a database + +``` +> CREATE DATABASE "NOAA_water_database" +> +``` + +The query creates a database called `NOAA_water_database`. +[By default](/enterprise_influxdb/v1.9/administration/config/#retention-autocreate-true), InfluxDB also creates the `autogen` retention policy and associates it with the `NOAA_water_database`. + +##### Create a database with a specific retention policy + +``` +> CREATE DATABASE "NOAA_water_database" WITH DURATION 3d REPLICATION 1 SHARD DURATION 1h NAME "liquid" +> +``` + +The query creates a database called `NOAA_water_database`. +It also creates a default retention policy for `NOAA_water_database` with a `DURATION` of three days, a [replication factor](/enterprise_influxdb/v1.9/concepts/glossary/#replication-factor) of one, a [shard group](/enterprise_influxdb/v1.9/concepts/glossary/#shard-group) duration of one hour, and with the name `liquid`. + +### Delete a database with DROP DATABASE + +The `DROP DATABASE` query deletes all of the data, measurements, series, continuous queries, and retention policies from the specified database. +The query takes the following form: +```sql +DROP DATABASE +``` + +Drop the database NOAA_water_database: +```bash +> DROP DATABASE "NOAA_water_database" +> +``` + +A successful `DROP DATABASE` query returns an empty result. +If you attempt to drop a database that does not exist, InfluxDB does not return an error. + +### Drop series from the index with DROP SERIES + +The `DROP SERIES` query deletes all points from a [series](/enterprise_influxdb/v1.9/concepts/glossary/#series) in a database, +and it drops the series from the index. + +The query takes the following form, where you must specify either the `FROM` clause or the `WHERE` clause: +```sql +DROP SERIES FROM WHERE ='' +``` + +Drop all series from a single measurement: +```sql +> DROP SERIES FROM "h2o_feet" +``` + +Drop series with a specific tag pair from a single measurement: +```sql +> DROP SERIES FROM "h2o_feet" WHERE "location" = 'santa_monica' +``` + +Drop all points in the series that have a specific tag pair from all measurements in the database: +```sql +> DROP SERIES WHERE "location" = 'santa_monica' +``` + +A successful `DROP SERIES` query returns an empty result. + +### Delete series with DELETE + +The `DELETE` query deletes all points from a +[series](/enterprise_influxdb/v1.9/concepts/glossary/#series) in a database. +Unlike +[`DROP SERIES`](/enterprise_influxdb/v1.9/query_language/manage-database/#drop-series-from-the-index-with-drop-series), `DELETE` does not drop the series from the index. + +You must include either the `FROM` clause, the `WHERE` clause, or both: + +``` +DELETE FROM WHERE [=''] | [