Go to file
Jason Stirnaman 9e276df194
Optimize writes: Use Telegraf to pre-process data before writing (#5282)
* chore(v3): preprocess writes (Telegraf, etc)

* feat: optimize writes using Telegraf

* WIP: dedup plugin

* WIP: Dedup

* chore(v3): Writes: Preprocess data using Telegraf: Dedup example

(#5155)
- Add example using processors.dedup in Telegraf.
- The plugin or configuration doesn't currently work as expected. Investigating...

* chore(v3): Write data: pre-process data using Telegraf

- Add Telegraf examples that pre-process data to avoid schema conflicts and reduce payload (bandwith, memory, etc.) when writing data.
- For Serverless. Will port to Dedicated and Clustered after approval.
- Doesn't include Dedup yet - I'm waiting on help to make it work (see branch https://github.com/influxdata/docs-v2/compare/issue5155-preprocess-with-dedup)

* Update content/influxdb/cloud-serverless/write-data/best-practices/optimize-writes.md

Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>

* chore(test): Test harness setup, use golang base image

- Use golang base image for ease of Go environment setup. Installing Python separately is easier than Go.
- Add flag to bypass python warning about using system python and pip.
- Remove redundant env.products from pytest.ini; we already load it into environment variables.

* chore(v3): Optimize writes: Use Telegraf

Closes Telegraf guide: how to use the Converter processor to coerce data before writing #5155

- Adds use cases to Dedicated and Clustered docs
- Adds example for Execd processor plugin to all
- Clean up and pass tests
- Remove shared code sample

* fix(v3): Optimize writes: Placeholders and indent fixes

* Update content/influxdb/cloud-dedicated/write-data/best-practices/optimize-writes.md

Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>

* Update content/influxdb/cloud-serverless/write-data/best-practices/optimize-writes.md

Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>

* Update content/influxdb/clustered/write-data/best-practices/optimize-writes.md

Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>

---------

Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
2023-12-22 11:23:49 -06:00
.ci Reorganize Serverless DBRP for v1 queries and writes, explain DBRP and auto bucket defaults (#5257) 2023-12-05 16:48:25 -06:00
.circleci Upgrade hugo to latest version 0.120.4 (#5241) 2023-11-27 13:39:23 -07:00
.github Update new feature issue template 2022-04-14 16:08:59 -06:00
.vscode chore(ci): Add vale script, configuration, and fixes (#5081) 2023-08-11 13:50:31 -05:00
api-docs feature(v3): Cloud Dedicated write API: partial writes write good lines and reject bad li… (#5273) 2023-12-18 12:43:44 -06:00
assets Update mentions of IOx to v3 (#5256) 2023-11-30 11:52:40 -07:00
content Optimize writes: Use Telegraf to pre-process data before writing (#5282) 2023-12-22 11:23:49 -06:00
data Release influxctl v2.4.2 2023-12-18 08:10:15 -07:00
deploy Upgrade hugo to latest version 0.120.4 (#5241) 2023-11-27 13:39:23 -07:00
flux-build-scripts fix: update flux version script to pull nightly versions from main-2.x 2023-12-04 10:32:21 -07:00
layouts fix: Prevent stable version block from appearing in platform and resources sections, closes #5262 (#5263) 2023-12-06 15:03:43 -07:00
resources Add files via upload (#4395) 2022-10-06 13:32:02 -06:00
shared/text Optimize writes: Use Telegraf to pre-process data before writing (#5282) 2023-12-22 11:23:49 -06:00
static chore(serverless-grafana): update "create a datasource" for InfluxQL 2023-08-29 17:18:45 -05:00
test Optimize writes: Use Telegraf to pre-process data before writing (#5282) 2023-12-22 11:23:49 -06:00
.gitignore Jts/api guides (#3862) 2022-06-15 19:15:46 -05:00
.prettierrc.yaml fix: Authorization, not Authentication, header (#3342) 2021-11-03 16:53:56 -05:00
.s3deploy.yml added font file extensions to s3deploy caching config 2019-01-19 08:30:43 -07:00
.vale.ini chore(ci): Vale: ignore words, ignore spelling and ellipses in table cells (#5066) 2023-08-01 15:00:28 -05:00
CONTRIBUTING.md InfluxDB Clustered documentation (#5126) 2023-09-06 06:21:47 -06:00
LICENSE added license and issue template, moved security.md 2019-10-07 10:27:36 -06:00
README.md Upgrade hugo to latest version 0.120.4 (#5241) 2023-11-27 13:39:23 -07:00
compose.yaml 5156 Writes: add v1 to Serverless get-started (#5233) 2023-11-15 14:41:29 -06:00
hugo.staging.toml Upgrade hugo to latest version 0.120.4 (#5241) 2023-11-27 13:39:23 -07:00
hugo.toml Upgrade hugo to latest version 0.120.4 (#5241) 2023-11-27 13:39:23 -07:00
package.json chore(deps): bump axios from 0.27.2 to 1.6.0 (#5227) 2023-11-13 09:15:41 -07:00
test.Dockerfile Optimize writes: Use Telegraf to pre-process data before writing (#5282) 2023-12-22 11:23:49 -06:00
test.sh 5156 Writes: add v1 to Serverless get-started (#5233) 2023-11-15 14:41:29 -06:00
yarn.lock Upgrade hugo to latest version 0.120.4 (#5241) 2023-11-27 13:39:23 -07:00

README.md

InfluxDB 2.0 Documentation

This repository contains the InfluxDB 2.x documentation published at docs.influxdata.com.

Contributing

We welcome and encourage community contributions. For information about contributing to the InfluxData documentation, see Contribution guidelines.

Reporting a Vulnerability

InfluxData takes security and our users' trust very seriously. If you believe you have found a security issue in any of our open source projects, please responsibly disclose it by contacting security@influxdata.com. More details about security vulnerability reporting, including our GPG key, can be found at https://www.influxdata.com/how-to-report-security-vulnerabilities/.

Running the docs locally

  1. Clone this repository to your local machine.

  2. Install NodeJS, Yarn, Hugo, & Asset Pipeline Tools

    The InfluxData documentation uses Hugo, a static site generator built in Go. The site uses Hugo's asset pipeline, which requires the extended version of Hugo along with NodeJS tools like PostCSS, to build and process stylesheets and JavaScript.

    To install the required dependencies and build the assets, do the following:

    1. Install NodeJS

    2. Install Yarn

    3. In your terminal, from the docs-v2 directory, install the dependencies:

      cd docs-v2
      yarn install
      

      Note: The most recent version of Hugo tested with this documentation is 0.120.4.

  3. To generate the API docs, see api-docs/README.md.

  4. Start the Hugo server

    Hugo provides a local development server that generates the HTML pages, builds the static assets, and serves them at localhost:1313.

    In your terminal, start the Hugo server:

    npx hugo server
    
  5. View the docs at localhost:1313.