* chore(v3): preprocess writes (Telegraf, etc)
* feat: optimize writes using Telegraf
* WIP: dedup plugin
* WIP: Dedup
* chore(v3): Writes: Preprocess data using Telegraf: Dedup example
(#5155)
- Add example using processors.dedup in Telegraf.
- The plugin or configuration doesn't currently work as expected. Investigating...
* chore(v3): Write data: pre-process data using Telegraf
- Add Telegraf examples that pre-process data to avoid schema conflicts and reduce payload (bandwith, memory, etc.) when writing data.
- For Serverless. Will port to Dedicated and Clustered after approval.
- Doesn't include Dedup yet - I'm waiting on help to make it work (see branch https://github.com/influxdata/docs-v2/compare/issue5155-preprocess-with-dedup)
* Update content/influxdb/cloud-serverless/write-data/best-practices/optimize-writes.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* chore(test): Test harness setup, use golang base image
- Use golang base image for ease of Go environment setup. Installing Python separately is easier than Go.
- Add flag to bypass python warning about using system python and pip.
- Remove redundant env.products from pytest.ini; we already load it into environment variables.
* chore(v3): Optimize writes: Use Telegraf
Closes Telegraf guide: how to use the Converter processor to coerce data before writing #5155
- Adds use cases to Dedicated and Clustered docs
- Adds example for Execd processor plugin to all
- Clean up and pass tests
- Remove shared code sample
* fix(v3): Optimize writes: Placeholders and indent fixes
* Update content/influxdb/cloud-dedicated/write-data/best-practices/optimize-writes.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Update content/influxdb/cloud-serverless/write-data/best-practices/optimize-writes.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Update content/influxdb/clustered/write-data/best-practices/optimize-writes.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
---------
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* chore(ci): Install Telegraf to test container
* chore(v3): test telegraf config and startup:
- creates the config file for the test run. TODO: remove the duplication and use the content for test setup.
- fix placeholder name
* fix(v3): make test pass for curl API example
* chore(ci): make tests pass for write. Add dependencies to Dockerfile.
* chore(tests): update test build
- Compose metadata: Add profile to test service. We can decouple services later and apply the profile to each. Rename. Pass the image name to the container.
- Fetch the parse_yaml.sh script during docker build, so we only fetch when necessary. Track the script in git. This is still a little janky, but better.
- test.sh: Use `HEAD` for diff (duh). Go back to using docker compose up so we're rebuilding only as needed and reusing the container - much faster, but the output isn't as pretty as `run`.
- pytest.ini, requirements.txt: Use pytest-dotenv to load env.products variables.
* fix(write): update JS instructions and pass tests.
* chore(ci): Install influx v2 CLI into the Docker test container.
* chore(ci): move .env file loading to pytest and run product tests separately with the proper .env file:
- Moves .env configs to be loaded by the test runner instead of by Docker. I'd prefer to ignore any product divergence and have them run interchangeably, but the pros outweigh the cons for now.
- Add example for passing custom commands in the compose file.
* fix(v3): replace curl example with v1 and v2, simplify JS example, pass tests.
feat(v3): add v1 write, test, update text:
- adds Serverless v1 /write example with note about autogen behavior
- adds link from v1 examples to Migrate guide
- adds description of autogen behavior to v1-compat guide
- updates "powered by..." text
- adapts examples to make them more testable, asserts `/write` status code
- replace "shared" examples (only used once) with inline to make them testable
fix(v3): promote migrate link
* Update content/influxdb/cloud-dedicated/get-started/write.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* chore(v3): Write: fix wording for curl success (#5156)
---------
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
- Adds a Docker image, scripts, and configuration that install and run pytest-codeblocks on Python and shell code samples in Markdown.
- Copied from https://github.com/jstirnaman/docs-codeblocks/
- TODO: The script copies files into a tmp directory configured as a bind mount for the Docker container. We might want to replace this with the experimental `watch` feature.
- TODO: Use a secret to configure the token. As is, values subbed from the dotenv file are exposed in your Docker container history.
Jason Stirnaman