- Fix the http-write-timeout definition, which should be similar to https://pkg.go.dev/net/http#Server.WriteTimeout
- Add a Vale spelling config for v2 server options
- Add influxdb to the test container packages (for testing service influxdb...)
- Add Dockerfile config and test setup for testing some InfluxDB startup config options (using influxd)
- Pre-commit hook:
- Redo test harness so commands can run in parallel
- Cleanup Vale command args
- Replace lint-staged and husky with Lefthook.
- Pre-push hook:
- Scan for package vulnerabilities
- Tests
- Run tests on demand, in parallel, or debugged using docker compose up, run, exec, etc.
- Each test container (service) runs the prepare-content script before running tests.
- Return to using Python instead of Docker to provide environment variables so they don't get exposed in Docker inspect.
- Bind mount influxctl config.toml and .env.test files
- Bind mount scripts for faster builds while iterating on test scripts.
- Vale:
- Override Acronyms style def--ignore acronym if it's linked.
- Adjusts times for writing and querying data within a recent retention period.
- Adjusts Unix timestamps (in line protocol samples) to yesterday's date.
- Adjusts UTC time bounds (in queries) to today and yesterday.
- Reverts mistakenly changed timestamps in clustered.
- Mount a volume to store test project files generated in tests.
- Allow URL and `url` in the linter
- Set timestamp precision for Telegraf example
- Fix Telegraf test, use Python to edit the TOML config file
- Remove date format linting in table cells. Copies Google's DateFormat style config to InfluxDataDocs style with modified scope.
- Update styles (vale sync)
- Skip database create in tests.
- Add example test for table create.
- Fix linting errors.
- Add TEST_RUN placeholder replacement in prepare-content.sh.
- For influxctl OAuth flow, add support for writing auth URLs to a container-shared file monitored by the host during the pre-commit hook.
- In prepare-content.sh, add Management API-associated placeholder substitutions.
- Update CONTRIBUTING.md with env.test requirements.
- Add python replacement for get-started database/bucket name
- Fixes race condition when deleting previous content:
- Assign each product it's own test content volume instead of sharing a volume.
- Delete existing content in the volume before copying test files.
- Pros and cons, but I suspect a similar approach will be better when ported to Circle CI.
- Reconfigures prettier linting.
- Adds .editorconfig to help with consistent editor settings
- Refactors test runs:
- Removes test configuration from compose.yaml (not suited for this use case).
- Splits test runner into test content setup and pytest that can be run separately or together (and with other test runners in the future).
- Configuration is in Dockerfiles and command line (`.lintstagedrc.mjs`)
- Updates CONTRIBUTING.md
- Updates client library write examples in cloud-dedicated and clustered.
- Pulls python client lib and CLI from Github repo instead of package repo so we can test the latest release. Specifies the last release tag.
- Adds CSV and JSON home sensor data samples to /static/downloads.
- Test image build bind mounts /static/downloads into /usr/src/app/test/tmp/data. Allows code samples to reference sample data as `./data/[file]`.
- Moves run-tests.sh to setup directory.
- Dockerfile cleanup. Add jq for inspection/troubleshooting.
* Strategies to improve query performance: operations, number and size of parquet files
Fixes#5108
- Add query performance strategies to optimize-queries
* Apply suggestions from code review
* Apply suggestions from code review
* Update content/influxdb/cloud-dedicated/query-data/execute-queries/optimize-queries.md
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* chore(v3): WIP query perf
Fixes#5108
* chore(v3): WIP: Query performance
* chore(v3): WIP: Explain the EXPLAIN report and indicators of query expense, performance problems.
* WIP: optimize queries - how to read a query plan, operators
* WIP: Read a query plan example 2
* WIP: moved how to read a query plan to its own page.
* WIP(v3): operators
* chore(v3): WIP add query plan info from DataFusion slides and @NGA-TRAN
* chore(v3): WIP read a query plan - explain tree format and reorganize
* WIP: query plan
- Adds Query Plan reference
- Completes Analyze a Query Plan, pending cleanup, continue at :471
- Added image from Nga's blog
- Updates EXPLAIN doc
- TODO: Create public docs for
https://github.com/influxdata/docs.influxdata.io/blob/main/content/operations/specifications/iox_runbooks/slow-queries.md
* chore(spelling): Vale config changes
- Add vale to package.json and use Yarn to manage the binary. You can use `npx vale` to run manually.
- Move InfluxData spelling ignore list into the style.
- Reorganize custom (product) spelling lists to comply with Vale 3.x
- Add InfluxDB v3 terms
* chore(spelling): Vale config changes
- Add vale to package.json and use Yarn to manage the binary. You can use `npx vale` to run manually.
- Move InfluxData spelling ignore list into the style.
- Reorganize custom (product) spelling lists to comply with Vale 3.x
- Add InfluxDB v3 terms
* chore(v3): Reorg of query troubleshooting and optimizing docs
- Adds query-data/troubleshoot-and-optimize
- Splits optimize docs into troubleshoot and optimize docs
- Moves Flight response doc to flight-responses.md
* chore: Fixes broken links, typos, missing content, etc.
- Fixes various errors and style violations reported by Vale.
- Fixes broken links and missing content in glossaries.
- Fixes missing and extraneous whitespace.
* Apply suggestions from code review
* Update content/influxdb/cloud-dedicated/query-data/troubleshoot-and-optimize/analyze-query-plan.md
* chore(clustered): Query plan: Apply review suggestions
Co-Authored-By: Nga Tran <20850014+NGA-TRAN@users.noreply.github.com>
* feature(v3): Analyze a query plan:
- Apply code formatting to plan implementor names
- Simplify some points
- Add links
* add query plan html diagram (#5365)
* Update content/influxdb/cloud-dedicated/query-data/troubleshoot-and-optimize/analyze-query-plan.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Update content/influxdb/cloud-dedicated/query-data/troubleshoot-and-optimize/optimize-queries.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Update content/influxdb/cloud-dedicated/query-data/troubleshoot-and-optimize/trace.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Update content/influxdb/cloud-dedicated/query-data/troubleshoot-and-optimize/trace.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Update content/influxdb/cloud-dedicated/query-data/troubleshoot-and-optimize/trace.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Update content/influxdb/cloud-serverless/query-data/troubleshoot-and-optimize/analyze-query-plan.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* fix(v3): finish the EXPLAIN descriptions and examples
* chore(tests): Setup a python venv in test containers
* fix(ci): Vale vocab
* fix(v3): Punctuation typo
* chore(ci): Update README
* fix(v3): Apply review suggestions and capitalization
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* fix(v3): Add note to optimize page and revise troubleshoot
* fix(v3): optimize-queries link
---------
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Nga Tran <20850014+NGA-TRAN@users.noreply.github.com>
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* chore(telegraf): Improve CSV parser doc and run Telegraf tests:
- Revise CSV parser documentation and add examples.
- Configure test runner for Telegraf tests.
* Update content/telegraf/v1/data_formats/input/csv.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Update content/telegraf/v1/data_formats/input/csv.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Update content/telegraf/v1/data_formats/input/csv.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
---------
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* chore(v3): preprocess writes (Telegraf, etc)
* feat: optimize writes using Telegraf
* WIP: dedup plugin
* WIP: Dedup
* chore(v3): Writes: Preprocess data using Telegraf: Dedup example
(#5155)
- Add example using processors.dedup in Telegraf.
- The plugin or configuration doesn't currently work as expected. Investigating...
* chore(v3): Write data: pre-process data using Telegraf
- Add Telegraf examples that pre-process data to avoid schema conflicts and reduce payload (bandwith, memory, etc.) when writing data.
- For Serverless. Will port to Dedicated and Clustered after approval.
- Doesn't include Dedup yet - I'm waiting on help to make it work (see branch https://github.com/influxdata/docs-v2/compare/issue5155-preprocess-with-dedup)
* Update content/influxdb/cloud-serverless/write-data/best-practices/optimize-writes.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* chore(test): Test harness setup, use golang base image
- Use golang base image for ease of Go environment setup. Installing Python separately is easier than Go.
- Add flag to bypass python warning about using system python and pip.
- Remove redundant env.products from pytest.ini; we already load it into environment variables.
* chore(v3): Optimize writes: Use Telegraf
Closes Telegraf guide: how to use the Converter processor to coerce data before writing #5155
- Adds use cases to Dedicated and Clustered docs
- Adds example for Execd processor plugin to all
- Clean up and pass tests
- Remove shared code sample
* fix(v3): Optimize writes: Placeholders and indent fixes
* Update content/influxdb/cloud-dedicated/write-data/best-practices/optimize-writes.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Update content/influxdb/cloud-serverless/write-data/best-practices/optimize-writes.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Update content/influxdb/clustered/write-data/best-practices/optimize-writes.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
---------
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* chore(v3): DBRP: Update Serverless API ref (#5234)
* chore(ci): Tests: substitute placeholders in JSON body parameters
* chore(v3): DBRP: copy page from influxql to admin/dbrps.
* chore(ci): Trying to prevent "a Influx..."
* chore(ci): Allow autogen
* chore(serverless): DBRP glossary entry
* chore(v2): add default DBRP note
* chore(v3): use product-name shortcode
* chore(v3): use product-name shortcode, clarify retention period and mapping
* chore(serverless): Create Bucket:
- add missing example for Data Explorer
- move Retention Period syntax and define retentionRules syntax
- setup for tests
* chore(serverless): Create Bucket:
- add missing example for Data Explorer
- move Retention Period syntax and define retentionRules syntax
- setup for tests
* chore(serverless): Update bucket: alias to match other products
* chore(v3): Serverless for prototyping: update with DBRP behavior
* Serverless auto-generated buckets have 3-day default retention period
Fixes#5234
- v1-compat guide:
- passes tests
- explains DBRPs (default, automatic, and management)
- migrate DBRP content from query-data/influxql/dbrp
- v2-compat guide:
- fix TOC
- add list code example
- rename influxdb-v1-api to api/v1-http, migrate params from v1-compat guide, and explain DBRP in context
- add write-data/api/v1-http and migrate content from v1-compat guide
- move optimize and troubleshooting up to /query-data
- add Related items support for paths containing http
- substitute placeholders in tests
* chore(test): update configuration in README
* fix(serverless): Optimize queries parent
* fix(serverless): create-bucket cleanup
* fix: cleanup
* fix: close tag
* Apply suggestions from code review
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* fix(v3): permission to create dbrp-mapped bucket
---------
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* chore(ci): Install Telegraf to test container
* chore(v3): test telegraf config and startup:
- creates the config file for the test run. TODO: remove the duplication and use the content for test setup.
- fix placeholder name
* fix(v3): make test pass for curl API example
* chore(ci): make tests pass for write. Add dependencies to Dockerfile.
* chore(tests): update test build
- Compose metadata: Add profile to test service. We can decouple services later and apply the profile to each. Rename. Pass the image name to the container.
- Fetch the parse_yaml.sh script during docker build, so we only fetch when necessary. Track the script in git. This is still a little janky, but better.
- test.sh: Use `HEAD` for diff (duh). Go back to using docker compose up so we're rebuilding only as needed and reusing the container - much faster, but the output isn't as pretty as `run`.
- pytest.ini, requirements.txt: Use pytest-dotenv to load env.products variables.
* fix(write): update JS instructions and pass tests.
* chore(ci): Install influx v2 CLI into the Docker test container.
* chore(ci): move .env file loading to pytest and run product tests separately with the proper .env file:
- Moves .env configs to be loaded by the test runner instead of by Docker. I'd prefer to ignore any product divergence and have them run interchangeably, but the pros outweigh the cons for now.
- Add example for passing custom commands in the compose file.
* fix(v3): replace curl example with v1 and v2, simplify JS example, pass tests.
feat(v3): add v1 write, test, update text:
- adds Serverless v1 /write example with note about autogen behavior
- adds link from v1 examples to Migrate guide
- adds description of autogen behavior to v1-compat guide
- updates "powered by..." text
- adapts examples to make them more testable, asserts `/write` status code
- replace "shared" examples (only used once) with inline to make them testable
fix(v3): promote migrate link
* Update content/influxdb/cloud-dedicated/get-started/write.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* chore(v3): Write: fix wording for curl success (#5156)
---------
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
- Bad Dockerfile merge.
- Add condition to abort if there are no files to test.
- Revert search path to `.`. If we need to change it, we can do it in pytest.ini.
* nest reference docs menu in v2 and cloud
* move migrate data into write data
* nest notebooks under tools
* nest templates under tools
* nest telegraf configs under tools
* cleanup weights and titles of tools sections
* reorg org and bucket management under admin
* add backup/restore to admin, clean up related links
* restructure security sections under admin
* Apply suggestions from code review
Co-authored-by: Jason Stirnaman <stirnamanj@gmail.com>
* updated serverless cli docs to address pr feedback
* Update content/influxdb/cloud/admin/organizations/delete-org.md
Co-authored-by: Jason Stirnaman <stirnamanj@gmail.com>
* port v2 install changes to new structure
* address pr feedback and ported missing content to serverless
---------
Co-authored-by: Jason Stirnaman <stirnamanj@gmail.com>
* chore(ci): Docker install gpg and wget for test image.
* chore(ci): add a config (temporary) for v2 tests.
* chore(ci): tests: replace latest-patch tag with environment variable.
* chore(ci): test runner
- test.sh diffs against origin/master
- test.sh builds the image for each run and then removes the container after the run.
- tesh.sh adds examples for running and debugging to remind myself.
- compose.yaml adds a bind mount to copy docs-v2/data into docs-v2/test.
- Dockerfile: simplify path use
- run-tests.sh: make default run less verbose, add support for a verbose option (and future options).
- run-tests.sh: generates a .env.products file from data/products.yml and sources the file variables as environment variables to use in test runs.
- run-tests.sh: adds some misc. test setup.
- requirements.txt: return to using the influxdb3-python release
* chore(v2): Update and test v2 OSS and CLI install instructions:
- Update download URLs and paths for all distributions.
- Update and fix install instructions, make consistent between browser downloading and shell
- Make testable.
* chore(v2): Ignore parse_yaml.sh download.
* chore(test): Use my python client fork (pending approval) to allow custom headers.
* feature(query): Add Optimize Queries page with query analysis help
- Closes Client library query traces: Python #5143
- Dedicated and Clustered examples for enabling query tracing and extracting headers
- System.queries table
- Explain and Analyze
- For now, skip tests for sample Flight responses until we add in code samples.
* Update content/influxdb/clustered/query-data/execute-queries/optimize-queries.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Update content/influxdb/cloud-dedicated/query-data/execute-queries/optimize-queries.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Update content/influxdb/cloud-dedicated/query-data/execute-queries/optimize-queries.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Update content/influxdb/cloud-dedicated/query-data/execute-queries/optimize-queries.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Update content/influxdb/clustered/query-data/execute-queries/optimize-queries.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Update content/influxdb/clustered/query-data/execute-queries/optimize-queries.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Update content/influxdb/clustered/query-data/execute-queries/optimize-queries.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Update content/influxdb/clustered/query-data/execute-queries/optimize-queries.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Update content/influxdb/cloud-dedicated/query-data/execute-queries/optimize-queries.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* Update content/influxdb/cloud-dedicated/query-data/execute-queries/optimize-queries.md
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* feat(v3): influx-trace-id for dedicated, tracing not ready for clustered (Client library query traces: Python #5143)
---------
Co-authored-by: Scott Anderson <sanderson@users.noreply.github.com>
* chore(ci): Update vocab.
* chore(test): Set language-specific replacements for code sample tests
This is a bit of kludge to allow for language-specific replacement patterns
while keeping the code samples valid when a user replaces placeholder strings with their own and copies the sample.
This is necessary for executing (not for validating syntax) of the code samples.
- Workaround for identifying language-specific replacements for pytest-codeblocks test runs.
- Uses Python f-strings to identify Python-specific placeholders.
- To import os for your code sample and hide it from users, wrap it in a comment and use the `pytest-codeblocks:cont` comment tag. See d0ca7e22b8/README.md?plain=1#L117
- Adds a Docker image, scripts, and configuration that install and run pytest-codeblocks on Python and shell code samples in Markdown.
- Copied from https://github.com/jstirnaman/docs-codeblocks/
- TODO: The script copies files into a tmp directory configured as a bind mount for the Docker container. We might want to replace this with the experimental `watch` feature.
- TODO: Use a secret to configure the token. As is, values subbed from the dotenv file are exposed in your Docker container history.
Jason Stirnaman
Andreas Deininger
Scott Anderson