From eef29546d776ce0d413cfdc6ec9ba251cd5914e8 Mon Sep 17 00:00:00 2001 From: meelahme Date: Thu, 16 Oct 2025 12:02:37 -0700 Subject: [PATCH 1/9] fix(influxdb3): add python directory requirement note to plugins setup --- .../influxdb3-get-started/processing-engine.md | 3 +++ content/shared/influxdb3-plugins/_index.md | 14 ++++++++++++++ 2 files changed, 17 insertions(+) diff --git a/content/shared/influxdb3-get-started/processing-engine.md b/content/shared/influxdb3-get-started/processing-engine.md index 009b3d4ee..cf54ca3e8 100644 --- a/content/shared/influxdb3-get-started/processing-engine.md +++ b/content/shared/influxdb3-get-started/processing-engine.md @@ -48,6 +48,9 @@ when starting the {{% product-name %}} server. `PLUGIN_DIR` is your file system location for storing [plugin](#plugin) files for the processing engine to run. +> [!Note] +> If you manually installed from tar.gz, ensure the `influxdb3` binary and `python/` directory remain in the same parent location. The install script handles this automatically. + {{% code-placeholders "PLUGIN_DIR" %}} ```bash diff --git a/content/shared/influxdb3-plugins/_index.md b/content/shared/influxdb3-plugins/_index.md index 14dace282..d127cbc73 100644 --- a/content/shared/influxdb3-plugins/_index.md +++ b/content/shared/influxdb3-plugins/_index.md @@ -34,6 +34,20 @@ Once you have all the prerequisites in place, follow these steps to implement th To activate the Processing Engine, start your {{% product-name %}} server with the `--plugin-dir` flag. This flag tells InfluxDB where to load your plugin files. +> [!Important] +> #### Keep influxdb3 and python/ together +> +> The influxdb3 binary requires the adjacent `python/` directory to function. +> If you manually extract from tar.gz, keep them in the same parent directory: +> +> {{< filesystem-diagram >}} +> - your-install-location/ +> - influxdb3 +> - python/ +> {{< /filesystem-diagram >}} +> +> Add the directory (not just the binary) to your PATH. Moving only the binary will cause Python runtime errors. + {{% code-placeholders "NODE_ID|OBJECT_STORE_TYPE|PLUGIN_DIR" %}} ```bash From b2109ce005421c99bcd2d31de8fdb1f0ba8d10b5 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 16 Oct 2025 12:07:16 -0700 Subject: [PATCH 2/9] Update content/shared/influxdb3-plugins/_index.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- content/shared/influxdb3-plugins/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-plugins/_index.md b/content/shared/influxdb3-plugins/_index.md index d127cbc73..7459735c1 100644 --- a/content/shared/influxdb3-plugins/_index.md +++ b/content/shared/influxdb3-plugins/_index.md @@ -35,7 +35,7 @@ Once you have all the prerequisites in place, follow these steps to implement th To activate the Processing Engine, start your {{% product-name %}} server with the `--plugin-dir` flag. This flag tells InfluxDB where to load your plugin files. > [!Important] -> #### Keep influxdb3 and python/ together +> **Keep influxdb3 and python/ together.** > > The influxdb3 binary requires the adjacent `python/` directory to function. > If you manually extract from tar.gz, keep them in the same parent directory: From 6bcec0050aaa790cc2ab5aba5d5774fadbbac8c0 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 16 Oct 2025 12:07:23 -0700 Subject: [PATCH 3/9] Update content/shared/influxdb3-plugins/_index.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- content/shared/influxdb3-plugins/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-plugins/_index.md b/content/shared/influxdb3-plugins/_index.md index 7459735c1..bf1b27505 100644 --- a/content/shared/influxdb3-plugins/_index.md +++ b/content/shared/influxdb3-plugins/_index.md @@ -46,7 +46,7 @@ To activate the Processing Engine, start your {{% product-name %}} server with t > - python/ > {{< /filesystem-diagram >}} > -> Add the directory (not just the binary) to your PATH. Moving only the binary will cause Python runtime errors. +> Add the parent directory that contains both influxdb3 and python/ to your PATH; do not move the binary out of this directory. {{% code-placeholders "NODE_ID|OBJECT_STORE_TYPE|PLUGIN_DIR" %}} From e25d29a5145e7797417415da82d5431a749fe59c Mon Sep 17 00:00:00 2001 From: meelahme Date: Thu, 16 Oct 2025 12:42:07 -0700 Subject: [PATCH 4/9] fix(influxdb3): fix broken links in plugins documentation --- content/shared/influxdb3-plugins/_index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/shared/influxdb3-plugins/_index.md b/content/shared/influxdb3-plugins/_index.md index bf1b27505..becaa9026 100644 --- a/content/shared/influxdb3-plugins/_index.md +++ b/content/shared/influxdb3-plugins/_index.md @@ -311,7 +311,7 @@ def process_request(influxdb3_local, query_parameters, request_headers, request_ After writing your plugin: - [Create a trigger](#use-the-create-trigger-command) to connect your plugin to database events -- [Install any Python dependencies](#install-python-dependencies) your plugin requires +- [Install any Python dependencies](#manage-plugin-dependencies) your plugin requires - Learn how to [extend plugins with the API](/influxdb3/version/extend-plugin/) ## Set up a trigger @@ -592,7 +592,7 @@ These examples install the specified Python package (for example, pandas) into t > If you need to create a custom virtual environment, use the Python interpreter bundled with InfluxDB 3. Don't use the system Python. > Creating a virtual environment with the system Python (for example, using `python -m venv`) can lead to runtime errors and plugin failures. > ->For more information, see the [processing engine README](https://github.com/influxdata/influxdb/blob/main/README_processing_engine.md#official-builds). +>For more information, see the [processing engine README](https://github.com/influxdata/influxdb/blob/main/README_processing_engine.md). {{% /code-placeholders %}} From f1dc112f4319c3feec58f4e13dec25a943d8861c Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 17 Oct 2025 10:40:45 -0700 Subject: [PATCH 5/9] Update content/shared/influxdb3-get-started/processing-engine.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-get-started/processing-engine.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-get-started/processing-engine.md b/content/shared/influxdb3-get-started/processing-engine.md index cf54ca3e8..954a01050 100644 --- a/content/shared/influxdb3-get-started/processing-engine.md +++ b/content/shared/influxdb3-get-started/processing-engine.md @@ -49,7 +49,7 @@ when starting the {{% product-name %}} server. the processing engine to run. > [!Note] -> If you manually installed from tar.gz, ensure the `influxdb3` binary and `python/` directory remain in the same parent location. The install script handles this automatically. +> If you manually installed {{% product-name %}} from a tar archive, ensure the `influxdb3` binary and `python/` directory remain in the same parent directory. The install script handles this automatically. {{% code-placeholders "PLUGIN_DIR" %}} From 631fb5c802e4975cbb7b23c42b75b1411075559b Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 17 Oct 2025 10:41:00 -0700 Subject: [PATCH 6/9] Update content/shared/influxdb3-plugins/_index.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-plugins/_index.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/content/shared/influxdb3-plugins/_index.md b/content/shared/influxdb3-plugins/_index.md index becaa9026..de65a5095 100644 --- a/content/shared/influxdb3-plugins/_index.md +++ b/content/shared/influxdb3-plugins/_index.md @@ -35,18 +35,20 @@ Once you have all the prerequisites in place, follow these steps to implement th To activate the Processing Engine, start your {{% product-name %}} server with the `--plugin-dir` flag. This flag tells InfluxDB where to load your plugin files. > [!Important] -> **Keep influxdb3 and python/ together.** +> #### Keep influxdb3 and python/ together > -> The influxdb3 binary requires the adjacent `python/` directory to function. -> If you manually extract from tar.gz, keep them in the same parent directory: +> The `influxdb3` tar archive includes the `influxdb3` binary and a `python/` directory. +> If you move the `influxdb3` binary, ensure that the `influxdb3` binary and the `python/` +> directory remain together. > > {{< filesystem-diagram >}} -> - your-install-location/ +> - your-install-location/ > - influxdb3 > - python/ > {{< /filesystem-diagram >}} > -> Add the parent directory that contains both influxdb3 and python/ to your PATH; do not move the binary out of this directory. +> Add the parent directory that contains both the `influxdb3` binary and the `python/` directory +> to your system $PATH. {{% code-placeholders "NODE_ID|OBJECT_STORE_TYPE|PLUGIN_DIR" %}} From 9e767bcd63edad4a852f0a72bd28dc6bfbceecad Mon Sep 17 00:00:00 2001 From: meelahme Date: Fri, 17 Oct 2025 10:55:09 -0700 Subject: [PATCH 7/9] fixing filesystem-diagram --- content/shared/influxdb3-plugins/_index.md | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/content/shared/influxdb3-plugins/_index.md b/content/shared/influxdb3-plugins/_index.md index de65a5095..0a860f656 100644 --- a/content/shared/influxdb3-plugins/_index.md +++ b/content/shared/influxdb3-plugins/_index.md @@ -37,18 +37,15 @@ To activate the Processing Engine, start your {{% product-name %}} server with t > [!Important] > #### Keep influxdb3 and python/ together > -> The `influxdb3` tar archive includes the `influxdb3` binary and a `python/` directory. -> If you move the `influxdb3` binary, ensure that the `influxdb3` binary and the `python/` -> directory remain together. +> The influxdb3 binary requires the adjacent `python/` directory to function. +> If you manually extract from tar.gz, keep them in the same parent directory: +> ``` +> your-install-location/ +> ├── influxdb3 +> └── python/ +> ``` > -> {{< filesystem-diagram >}} -> - your-install-location/ -> - influxdb3 -> - python/ -> {{< /filesystem-diagram >}} -> -> Add the parent directory that contains both the `influxdb3` binary and the `python/` directory -> to your system $PATH. +> Add the parent directory to your PATH; do not move the binary out of this directory. {{% code-placeholders "NODE_ID|OBJECT_STORE_TYPE|PLUGIN_DIR" %}} From 059d60117c8da204736dfd870d5928934f8b06d6 Mon Sep 17 00:00:00 2001 From: meelahme Date: Fri, 17 Oct 2025 11:19:46 -0700 Subject: [PATCH 8/9] update to title in importatnt note to make it more clear --- content/shared/influxdb3-plugins/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-plugins/_index.md b/content/shared/influxdb3-plugins/_index.md index 0a860f656..145a34bfd 100644 --- a/content/shared/influxdb3-plugins/_index.md +++ b/content/shared/influxdb3-plugins/_index.md @@ -35,7 +35,7 @@ Once you have all the prerequisites in place, follow these steps to implement th To activate the Processing Engine, start your {{% product-name %}} server with the `--plugin-dir` flag. This flag tells InfluxDB where to load your plugin files. > [!Important] -> #### Keep influxdb3 and python/ together +> #### Keep the influxdb3 binary with its python directory > > The influxdb3 binary requires the adjacent `python/` directory to function. > If you manually extract from tar.gz, keep them in the same parent directory: From ca24b3be54df8db7ea27797882c92ad3b25b58e0 Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Mon, 20 Oct 2025 10:17:56 -0500 Subject: [PATCH 9/9] Influxdb3 config options (#6419) * docs(cli): fix global vs serve-specific flag documentation - Remove --verbose from global flags (it's serve-specific) - Document --num-io-threads as global-only flag - Add clear examples showing correct flag positioning - Update serve.md files with global flag usage notes - Fix config-options.md to separate Core/Enterprise examples Resolves incorrect CLI usage patterns that would cause errors. Global flags must go before 'serve', serve-specific flags go after. * docs(cli): remove Tokio runtime options from CLI index pages - Remove detailed Tokio runtime options tables from CLI index pages - Replace with simplified global options and link to config-options - Add examples showing correct global flag positioning - Fix --verbose usage to be serve-specific (after serve command) - Add --num-io-threads example as global flag (before serve command) These detailed options are now documented in config-options.md with proper global vs serve-specific categorization. --- .../core/reference/cli/influxdb3/_index.md | 51 ++-- .../core/reference/cli/influxdb3/serve.md | 13 +- .../reference/cli/influxdb3/_index.md | 52 ++-- .../reference/cli/influxdb3/serve.md | 14 +- .../shared/influxdb3-cli/config-options.md | 259 ++++++++++-------- 5 files changed, 212 insertions(+), 177 deletions(-) diff --git a/content/influxdb3/core/reference/cli/influxdb3/_index.md b/content/influxdb3/core/reference/cli/influxdb3/_index.md index b0788e001..495a03f3d 100644 --- a/content/influxdb3/core/reference/cli/influxdb3/_index.md +++ b/content/influxdb3/core/reference/cli/influxdb3/_index.md @@ -37,36 +37,13 @@ influxdb3 [GLOBAL-OPTIONS] [COMMAND] ## Global options -| Option | | Description | -| :----- | :------------------------------------ | :------------------------------------------------------------------------------------------------ | -| | `--num-threads` | Maximum number of IO runtime threads to use | -| | `--io-runtime-type` | IO tokio runtime type (`current-thread`, `multi-thread` _(default)_, or `multi-thread-alt`) | -| | `--io-runtime-disable-lifo-slot` | Disable LIFO slot of IO runtime | -| | `--io-runtime-event-interval` | Number of scheduler ticks after which the IOtokio runtime scheduler will poll for external events | -| | `--io-runtime-global-queue-interval` | Number of scheduler ticks after which the IO runtime scheduler will poll the global task queue | -| | `--io-runtime-max-blocking-threads` | Limit for additional threads spawned by the IO runtime | -| | `--io-runtime-max-io-events-per-tick` | Maximum number of events to be processed per tick by the tokio IO runtime | -| | `--io-runtime-thread-keep-alive` | Custom timeout for a thread in the blocking pool of the tokio IO runtime | -| | `--io-runtime-thread-priority` | Set thread priority tokio IO runtime workers | -| `-h` | `--help` | Print help information | -| | `--help-all` | Print detailed help information including runtime configuration options | -| `-V` | `--version` | Print version | +| Option | | Description | +| :----- | :---------------- | :-------------------------------------------------------------------- | +| `-h` | `--help` | Print help information | +| | `--help-all` | Print detailed help information including runtime configuration options | +| `-V` | `--version` | Print version | -### Option environment variables - -You can use the following environment variables to set `influxdb3` global options: - -| Environment Variable | Option | -| :-------------------------------------------- | :------------------------------------ | -| `INFLUXDB3_NUM_THREADS` | `--num-threads` | -| `INFLUXDB3_IO_RUNTIME_TYPE` | `--io-runtime-type` | -| `INFLUXDB3_IO_RUNTIME_DISABLE_LIFO_SLOT` | `--io-runtime-disable-lifo-slot` | -| `INFLUXDB3_IO_RUNTIME_EVENT_INTERVAL` | `--io-runtime-event-interval` | -| `INFLUXDB3_IO_RUNTIME_GLOBAL_QUEUE_INTERVAL` | `--io-runtime-global-queue-interval` | -| `INFLUXDB3_IO_RUNTIME_MAX_BLOCKING_THREADS` | `--io-runtime-max-blocking-threads` | -| `INFLUXDB3_IO_RUNTIME_MAX_IO_EVENTS_PER_TICK` | `--io-runtime-max-io-events-per-tick` | -| `INFLUXDB3_IO_RUNTIME_THREAD_KEEP_ALIVE` | `--io-runtime-thread-keep-alive` | -| `INFLUXDB3_IO_RUNTIME_THREAD_PRIORITY` | `--io-runtime-thread-priority` | +For advanced global configuration options (including `--num-io-threads` and other runtime settings), see [Configuration options](/influxdb3/core/reference/config-options/#global-configuration-options). ## Examples @@ -88,6 +65,17 @@ influxdb3 serve \ --node-id my-host-01 ``` +### Run the InfluxDB 3 server with custom IO threads + + + +```bash +influxdb3 --num-io-threads=8 serve \ + --object-store file \ + --data-dir ~/.influxdb3 \ + --node-id my-host-01 +``` + ### Display short-form help for all commands @@ -109,10 +97,11 @@ influxdb3 --help ```bash -influxdb3 serve -v \ +influxdb3 serve \ --object-store file \ --data-dir ~/.influxdb3 \ - --node-id my-host-01 + --node-id my-host-01 \ + --verbose ``` ### Run {{% product-name %}} with debug logging using LOG_FILTER diff --git a/content/influxdb3/core/reference/cli/influxdb3/serve.md b/content/influxdb3/core/reference/cli/influxdb3/serve.md index a66a59209..ef5b9e019 100644 --- a/content/influxdb3/core/reference/cli/influxdb3/serve.md +++ b/content/influxdb3/core/reference/cli/influxdb3/serve.md @@ -30,6 +30,16 @@ influxdb3 serve [OPTIONS] --node-id > [!NOTE] > `--node-id` supports alphanumeric strings with optional hyphens. +> [!Important] +> #### Global configuration options +> Some configuration options (like [`--num-io-threads`](/influxdb3/core/reference/config-options/#num-io-threads)) are **global** and must be specified **before** the `serve` command: +> +> ```bash +> influxdb3 --num-io-threads=8 serve --node-id=node0 --object-store=file --verbose +> ``` +> +> See [Global configuration options](/influxdb3/core/reference/config-options/#global-configuration-options) for the complete list. + ## Options | Option | | Description | @@ -116,7 +126,6 @@ influxdb3 serve [OPTIONS] --node-id | | `--traces-jaeger-debug-name` | _See [configuration options](/influxdb3/core/reference/config-options/#traces-jaeger-debug-name)_ | | | `--traces-jaeger-max-msgs-per-second` | _See [configuration options](/influxdb3/core/reference/config-options/#traces-jaeger-max-msgs-per-second)_ | | | `--traces-jaeger-tags` | _See [configuration options](/influxdb3/core/reference/config-options/#traces-jaeger-tags)_ | -| `-v` | `--verbose` | Enable verbose output | | | `--virtual-env-location` | _See [configuration options](/influxdb3/core/reference/config-options/#virtual-env-location)_ | | | `--wal-flush-interval` | _See [configuration options](/influxdb3/core/reference/config-options/#wal-flush-interval)_ | | | `--wal-max-write-buffer-size` | _See [configuration options](/influxdb3/core/reference/config-options/#wal-max-write-buffer-size)_ | @@ -164,10 +173,10 @@ influxdb3 serve \ ```bash influxdb3 serve \ - --verbose \ --object-store file \ --data-dir ~/.influxdb3 \ --node-id my-host-01 + --verbose ``` ### Run InfluxDB 3 with debug logging using LOG_FILTER diff --git a/content/influxdb3/enterprise/reference/cli/influxdb3/_index.md b/content/influxdb3/enterprise/reference/cli/influxdb3/_index.md index 863318111..1a52c7f40 100644 --- a/content/influxdb3/enterprise/reference/cli/influxdb3/_index.md +++ b/content/influxdb3/enterprise/reference/cli/influxdb3/_index.md @@ -37,36 +37,13 @@ influxdb3 [GLOBAL-OPTIONS] [COMMAND] ## Global options -| Option | | Description | -| :----- | :------------------------------------ | :------------------------------------------------------------------------------------------------ | -| | `--num-threads` | Maximum number of IO runtime threads to use | -| | `--io-runtime-type` | IO tokio runtime type (`current-thread`, `multi-thread` _(default)_, or `multi-thread-alt`) | -| | `--io-runtime-disable-lifo-slot` | Disable LIFO slot of IO runtime | -| | `--io-runtime-event-interval` | Number of scheduler ticks after which the IOtokio runtime scheduler will poll for external events | -| | `--io-runtime-global-queue-interval` | Number of scheduler ticks after which the IO runtime scheduler will poll the global task queue | -| | `--io-runtime-max-blocking-threads` | Limit for additional threads spawned by the IO runtime | -| | `--io-runtime-max-io-events-per-tick` | Maximum number of events to be processed per tick by the tokio IO runtime | -| | `--io-runtime-thread-keep-alive` | Custom timeout for a thread in the blocking pool of the tokio IO runtime | -| | `--io-runtime-thread-priority` | Set thread priority tokio IO runtime workers | -| `-h` | `--help` | Print help information | -| | `--help-all` | Print detailed help information | -| `-V` | `--version` | Print version | +| Option | | Description | +| :----- | :---------------- | :-------------------------------------------------------------------- | +| `-h` | `--help` | Print help information | +| | `--help-all` | Print detailed help information including runtime configuration options | +| `-V` | `--version` | Print version | -### Option environment variables - -You can use the following environment variables to set `influxdb3` global options: - -| Environment Variable | Option | -| :-------------------------------------------- | :------------------------------------ | -| `INFLUXDB3_NUM_THREADS` | `--num-threads` | -| `INFLUXDB3_IO_RUNTIME_TYPE` | `--io-runtime-type` | -| `INFLUXDB3_IO_RUNTIME_DISABLE_LIFO_SLOT` | `--io-runtime-disable-lifo-slot` | -| `INFLUXDB3_IO_RUNTIME_EVENT_INTERVAL` | `--io-runtime-event-interval` | -| `INFLUXDB3_IO_RUNTIME_GLOBAL_QUEUE_INTERVAL` | `--io-runtime-global-queue-interval` | -| `INFLUXDB3_IO_RUNTIME_MAX_BLOCKING_THREADS` | `--io-runtime-max-blocking-threads` | -| `INFLUXDB3_IO_RUNTIME_MAX_IO_EVENTS_PER_TICK` | `--io-runtime-max-io-events-per-tick` | -| `INFLUXDB3_IO_RUNTIME_THREAD_KEEP_ALIVE` | `--io-runtime-thread-keep-alive` | -| `INFLUXDB3_IO_RUNTIME_THREAD_PRIORITY` | `--io-runtime-thread-priority` | +For advanced global configuration options (including `--num-io-threads` and other runtime settings), see [Configuration options](/influxdb3/enterprise/reference/config-options/#global-configuration-options). ## Examples @@ -93,6 +70,18 @@ influxdb3 serve \ --cluster-id my-cluster-01 ``` +### Run the InfluxDB 3 server with custom IO threads + + + +```bash +influxdb3 --num-io-threads=8 serve \ + --object-store file \ + --data-dir ~/.influxdb3 \ + --node-id my-host-01 \ + --cluster-id my-cluster-01 +``` + ### Display short-form help for all commands @@ -114,11 +103,12 @@ influxdb3 --help ```bash -influxdb3 serve -v \ +influxdb3 serve \ --object-store file \ --data-dir ~/.influxdb3 \ --node-id my-host-01 \ - --cluster-id my-cluster-01 + --cluster-id my-cluster-01 \ + --verbose ``` ### Run {{% product-name %}} with debug logging using LOG_FILTER diff --git a/content/influxdb3/enterprise/reference/cli/influxdb3/serve.md b/content/influxdb3/enterprise/reference/cli/influxdb3/serve.md index c44102dd3..bded5f63b 100644 --- a/content/influxdb3/enterprise/reference/cli/influxdb3/serve.md +++ b/content/influxdb3/enterprise/reference/cli/influxdb3/serve.md @@ -33,6 +33,16 @@ influxdb3 serve [OPTIONS] \ > [!NOTE] > `--node-id` and `--cluster-id` support alphanumeric strings with optional hyphens. +> [!Important] +> #### Global configuration options +> Some configuration options (like [`--num-io-threads`](/influxdb3/enterprise/reference/config-options/#num-io-threads)) are **global** and must be specified **before** the `serve` command: +> +> ```bash +> influxdb3 --num-io-threads=8 serve --node-id=node0 --cluster-id=cluster0 --verbose +> ``` +> +> See [Global configuration options](/influxdb3/enterprise/reference/config-options/#global-configuration-options) for the complete list. + ## Options | Option | | Description | @@ -100,6 +110,7 @@ influxdb3 serve [OPTIONS] \ | {{< req "\*" >}} | `--node-id` | _See [configuration options](/influxdb3/enterprise/reference/config-options/#node-id)_ | | | `--node-id-from-env` | _See [configuration options](/influxdb3/enterprise/reference/config-options/#node-id-from-env)_ | | | `--num-cores` | _See [configuration options](/influxdb3/enterprise/reference/config-options/#num-cores)_ | +| | `--num-datafusion-threads` | _See [configuration options](/influxdb3/enterprise/reference/config-options/#num-datafusion-threads)_ | | | `--num-database-limit` | _See [configuration options](/influxdb3/enterprise/reference/config-options/#num-database-limit)_ | | | `--num-table-limit` | _See [configuration options](/influxdb3/enterprise/reference/config-options/#num-table-limit)_ | | | `--num-total-columns-per-table-limit` | _See [configuration options](/influxdb3/enterprise/reference/config-options/#num-total-columns-per-table-limit)_ | @@ -140,7 +151,6 @@ influxdb3 serve [OPTIONS] \ | | `--traces-jaeger-max-msgs-per-second` | _See [configuration options](/influxdb3/enterprise/reference/config-options/#traces-jaeger-max-msgs-per-second)_ | | | `--traces-jaeger-tags` | _See [configuration options](/influxdb3/enterprise/reference/config-options/#traces-jaeger-tags)_ | | | `--use-pacha-tree` | _See [configuration options](/influxdb3/enterprise/reference/config-options/#use-pacha-tree)_ | -| `-v` | `--verbose` | Enable verbose output | | | `--virtual-env-location` | _See [configuration options](/influxdb3/enterprise/reference/config-options/#virtual-env-location)_ | | | `--wait-for-running-ingestor` | _See [configuration options](/influxdb3/enterprise/reference/config-options/#wait-for-running-ingestor)_ | | | `--wal-flush-interval` | _See [configuration options](/influxdb3/enterprise/reference/config-options/#wal-flush-interval)_ | @@ -220,11 +230,11 @@ influxdb3 serve \ ```bash influxdb3 serve \ - --verbose \ --object-store file \ --data-dir ~/.influxdb3 \ --node-id my-host-01 \ --cluster-id my-cluster-01 + --verbose ``` ### Run InfluxDB 3 with debug logging using LOG_FILTER diff --git a/content/shared/influxdb3-cli/config-options.md b/content/shared/influxdb3-cli/config-options.md index 8d910f1fc..2eca2caa8 100644 --- a/content/shared/influxdb3-cli/config-options.md +++ b/content/shared/influxdb3-cli/config-options.md @@ -1,26 +1,48 @@ -{{< product-name >}} lets you customize your server configuration by using -`influxdb3 serve` command options or by setting environment variables. +{{< product-name >}} lets you customize your configuration by using +`influxdb3` command options or by setting environment variables. ## Configure your server -Pass configuration options to the `influxdb serve` server using either command -options or environment variables. Command options take precedence over -environment variables. +Pass configuration options using either command options or environment variables. +Command options take precedence over environment variables. -##### Example `influxdb3 serve` command options +### Global vs serve-specific options +Some options are **global** (specified before the command) while others are **serve-specific** (specified after `serve`): + +- **Global options**: Apply to the `influxdb3` CLI itself (for example, `--num-io-threads`) +- **Serve options**: Apply only to the `serve` command (for example, `--node-id`, `--object-store`, `--verbose`) + +#### Example command with global and serve-specific options + +{{% show-in "core" %}} ```sh -influxdb3 serve \ +influxdb3 --num-io-threads=4 serve \ --node-id node0 \ -{{% show-in "enterprise" %}} --cluster-id cluster0 \ - --license-email example@email.com \{{% /show-in %}} --object-store file \ --data-dir ~/.influxdb3 \ + --verbose \ --log-filter info ``` +{{% /show-in %}} + +{{% show-in "enterprise" %}} + + +```sh +influxdb3 --num-io-threads=4 serve \ + --node-id node0 \ + --cluster-id cluster0 \ + --license-email example@email.com \ + --object-store file \ + --data-dir ~/.influxdb3 \ + --verbose \ + --log-filter info +``` +{{% /show-in %}} ##### Example environment variables @@ -37,6 +59,30 @@ export LOG_FILTER=info influxdb3 serve ``` +## Global configuration options + +The following options apply to the `influxdb3` CLI globally and must be specified **before** any subcommand (for example, `serve`): + +### num-io-threads + +Sets the number of threads allocated to the IO runtime thread pool. IO threads handle HTTP request serving, line protocol parsing, and file operations. + +> [!Important] +> This is a **global option** that must be specified before the `serve` command. + +```bash +# Set IO threads (global option before serve) +influxdb3 --num-io-threads=8 serve --node-id=node0 --object-store=file +``` + +For detailed information about thread allocation, see the [Resource Limits](#resource-limits) section. + +| influxdb3 option | Environment variable | +| :--------------- | :------------------------- | +| `--num-io-threads` | `INFLUXDB3_NUM_IO_THREADS` | + +--- + ## Server configuration options - [General](#general) @@ -104,14 +150,16 @@ influxdb3 serve - [traces-jaeger-max-msgs-per-second](#traces-jaeger-max-msgs-per-second) - [DataFusion](#datafusion) - [datafusion-num-threads](#datafusion-num-threads) - - [datafusion-runtime-type](#datafusion-runtime-type) - - [datafusion-runtime-disable-lifo-slot](#datafusion-runtime-disable-lifo-slot) - - [datafusion-runtime-event-interval](#datafusion-runtime-event-interval) - - [datafusion-runtime-global-queue-interval](#datafusion-runtime-global-queue-interval) - - [datafusion-runtime-max-blocking-threads](#datafusion-runtime-max-blocking-threads) - - [datafusion-runtime-max-io-events-per-tick](#datafusion-runtime-max-io-events-per-tick) - - [datafusion-runtime-thread-keep-alive](#datafusion-runtime-thread-keep-alive) - - [datafusion-runtime-thread-priority](#datafusion-runtime-thread-priority) + - [datafusion-max-parquet-fanout](#datafusion-max-parquet-fanout) - [datafusion-use-cached-parquet-loader](#datafusion-use-cached-parquet-loader) - [datafusion-config](#datafusion-config) @@ -223,7 +271,7 @@ Required when using the `file` [object store](#object-store). {{% show-in "enterprise" %}} #### mode -Sets the mode to start the server in. +Sets the mode to start the server in, allowing you to create specialized nodes in a distributed cluster. This option supports the following values: @@ -237,6 +285,32 @@ You can specify multiple modes using a comma-delimited list (for example, `inges **Default:** `all` +> [!Important] +> **Thread allocation for different modes:** +> +> - **Ingest mode**: Benefits from additional IO threads for line protocol parsing. For high-throughput +> scenarios with multiple concurrent writers, consider increasing [`--num-io-threads`](#num-io-threads) (global option) +> to 8-16+ to optimize performance. DataFusion threads are still needed for snapshot operations. +> +> - **Query mode**: Benefits from maximizing DataFusion threads. Use most available cores for DataFusion +> with minimal IO threads (2-4). +> +> - **Compact mode**: Primarily uses DataFusion threads for sort/dedupe operations. +> +> - **All mode**: Requires balanced thread allocation based on your workload mix. + +**Example configurations:** +```bash +# High-throughput ingest node (32 cores) +influxdb3 --num-io-threads=12 serve --mode=ingest --num-datafusion-threads=20 + +# Query-optimized node (32 cores) +influxdb3 --num-io-threads=4 serve --mode=query --num-datafusion-threads=28 + +# Balanced all-in-one (32 cores) +influxdb3 --num-io-threads=6 serve --mode=all --num-datafusion-threads=26 +``` + | influxdb3 serve option | Environment variable | | :--------------------- | :-------------------------- | | `--mode` | `INFLUXDB3_ENTERPRISE_MODE` | @@ -1126,102 +1200,25 @@ Sets the maximum number of DataFusion runtime threads to use. --- -#### datafusion-runtime-type + --- @@ -1871,28 +1868,68 @@ Specifies how long to wait for a running ingestor during startup. - [num-cores](#num-cores) +{{% show-in "enterprise" %}}- [num-datafusion-threads](#num-datafusion-threads){{% /show-in %}} +- _[num-io-threads](#num-io-threads) - See [Global configuration options](#global-configuration-options)_ - [num-database-limit](#num-database-limit) - [num-table-limit](#num-table-limit) - [num-total-columns-per-table-limit](#num-total-columns-per-table-limit) #### num-cores +{{% show-in "enterprise" %}} Limits the number of CPU cores that the InfluxDB 3 Enterprise process can use when running on systems where resources are shared. When specified, InfluxDB automatically assigns the number of DataFusion threads and IO threads based on the core count. -**Thread assignment logic:** +**Default thread assignment logic when `num-cores` is set:** - **1-2 cores**: 1 IO thread, 1 DataFusion thread -- **3 cores**: 1 IO thread, 2 DataFusion threads +- **3 cores**: 1 IO thread, 2 DataFusion threads - **4+ cores**: 2 IO threads, (n-2) DataFusion threads +> [!Note] +> You can override the automatic thread assignment by explicitly setting [`--num-io-threads`](#num-io-threads) (global option) +> and [`--num-datafusion-threads`](#num-datafusion-threads). +> This is particularly important for specialized +> workloads like [ingest mode](#mode) where you may need more IO threads than the default allocation. + **Constraints:** - Must be at least 2 - Cannot exceed the number of cores available on the system -- Total thread count from other thread options cannot exceed the `num-cores` value +- Total thread count from `--num-io-threads` (global option) and `--num-datafusion-threads` cannot exceed the `num-cores` value | influxdb3 serve option | Environment variable | | :--------------------- | :-------------------------------- | | `--num-cores` | `INFLUXDB3_ENTERPRISE_NUM_CORES` | +{{% /show-in %}} + +{{% show-in "enterprise" %}} + +> [!Note] +> The [`--num-io-threads`](#num-io-threads) option is a global flag. +--- + +#### num-datafusion-threads + +Sets the number of threads allocated to the DataFusion runtime thread pool. DataFusion threads handle: +- Query execution and processing +- Data aggregation and transformation +- Snapshot creation (sort/dedupe operations) +- Parquet file generation + +**Default behavior:** +- If not specified and `--num-cores` is not set: All available cores minus IO threads +- If not specified and `--num-cores` is set: Automatically determined based on core count (see [`--num-cores`](#num-cores)) + +> [!Note] +> DataFusion threads are used for both query processing and snapshot operations. +> Even ingest-only nodes use DataFusion threads during WAL snapshot creation. + +**Constraints:** +- When used with `--num-cores`, the sum of `--num-io-threads` and `--num-datafusion-threads` cannot exceed the `num-cores` value + +| influxdb3 serve option | Environment variable | +| :----------------------------- | :-------------------------------------- | +| `--num-datafusion-threads` | `INFLUXDB3_NUM_DATAFUSION_THREADS` | +{{% /show-in %}} ---