03ea565802
This allows the `max_parquet_fanout` to be specified in the CLI for the `influxdb3 serve` command. This could be done previously via the `--datafusion-config` CLI argument, but the drawbacks to that were:
1. that is a fairly advanced option given the available key/value pairs are not well documented
2. if `iox.max_parquet_fanout` was not provided to that argument, the default would be set to `40`
This PR maintains the existing `--datafusion-config` CLI argument (with one caveat, see below) which allows users to provide a set key/value pairs that will be used to build the internal DataFusion config, but in addition provides the `--datafusion-max-parquet-fanout` argument:
```
--datafusion-max-parquet-fanout <MAX_PARQUET_FANOUT>
When multiple parquet files are required in a sorted way (e.g. for de-duplication), we have two options:
1. **In-mem sorting:** Put them into `datafusion.target_partitions` DataFusion partitions. This limits the fan-out, but requires that we potentially chain multiple parquet files into a single DataFusion partition. Since chaining sorted data does NOT automatically result in sorted data (e.g. AB-AB is not sorted), we need to preform an in-memory sort using `SortExec` afterwards. This is expensive. 2. **Fan-out:** Instead of chaining files within DataFusion partitions, we can accept a fan-out beyond `target_partitions`. This prevents in-memory sorting but may result in OOMs (out-of-memory) if the fan-out is too large.
We try to pick option 2 up to a certain number of files, which is configured by this setting.
[env: INFLUXDB3_DATAFUSION_MAX_PARQUET_FANOUT=]
[default: 1000]
```
with the default value of `1000`, which will override the core `iox_query` default of `40`.
A test was added to check that this is propagated down to the `IOxSessionContext` that is used during queries.
The only change to the `datafusion-config` CLI argument was to rename `INFLUXDB_IOX` in the environment variable to `INFLUXDB3`:
```
--datafusion-config <DATAFUSION_CONFIG>
Provide custom configuration to DataFusion as a comma-separated list of key:value pairs.
# Example ```text --datafusion-config "datafusion.key1:value1, datafusion.key2:value2" ```
[env: INFLUXDB3_DATAFUSION_CONFIG=]
[default: ]
```
|
||
|---|---|---|
| .cargo | ||
| .circleci | ||
| .github | ||
| assets | ||
| docker | ||
| influxdb3 | ||
| influxdb3_cache | ||
| influxdb3_catalog | ||
| influxdb3_clap_blocks | ||
| influxdb3_client | ||
| influxdb3_id | ||
| influxdb3_load_generator | ||
| influxdb3_process | ||
| influxdb3_py_api | ||
| influxdb3_server | ||
| influxdb3_sys_events | ||
| influxdb3_telemetry | ||
| influxdb3_test_helpers | ||
| influxdb3_wal | ||
| influxdb3_write | ||
| iox_query_influxql_rewrite | ||
| .editorconfig | ||
| .gitattributes | ||
| .gitignore | ||
| .kodiak.toml | ||
| CONTRIBUTING.md | ||
| Cargo.lock | ||
| Cargo.toml | ||
| Dockerfile | ||
| Dockerfile.dockerignore | ||
| LICENSE-APACHE | ||
| LICENSE-MIT | ||
| PROFILING.md | ||
| README.md | ||
| SECURITY.md | ||
| deny.toml | ||
| run-tests.sh | ||
| rust-toolchain.toml | ||
| rustfmt.toml | ||
README.md
InfluxDB is the leading open source time series database for metrics, events, and real-time analytics.
Project Status
This main branch contains InfluxDB v3 in pre-release and under active development. Build artifacts are not yet generally available and official installation instructions will be coming later this year. For now, a Dockerfile is provided and can be adapted or used for inspiration by intrepid users.
Learn InfluxDB
Documentation | Community Forum | Community Slack | Blog | InfluxDB University | YouTube
Try InfluxDB Cloud for free and get started fast with no local setup required. Click here to start building your application on InfluxDB Cloud.
Installation
We have nightly and versioned Docker images, Debian packages, RPM packages, and tarballs of InfluxDB available on the InfluxData downloads page. We also provide the InfluxDB command line interface (CLI) client as a separate binary available at the same location.
- For v1 installation, use the main 1.x branch or install InfluxDB OSS directly.
- For v2 installation, use the main 2.x branch.
- v3 development is on this main branch. This project is actively under development and is not considered stable.
If you are interested in building from source, see the building from source guide for contributors.
To begin using InfluxDB, visit our Getting Started with InfluxDB documentation.
License
The open source software we build is licensed under the permissive MIT and Apache 2 licenses. We’ve long held the view that our open source code should be truly open and our commercial code should be separate and closed.
Interested in joining the team building InfluxDB?
Check out current job openings at www.influxdata.com/careers today!