3265960010
* feat: refactor WAL and WriteBuffer There is a ton going on here, but here are the high level things. This implements a new WAL, which is backed entirely by object store. It then updates the WriteBuffer to be able to work with how the new WAL works, which also required an update to how the Catalog is modified and persisted. The concept of Segments has been removed. Previously there was a separate WAL per segment of time. Instead, there is now a single WAL that all writes and updates flow into. Data within the write buffer is organized by Chunk(s) within tables, which is based on the timestamp of the row data. These are known as the Level0 files, which will be persisted as Parquet into object store. The default chunk duration for level 0 files is 10 minutes. The WAL is written as single files that get created at the configured WAL flush interval (1s by default). After a certain number of files have been created, the server will attempt to snapshot the WAL (default is to snapshot the first 600 files of the WAL after we have 900 total, i.e. snapshot 10 minutes of WAL data). The design goal with this is to persist 10 minute chunks of data that are no longer receiving writes, while clearing out old WAL files. This works if data getting written in around "now" with no more than 5 minutes of delay. If we continue to have delayed writes, a snapshot of all data will be forced in order to clear out the WAL and free up memory in the buffer. Overall, this structure of a single wal, with flushes and snapshots and chunks in the queryable buffer led to a simpler setup for the write buffer overall. I was able to clear out quite a bit of code related to the old segment organization. Fixes #25142 and fixes #25173 * refactor: address PR feedback * refactor: wal to replay and background flush on new * chore: remove stray println |
||
|---|---|---|
| .cargo | ||
| .circleci | ||
| .github | ||
| assets | ||
| docker | ||
| influxdb3 | ||
| influxdb3_client | ||
| influxdb3_load_generator | ||
| influxdb3_process | ||
| influxdb3_server | ||
| influxdb3_wal | ||
| influxdb3_write | ||
| iox_query_influxql_rewrite | ||
| .editorconfig | ||
| .gitattributes | ||
| .gitignore | ||
| .kodiak.toml | ||
| CONTRIBUTING.md | ||
| Cargo.lock | ||
| Cargo.toml | ||
| Dockerfile | ||
| Dockerfile.dockerignore | ||
| LICENSE-APACHE | ||
| LICENSE-MIT | ||
| PROFILING.md | ||
| README.md | ||
| SECURITY.md | ||
| deny.toml | ||
| rust-toolchain.toml | ||
| rustfmt.toml | ||
README.md
InfluxDB is the leading open source time series database for metrics, events, and real-time analytics.
Project Status
This main branch contains InfluxDB v3 in pre-release and under active development. Build artifacts are not yet generally available and official installation instructions will be coming later this year. For now, a Dockerfile is provided and can be adapted or used for inspiration by intrepid users.
Learn InfluxDB
Documentation | Community Forum | Community Slack | Blog | InfluxDB University | YouTube
Try InfluxDB Cloud for free and get started fast with no local setup required. Click here to start building your application on InfluxDB Cloud.
Installation
We have nightly and versioned Docker images, Debian packages, RPM packages, and tarballs of InfluxDB available on the InfluxData downloads page. We also provide the InfluxDB command line interface (CLI) client as a separate binary available at the same location.
- For v1 installation, use the main 1.x branch or install InfluxDB OSS directly.
- For v2 installation, use the main 2.x branch.
- v3 development is on this main branch. This project is actively under development and is not considered stable.
If you are interested in building from source, see the building from source guide for contributors.
To begin using InfluxDB, visit our Getting Started with InfluxDB documentation.
License
The open source software we build is licensed under the permissive MIT and Apache 2 licenses. We’ve long held the view that our open source code should be truly open and our commercial code should be separate and closed.
Interested in joining the team building InfluxDB?
Check out current job openings at www.influxdata.com/careers today!