c91d556edf
This commit brings over `TableIndexCache` support from the enterprise
repo. It primarily focuses on efficient automatic cleanup of expired
gen1 parquet files based on retention policies and hard deletes. It
- Adds purge operations for tables and retention period expired data.
- Integrates `TableIndexCache` into `PersistedFiles` for the sake of
parquet data deletion handling in `ObjectDeleter` impl.
- Introduces a new background loop for applying data retention polices
with a 30m default interval.
- Includes comprehensive test coverage for cache operations, concurrent
access, persisted snapshot to table index snapshot splits, purge
scenario, object store path parsing, etc.
\## New Types
- `influxdb3_write::table_index::TableIndex`:
- A new trait that tracks gen1 parquet file metadata on a per-table
basis.
- `influxdb3_write::table_index::TableIndexSnapshot`:
- An incremental snapshot of added and removed gen1 parquet files.
- Created by splitting a `PersistedSnapshot` (ie a whole-database
snapshot) into individual table snapshots.
- Uses the existing snapshot sequence number.
- Removed from object store after successful aggregation into
`CoreTableIndex`.
- `influxdb3_write::table_index::CoreTableIndex`:
- Implements of `TableIndex` trait.
- Aggregation of `TableIndexSnapshot`s.
- Not versioned -- assumes that we will migrate away from Parquet in
favor of PachaTree in the medium/long term.
- `influxdb3_write::table_index_cache::TableIndexCache`
- LRU cache
- Configurable via CLI parameters:
- Concurrency of object store operations.
- Maximum number of `CachedTableIndex` to allow before evicting
oldest entries.
- Entrypoint for handling conversion of `PersistedSnapshot` to
`TableIndexSnapshot` to `TableIndex`
- `influxdb3_write::table_index_cache::CachedTableIndex`
- Implements `TableIndex` trait
- Accessing ParquetFile or TableIndex causes last access time to be
updated.
- Stores a mutable `CoreTableIndex` as implementation detail.
- `influxdb3_write::retention_period_handler::RetentionPeriodHandler`
- Runs a top-level background task that periodically applies retention
periods to gen1 files via the `TableIndexCache`.
- Configurable via CLI parameters:
- Retention period handling interval
\## Updated Types
- `influxdb3_write::persisted_files::PersistedFiles`
- Now holds an `Arc` reference to `TableIndexCache`
- Uses its `TableIndexCache` to apply hard deletion to all historical
gen1 files and update associated `CoreTableIndex` in the object
store.
|
||
|---|---|---|
| .cargo | ||
| .circleci | ||
| .github | ||
| assets | ||
| docker | ||
| influxdb3 | ||
| influxdb3_authz | ||
| influxdb3_cache | ||
| influxdb3_catalog | ||
| influxdb3_clap_blocks | ||
| influxdb3_client | ||
| influxdb3_id | ||
| influxdb3_internal_api | ||
| influxdb3_load_generator | ||
| influxdb3_process | ||
| influxdb3_processing_engine | ||
| influxdb3_py_api | ||
| influxdb3_server | ||
| influxdb3_shutdown | ||
| influxdb3_sys_events | ||
| influxdb3_telemetry | ||
| influxdb3_test_helpers | ||
| influxdb3_types | ||
| influxdb3_wal | ||
| influxdb3_write | ||
| iox_query_influxql_rewrite | ||
| .editorconfig | ||
| .gitattributes | ||
| .gitignore | ||
| .kodiak.toml | ||
| CONTRIBUTING.md | ||
| Cargo.lock | ||
| Cargo.toml | ||
| Dockerfile | ||
| Dockerfile.dockerignore | ||
| LICENSE-APACHE | ||
| LICENSE-MIT | ||
| PROFILING.md | ||
| README.md | ||
| README_processing_engine.md | ||
| RELEASE.md | ||
| SECURITY.md | ||
| deny.toml | ||
| install_influxdb.sh | ||
| run-tests.sh | ||
| rust-toolchain.toml | ||
| rustfmt.toml | ||
README.md
InfluxDB Core is a database built to collect, process, transform, and store event and time series data. It is ideal for use cases that require real-time ingest and fast query response times to build user interfaces, monitoring, and automation solutions.
Common use cases include:
- Monitoring sensor data
- Server monitoring
- Application performance monitoring
- Network monitoring
- Financial market and trading analytics
- Behavioral analytics
InfluxDB is optimized for scenarios where near real-time data monitoring is essential and queries need to return quickly to support user experiences such as dashboards and interactive user interfaces.
InfluxDB 3 Core’s feature highlights include:
- Diskless architecture with object storage support (or local disk with no dependencies)
- Fast query response times (under 10ms for last-value queries, or 30ms for distinct metadata)
- Embedded Python VM for plugins and triggers
- Parquet file persistence
- Compatibility with InfluxDB 1.x and 2.x write APIs
- Compatability with InfluxDB 1.x query API (InfluxQL)
- SQL query engine with support for FlightSQL and HTTP query API
Project Status
InfluxDB 3 Core is GA as of April 15, 2025! We plan to have monthly point releases for the following six months, with patch releases as needed. We will move to a quarterly cadence after that for 3-4 releases, after which we'll reevaluate our release schedule.
Join the InfluxDB3 Discord or the public channels below to share your feedback, feature requests, and bug reports.
See the InfluxDB 3 Core & Enterprise GA release announcement here or dig into the InfluxDB 3 getting started guide here.
Learn InfluxDB
Documentation | Community Forum | Community Slack | Blog | InfluxDB University | YouTube
Try InfluxDB Cloud for free and get started fast with no local setup required. Click here to start building your application on InfluxDB Cloud.
Installation
We have nightly and versioned Docker images, Debian packages, RPM packages, and tarballs of InfluxDB available on the InfluxData downloads page. We also provide the InfluxDB command line interface (CLI) client as a separate binary available at the same location.
- For v1 installation, use the main 1.x branch or install InfluxDB OSS directly.
- For v2 installation, use the main 2.x branch.
- For InfluxDB 3 Core see the InfluxDB 3 Core getting started guide.
- For InfluxDB 3 Enterprise see the InfluxDB 3 Enterprise getting started guide.
If you are interested in building from source, see the building from source guide for contributors.
To begin using InfluxDB, visit our Getting Started with InfluxDB documentation.
License
The open source software we build is licensed under the permissive MIT or Apache 2 licenses at the user's choosing. We’ve long held the view that our open source code should be truly open and our commercial code should be separate and closed.
Interested in joining the team building InfluxDB?
Check out current job openings at www.influxdata.com/careers today!