This commit brings over `TableIndexCache` support from the enterprise
repo. It primarily focuses on efficient automatic cleanup of expired
gen1 parquet files based on retention policies and hard deletes. It
- Adds purge operations for tables and retention period expired data.
- Integrates `TableIndexCache` into `PersistedFiles` for the sake of
parquet data deletion handling in `ObjectDeleter` impl.
- Introduces a new background loop for applying data retention polices
with a 30m default interval.
- Includes comprehensive test coverage for cache operations, concurrent
access, persisted snapshot to table index snapshot splits, purge
scenario, object store path parsing, etc.
\## New Types
- `influxdb3_write::table_index::TableIndex`:
- A new trait that tracks gen1 parquet file metadata on a per-table
basis.
- `influxdb3_write::table_index::TableIndexSnapshot`:
- An incremental snapshot of added and removed gen1 parquet files.
- Created by splitting a `PersistedSnapshot` (ie a whole-database
snapshot) into individual table snapshots.
- Uses the existing snapshot sequence number.
- Removed from object store after successful aggregation into
`CoreTableIndex`.
- `influxdb3_write::table_index::CoreTableIndex`:
- Implements of `TableIndex` trait.
- Aggregation of `TableIndexSnapshot`s.
- Not versioned -- assumes that we will migrate away from Parquet in
favor of PachaTree in the medium/long term.
- `influxdb3_write::table_index_cache::TableIndexCache`
- LRU cache
- Configurable via CLI parameters:
- Concurrency of object store operations.
- Maximum number of `CachedTableIndex` to allow before evicting
oldest entries.
- Entrypoint for handling conversion of `PersistedSnapshot` to
`TableIndexSnapshot` to `TableIndex`
- `influxdb3_write::table_index_cache::CachedTableIndex`
- Implements `TableIndex` trait
- Accessing ParquetFile or TableIndex causes last access time to be
updated.
- Stores a mutable `CoreTableIndex` as implementation detail.
- `influxdb3_write::retention_period_handler::RetentionPeriodHandler`
- Runs a top-level background task that periodically applies retention
periods to gen1 files via the `TableIndexCache`.
- Configurable via CLI parameters:
- Retention period handling interval
\## Updated Types
- `influxdb3_write::persisted_files::PersistedFiles`
- Now holds an `Arc` reference to `TableIndexCache`
- Uses its `TableIndexCache` to apply hard deletion to all historical
gen1 files and update associated `CoreTableIndex` in the object
store.
This commit touches quite a few things, but the main changes that need
to be taken into account are:
- An update command has been added to the CLI. This could be further
extended in the future to update more than just Database retention
periods. The API call for that has been written in such a
way as to allow other qualities of the database to be updated
at runtime from one API call. For now it only allows the retention
period to be updated, but it could in theory allow us to rename a
database without needing to wipe things, especially with a stable ID
underlying everything.
- The create database command has been extended to allow
its creation with a retention period. In tandem with the update
command users can now assign or delete retention periods at will
- The ability to query catalog data about both databases and tables has
been added as well. This has been used in tests added in this commit,
but is also a fairly useful query when wanting to look at things such
as the series key. This could be extended to a CLI command as well if
we want to allow users to look at this data, but for now it's in the
_internal table.
With these changes a nice UX has been created to allow our customers to
work with retention periods.
* refactor: Use iox_http_util::Request instead of hyper::Request
* refactor: Use iox_http_util::RequestBuilder instead of hyper::Request::builder
* refactor: Use iox_http_util::empty_request_body instead of Body::empty
* refactor: Use iox_http_util::bytes_to_request_body instead of Body::from
* refactor: Use http_body::Body instead of hyper::body::HttpBody
* refactor: Use iox_http_util::Response instead of hyper::Response
* refactor: Use iox_http_util::ResponseBuilder instead of hyper::Response::builder
* refactor: Use iox_http_util::empty_response_body instead of Body::empty
* refactor: Use iox_http_util::bytes_to_response_body instead of Body::from
* refactor: Use iox_http_util::stream_results_to_response_body instead of Body::wrap_stream
* refactor: Use the read_body_bytes_for_tests helper fn
* chore: update to latest core
* chore: allow CDLA permissive 2 license
* chore: update insta snapshot for new internal df tables
* test: update assertion in flightsql test
* fix: object store size hinting workaround in clap_blocks
Applied a workaround from upstream to strip size hinting from the object
store get request options. See:
https://github.com/influxdata/influxdb_iox/issues/13771
* fix: query_executor tests use object store size hinting workaround
* fix: insta snapshot test for show system summary command
* chore: update windows- crates for advisories
* chore: update to latest sha on influxdb3_core branch
* chore: update to latest influxdb3_core rev
* refactor: pr feedback
* refactor: do not use object store size hint layer
Instead of using the ObjectStoreStripSizeHint layer, just provide the
configuration to datafusion to disable the use of size hinting from
iox_query.
This is used in IOx and not relevant to Monolith.
* fix: use parquet cache for get_opts requests
* test: that the parquet cache is being hit from write buffer
* refactor: remove unused Key type from write buffer
The write buffer had a Key variant for handling the experimental v3
write API that was phased out and removed from an earlier iteration
of influxdb3.
* refactor: remove key column type from last cache
* feat: `/ping` API contains versioning headers
Further, the product version can be modified by updating the metadata in
the `influxdb3_process` `Cargo.toml`.
* chore: PR feedback
* chore: placate linter
This will have revision info displayed with `-nightly` added to the version, which
leads the most recent released version by 1 minor release.
For example,
* Using the `/ping` API:
```
❯ curl localhost:8181/ping
{"version":"3.1.0-nightly","revision":"f6aa0c1b29","process_id":"ccbd5879-a83b-4091-b68c-92c6b704cf8a"}
```
* Checking the binary version:
```
❯ influxdb3 --version
influxdb3 3.1.0-nightly, revision f6aa0c1b29
```
This commit allows users to set a minimum TLS version. The default is
1.2. The choices are TLS 1.2 or TLS 1.3 which can be set via env var:
INFLUXDB3_TLS_MINIMUM_VERSION="tls-1.2"
or
INFLUXDB3_TLS_MINIMUM_VERSION="tls-1.3"
and for the command line flag for the serve command:
--tls-minimum-version tls-1.2
or
--tls-minimum-version tls-1.3
With this users have more fine grained control over what tls version
they require.
Closes#26255
This commit is a follow up to #26246 and generates test certs on the fly
for our test suite. In practice this will only need to be done once with
a fresh repo check out as the certs will expire long after anyone would
reasonably be working on this code in the year 4096! This could be
extended in the future to generate negative tls tests where the file
should be expired.
Closes#26254
This commit adds TLS support to influxdb3 and allows users to pass in a
path to a key and cert file with the --tls-key and --tls-cert flags in
the serve command. It also adds the ability for every command to specify
a certificate authority for requests. This is mostly needed when the
cert is self signed, but there are other use cases for this.
The big thing is that most of our tests now use TLS by default. Included
are self signed certs for localhost and the the CA cert included in the
commit. Since these are *only* used for testing this should be fine to
include as they are not used in nor are they intended to be used in any
production system. The expiry has been set for 365 days and the file
perms are set to o600 like the original issue mentioned. The tests pass
with this restriction.
I've verified that the API works via curl with the self signed certs as
I did *not* need to pass in the -k option to bypass checking the certs
were valid. The same goes for our tests. They use the rootCA.pem file
to verify the self signed cert when connecting and reject it otherwise.
With this users can be confident that their queries are safely encrypted
during transport.
Note that TLS works for both FlightSQL and our normal APIs.
Closes#25774
This changes the help messages for the serve command and the influxdb3
top level help output. This is to provide better help output and options
for users compared to the clap defaults. After many iterations this code
was the final design I landed on. In order to make our custom help work
we:
1. Roughly parse the args to check for a given command and help message
2. If there is no subcommand we print out our custom help message for
the top level command
3. If there is a subcommand we only print it out for serve currently
4. --help-all prints a more detailed message
This lets us upgrade our help messages in place over time as the
subcommands themselves have subcommands that also need their own help
messages. For now we only include these two.
We could not use something like `clap-help` or derive the output from
the derived clap struct automatically due to multiple issues, but
suffice to say the main thing was that we could not parse the args into
the `Config` struct and then figure out our help message and `clap`'s
message template function was too underpowered. We also wanted to create
a `help-all` flag and this was quite hard to make work with `clap`'s
short and long help.
In the end the best option was to roll our own. While a bit hacky in
terms of us maintaining it, the overall outcome should be a much nicer
user experience in which they can have much better introductions to the
tool. `serve` was quite egregious with a lot of output and very little
actionable options to use for most users.
Closes#26201
* feat: generate persistable admin token
- this commit allows admin token creation using `influxdb3 create token
--admin` and also allows regeneration of admin token by `influxdb3
create token --admin --regenerate`
- `influxdb3_authz` crate hosts all low level token types and behaviour
- catalog log and snapshot types updated to use the token repo
- tests that relied on auth have been updated to use the new token
generation mechanism and new admin token generation/regeneration tests
have been added
* feat: list admin tokens
- allows listing admin tokens
- uses _internal db for token system table
- mostly test fixes due to _internal db
* chore: couple of updates to fix cargo audit job
- remove humantime ignore in deny.toml
- update pyo3 to use 0.24.1 (https://rustsec.org/advisories/RUSTSEC-2025-0020.html)
* chore: moved pyo3 version to root cargo.toml
* feat: add influxdb3_shutdown crate
provides basic wait methods for unix/windows OS's
* feat: graceful shutdown
* docs: add rust docs and test to influxdb3_shutdown
Added rustdoc comments to types and methods in the influxdb3_shutdown
crate as well as a test that shows the ordering of a shutdown.
This commit restores the old behavior we had where new tags can be added
to a schema. To do this we made tags nullable and brings us in line with
our other products. These changes were made in this PR:
https://github.com/influxdata/influxdb3_core/pull/41.
Changes to accomplish this new behavior were:
- Queries now do not return an empty string for null tags instead they
are returned as null, or in many formats not at all.
- References to v1 for parsing and validating lines were removed as we
only have one path for doing so these days shared amongst all the
write_lp endpoints.
- We fixed failing tests that expected us to not be able to have new
tags or depended on that functionality indirectly
- Tests had their snapshot files updated to reflect that tags are
nullable by default
- Behavior for making a schema and checking whether a column can be null
were updated in a separate repo and integrated here
- The series_key is updated whenever we get a new tag added to the
schema
- New tests were added to show that you can add a new tag and that the
series key is updated as part of that
With the above changes we can now allow tags to be added again by users
like they would expect, especially with v1 and v2 apis and Telegraf
plugins.
The distinct cache info for tables was not serialized in the catalog.
This fixes it, but also updates the catalog serialization to use the
snapshot type serialization from the Catalog type all the way down.
The Eq and PartialEq impls were removed from Catalog and InnerCatalog
as they were only used in tests, and wer replaced by pure insta snapshot
tests.
A test was added to check that the distinct cache serializes/deserializes
Partially fixes https://github.com/influxdata/influxdb/issues/24672
* move most HTTP req/resp types into `influxdb3_types` crate
* removes the use of locally-scoped request type structs from the `influxdb3_client` crate
* fix plugin dependency/package install bug
* it looks like the `DELETE` http method was being used where `POST` was expected for `/api/v3/configure/plugin_environment/install_packages` and `/api/v3/configure/plugin_environment/install_requirements`
* feat(processing_engine): Add cron plugins and triggers to the processing engine.
* feat(processing_engine): switch from 'cron plugin' to 'schedule plugin', use TimeProvider.
* feat(processing_engine): add test for test scheduled plugin.
This updates the v1 /query API hanlder to handle InfluxDB v1's unique
query response structure when GROUP BY clauses are provided.
The distinction is in the addition of a "tags" field to the emitted series
data that contains a map of the GROUP BY tags along with their distinct
values associated with the data in the "values" field.
This required splitting the QueryExecutor into two query paths for InfluxQL
and SQL, as this allowed for handling InfluxQL query parsing in advance
of query planning.
A set of snapshot tests were added to check that it all works.
* feat: Update WAL plugin for new structure
This ended up being a very large change set. In order to get around circular dependencies, the processing engine had to be moved into its own crate, which I think is ultimately much cleaner.
Unfortunately, this required changing a ton of things. There's more testing and things to add on to this, but I think it's important to get this through and build on it.
Importantly, the processing engine no longer resides inside the write buffer. Instead, it is attached to the HTTP server. It is now able to take a query executor, write buffer, and WAL so that the full range of functionality of the server can be exposed to the plugin API.
There are a bunch of system-py feature flags littered everywhere, which I'm hoping we can remove soon.
* refactor: PR feedback
This ended up being a couple things rolled into one. In order to add a query API to the Python plugin, I had to pull the QueryExecutor trait out of server into a place so that the python crate could use it.
This implements the query API, but also fixes up the WAL plugin test CLI a bit. I've added a test in the CLI section so that it shows end-to-end operation of the WAL plugin test API and exercise of the entire Plugin API.
Closes#25757
This makes quite a few major changes to our CLI and how users interact
with it:
1. All commands are now in the form <verb> <noun> this was to make the
commands consistent. We had last-cache as a noun, but serve as a
verb in the top level. Given that we could only create or delete
All noun based commands have been move under a create and delete
command
2. --host short form is now -H not -h which is reassigned to -h/--help
for shorter help text and is in line with what users would expect
for a CLI
3. Only the needed items from clap_blocks have been moved into
`influxdb3_clap_blocks` and any IOx specific references were changed
to InfluxDB 3 specific ones
4. References to InfluxDB 3.0 OSS have been changed to InfluxDB 3 Core
in our CLI tools
5. --dbname has been changed to --database to be consistent with --table
in many commands. The short -d flag still remains. In the create/
delete command for the database however the name of the database is
a positional arg
e.g. `influxbd3 create database foo` rather than
`influxdb3 database create --dbname foo`
6. --table has been removed from the delete/create command for tables
and is now a positional arg much like database
7. clap_blocks was removed as dependency to avoid having IOx specific
env vars
8. --cache-name is now an optional positional arg for last_cache and meta_cache
9. last-cache/meta-cache commands are now last_cache and meta_cache respectively
Unfortunately we have quite a few options to run the software and I
couldn't cut down on them, but at least with this commands and options
will be more discoverable and we have full control over our CLI options
now.
Closes#25646
Prior to this change we would deny writes that used n and u for the
precision argument when doing writes. We only accepted ns and us for
those apis. However, to be backwards compatible we would need to enable
accepting writes with n and u. This is mostly just upgrading our deps
as this was a change that landed in IOx first. We test that it works for
our code by adding test cases for their precision in this repo.
Fixes bug in queryable buffer where if a block of data was missing one of the columns defined in a table sort key, the creation of the logical plan to sort and dedupe the data would fail, causing a panic.
Fixes#25670
* feat: add influxdb3_clap_blocks with runtime config
Added a new workspace crate `influxdb3_clap_blocks` which will be a
starting point for adding InfluxDB 3 OSS/Pro specific CLI configuration
that no longer references IOx, and allows for us to trim out unneeded
configurations for the monolithic InfluxDB 3.
Other than changing references from IOX to INFLUXDB3, this makes one
important change: it enables IO on the DataFusion runtime. This, for now,
is an experimental change to see if we can relieve some concurrency
issues that we have been experiencing.
* chore: add observability deps for windows
This changes the code to reference InfluxDB 3 OSS rather than Edge which
had been it's original name when we first started the project. With this
we now have the code reflect what we are actually calling it. On top of
this the long help text has been changed to give advice about how to
actually run the code now with the bare minimum set of flags needed now
as `influxdb serve` is no longer a viable command on it's own.
Closes#25649
* feat: add startup time to logging output
This change adds a startup time counter to the output when starting up
a server. The main purpose of this is to verify whether the impact of
changes actually speeds up the loading of the server.
* feat: Significantly decrease startup times for WAL
This commit does a few important things to speedup startup times:
1. We avoid changing an Arc<str> to a String with the series key as the
From<String> impl will call with_column which will then turn it into
an Arc<str> again. Instead we can just call `with_column` directly
and pass in the iterator without also collecting into a Vec<String>
2. We switch to using bitcode as the serialization format for the WAL.
This significantly reduces startup time as this format is faster to
use instead of JSON, which was eating up massive amounts of time.
Part of this change involves not using the tag feature of serde as
it's currently not supported by bincode
3. We also parallelize reading and deserializing the WAL files before
we then apply them in order. This reduces time waiting on IO and we
eagerly evaluate each spawned task in order as much as possible.
This gives us about a 189% speedup over what we were doing before.
Closes#25534
In this commit the vec backing the buffer is swapped for an array.
Criterion benchmarks were added to compare the perf to make sure it has
not made it worse. The vec implementation has been removed after the
benchmarks done locally
* feat: core metadata cache structs with basic tests
Implement the base MetaCache type that holds the hierarchical structure
of the metadata cache providing methods to create and push rows from the
WAL into the cache.
Added a prune method as well as a method for gathering record batches
from a meta cache. A test was added to check the latter for various
predicates and that the former works, though, pruning shows that we need
to modify how record batches are produced such that expired entries are
not emitted.
* refactor: filter expired entries and do some clean up in the meta cache
* chore: update core deps
- arrow/parquet deps are patched (as in core)
- three specific code changes to cope with changes in core crates
- TransitionPartitionId, use `from_parts` instead of `new`
- arrow buffers can take &[u8] directly without `to_vec()`/`vec!`
(used only in tests)
- `schema` and `influxdb_line_protocol` crates need `v3` feature enabled
* chore: update deny.toml
* chore: formatting and deny toml changes
Unicode-3.0 license is added to allowed licenses list, without it
end up with 19 errors (`zerovec`, `zerovec-derive` etc.)
* chore: address PR feedback
- move enabling v3 feature to root Cargo.toml
- added the upstream PR for datafusion-common that introduced RUSTSEC-2024-0384
`cargo deny` was showing that no crate matched the advisory criteria for this [RUSTSEC advisory](https://rustsec.org/advisories/RUSTSEC-2024-0376.html), so this PR removes the ignore entry.
In addition, the `hashbrown` crate was causing a new audit failure, and updating it required that the `Zlib` license be added to our list of allowed licenses.
No issue for this, but it is blocking another PR at the moment (https://github.com/influxdata/influxdb/pull/25515).
- Introduced traits, `ParquetMetrics` and `SystemInfoProvider` to enable
writing easier tests
- Uses mockito for code that depends on reqwest::Client and also uses
mockall to generally mock any traits like `SystemInfoProvider`
- Minor updates to docs
Trevor Hilton
wayne
praveen-influx
Michael Gattozzi
Carol (Nichols || Goulding)
Stuart Carnie
Paul Dix
Jackson Newhouse