Adds a metric to track total retried catalog operations due to the catalog
being updated elsewhere. Includes a test to check the counter increments
on basic catalog operations.
Catalog update APIs were returning an Option that was not necessary. It
was always Some, so this removes the Option from the API to make the
intent clear - if there is an update made by the requested change, there
is a Batch; whereas if the requested change is erroneous, or would not
produce a change, the response is Err.
* feat: enable auth by default
- Removes `--bearer-token` support and starts the server with auth by
default.
- Adds `--without-auth` switch to start the server without any auth
* feat: changes for auth being turned off
when auth is turned off,
- disallow token endpoints (returns 405)
- remove hash column when querying tokens system table
* refactor: address PR feedback
This commit allows deletion of tokens by name. Below is an example,
`influxdb3 delete token --token-name _admin --token $CURRENT_ADMIN_TOKEN`
It needs user confirmation before proceeding with the delete
This commit adds TLS support to influxdb3 and allows users to pass in a
path to a key and cert file with the --tls-key and --tls-cert flags in
the serve command. It also adds the ability for every command to specify
a certificate authority for requests. This is mostly needed when the
cert is self signed, but there are other use cases for this.
The big thing is that most of our tests now use TLS by default. Included
are self signed certs for localhost and the the CA cert included in the
commit. Since these are *only* used for testing this should be fine to
include as they are not used in nor are they intended to be used in any
production system. The expiry has been set for 365 days and the file
perms are set to o600 like the original issue mentioned. The tests pass
with this restriction.
I've verified that the API works via curl with the self signed certs as
I did *not* need to pass in the -k option to bypass checking the certs
were valid. The same goes for our tests. They use the rootCA.pem file
to verify the self signed cert when connecting and reject it otherwise.
With this users can be confident that their queries are safely encrypted
during transport.
Note that TLS works for both FlightSQL and our normal APIs.
Closes#25774
This changes the help messages for the serve command and the influxdb3
top level help output. This is to provide better help output and options
for users compared to the clap defaults. After many iterations this code
was the final design I landed on. In order to make our custom help work
we:
1. Roughly parse the args to check for a given command and help message
2. If there is no subcommand we print out our custom help message for
the top level command
3. If there is a subcommand we only print it out for serve currently
4. --help-all prints a more detailed message
This lets us upgrade our help messages in place over time as the
subcommands themselves have subcommands that also need their own help
messages. For now we only include these two.
We could not use something like `clap-help` or derive the output from
the derived clap struct automatically due to multiple issues, but
suffice to say the main thing was that we could not parse the args into
the `Config` struct and then figure out our help message and `clap`'s
message template function was too underpowered. We also wanted to create
a `help-all` flag and this was quite hard to make work with `clap`'s
short and long help.
In the end the best option was to roll our own. While a bit hacky in
terms of us maintaining it, the overall outcome should be a much nicer
user experience in which they can have much better introductions to the
tool. `serve` was quite egregious with a lot of output and very little
actionable options to use for most users.
Closes#26201
* feat: generate persistable admin token
- this commit allows admin token creation using `influxdb3 create token
--admin` and also allows regeneration of admin token by `influxdb3
create token --admin --regenerate`
- `influxdb3_authz` crate hosts all low level token types and behaviour
- catalog log and snapshot types updated to use the token repo
- tests that relied on auth have been updated to use the new token
generation mechanism and new admin token generation/regeneration tests
have been added
* feat: list admin tokens
- allows listing admin tokens
- uses _internal db for token system table
- mostly test fixes due to _internal db
* refactor: make ShutdownManager Clone
ShutdownManager can be clone since its underlying types from tokio are
all shareable via clone.
* refactor: make ShutdownToken not Clone
Alters the API so that the ShutdownToken is not cloneable. This will help
ensure that the Drop implementation is invoked from the correct place.
* chore: couple of updates to fix cargo audit job
- remove humantime ignore in deny.toml
- update pyo3 to use 0.24.1 (https://rustsec.org/advisories/RUSTSEC-2025-0020.html)
* chore: moved pyo3 version to root cargo.toml
Added an additional check in the serve command to ensure that the frontend
has shutdown before exiting so that we don't close any connections pre-
emptively.
* feat: trigger shutdown if wal has been overwritten
WAL persist uses PutMode::Create in order to invoke shutdown if another
process writes to the WAL ahead of it.
A test was added to check that it works from CLI test suite.
* chore: clippy
This ensures a ShutdownToken will invoke `complete` by calling it from
its `Drop` implementation. This means registered components are not
required to signal completion, but can if needed.
Some comments and other cleanup refactoring was done as well.
* feat: add influxdb3_shutdown crate
provides basic wait methods for unix/windows OS's
* feat: graceful shutdown
* docs: add rust docs and test to influxdb3_shutdown
Added rustdoc comments to types and methods in the influxdb3_shutdown
crate as well as a test that shows the ordering of a shutdown.
This introduces a new version for the catalog file formats ([snapshot](3b57682214/influxdb3_catalog/src/snapshot/versions/mod.rs (L2)) files and [log](3b57682214/influxdb3_catalog/src/log/versions/mod.rs (L2)) files). The reason for introducing a new version is to change the serialization/deserialization format from [`bitcode`](https://docs.rs/bitcode/latest/bitcode/) to JSON. See #26180.
The approach taken was to copy the existing type definitions for both log and snapshot files into two places: a `v1` module and a `v2` module. Going forward:
* Types defined in `v1` should not be changed. They are only there to enable deserialization of existing bitcode-serialized catalog files.
* Types defined in `v2` can be modified in a backward-compatible manor, and new types can be added to the `v2` modules
With this PR, old files are not overwritten. The server does not migrate any files on startup. See https://github.com/influxdata/influxdb/pull/26183#issuecomment-2748238191Closes#26180
* simplify FieldValue types by making load generator functions should be
generic over RngCore and passing the RNG in to methods rather than
depending on it being available on every type instance that needs it
* expose influxdb3_load_generator as library crate
* export config, spec, and measurement types publicly to suppore use in
the antithesis-e2e crate
* fix bug that surfaced whenever the cardinality value was less than the
lines per sample value by forcing LP lines in a set of samples to be
distinct from one another with nanosecond increments
This adds a sleep so that the parquet cache has a little bit of time to
populate before we make another request to the query buffer. Sometimes
it does not populate and so we have a race condition where the new
request comes in and actually goes to object store. This is fine in
practice because it would also take time to fill the cache in production
as well. I haven't really seen the test fail since adding this, but
triggering it in the first place is really hard and in practice does not
happen all that often.
When starting up a new cluster in Enterprise we might have multiple
nodes starting at the same time. We might have an issue wherby we have
multiple catalogs with different UUIDs in their in memory
representation.
For example:
- Let's say we have node0 and node1
- node0 and node1 start at the same time and both check object storage
to see if there is a catalog to load
- They both see there is no catalog
- They both create a new one by generating a UUID and persisting it to
object storage
- Whichever is written second is now the one with the correct UUID in
their in memory representation while the other will not have the
correct one until restarted likely
This in practice isn't an issue today as Trevor notes in
https://github.com/influxdata/influxdb_pro/issues/600, but it could be
once we start using `--cluster-id` for licensing purposes. In order to
prevent this we instead make the write to object storage use the Put
mode. If it exists then the write will fail and the node that lost the
race will instead just load the other's catalog.
For example if node1 wins the race then node0 will load the catalog
created by node1 and use that UUID instead.
As this is hard to create a test for as it involves a race condition to
happen I have not included one as we could never really be sure it was
taken care of and we rely on the underlying object store we are writing
to to handle this for us. It's also not likely to happen given this is
only on a new cluster being initiated for the first time decreasing the
chances of it occurring in the first place.
This creates a CatalogUpdateMessage type that is used to send
CatalogUpdates; this type performs the send on the oneshot Sender so
that the consumer of the message does not need to do so.
Subscribers to the catalog get a CatalogSubscription, which uses the
CatalogUpdateMessage type to ACK the message broadcast from the catalog.
This means that catalog message broadcast can fail, but this commit does
not provide any means of rolling back a catalog update.
A test was added to check that it works.
Trevor Hilton
praveen-influx
Paul Dix
Michael Gattozzi
Jamie Strandboge
Jackson Newhouse
wayne
peterbarnett03