* fix: do not allow operator token from being deleted
closes: https://github.com/influxdata/influxdb_pro/issues/819
* refactor: address PR feedback
* fix: add a word and clarifying colon
* fix: failing test
---------
Co-authored-by: Peter Barnett <peter.barnett03@gmail.com>
* feat: allow health,ping,metrics to opt out of auth
This commit introduces `--disable-authz <DISABLE_AUTHZ_RESOURCES>`. The
options for `DISABLE_AUTHZ_RESOURCES` are health, ping and metrics. By
default all these resources will be guarded
closes: https://github.com/influxdata/influxdb_pro/issues/774
* chore: update influxdb3/src/commands/helpers.rs
space after comma in help text
Co-authored-by: Trevor Hilton <thilton@influxdata.com>
* chore: update influxdb3/src/help/serve.txt
space after comma in help text
Co-authored-by: Trevor Hilton <thilton@influxdata.com>
* chore: update influxdb3/src/help/serve_all.txt
space after comma in help text
Co-authored-by: Trevor Hilton <thilton@influxdata.com>
* refactor: use statics to reduce clones/copies
---------
Co-authored-by: Trevor Hilton <thilton@influxdata.com>
* feat: support `Basic $TOKEN` for all apis
closes: https://github.com/influxdata/influxdb/issues/25833
* refactor: address PR feedback to return MalformedRequest error when `:` is used more than once in user-pass pair
* refactor: change the message sent back for malformed auth header
This commit adds support for CORS by modifying our requests to make
preflight checks valid and to handle responses containing the necessary
headers for browsers to access the data they need. We keep what we
accept as open as this is essentially what requests to the server are
normally like and we gate the requests with an auth token.
Closes#26313
This commit allows users to set a minimum TLS version. The default is
1.2. The choices are TLS 1.2 or TLS 1.3 which can be set via env var:
INFLUXDB3_TLS_MINIMUM_VERSION="tls-1.2"
or
INFLUXDB3_TLS_MINIMUM_VERSION="tls-1.3"
and for the command line flag for the serve command:
--tls-minimum-version tls-1.2
or
--tls-minimum-version tls-1.3
With this users have more fine grained control over what tls version
they require.
Closes#26255
Adds a metric to track total retried catalog operations due to the catalog
being updated elsewhere. Includes a test to check the counter increments
on basic catalog operations.
Catalog update APIs were returning an Option that was not necessary. It
was always Some, so this removes the Option from the API to make the
intent clear - if there is an update made by the requested change, there
is a Batch; whereas if the requested change is erroneous, or would not
produce a change, the response is Err.
* feat: enable auth by default
- Removes `--bearer-token` support and starts the server with auth by
default.
- Adds `--without-auth` switch to start the server without any auth
* feat: changes for auth being turned off
when auth is turned off,
- disallow token endpoints (returns 405)
- remove hash column when querying tokens system table
* refactor: address PR feedback
This commit allows deletion of tokens by name. Below is an example,
`influxdb3 delete token --token-name _admin --token $CURRENT_ADMIN_TOKEN`
It needs user confirmation before proceeding with the delete
This commit adds TLS support to influxdb3 and allows users to pass in a
path to a key and cert file with the --tls-key and --tls-cert flags in
the serve command. It also adds the ability for every command to specify
a certificate authority for requests. This is mostly needed when the
cert is self signed, but there are other use cases for this.
The big thing is that most of our tests now use TLS by default. Included
are self signed certs for localhost and the the CA cert included in the
commit. Since these are *only* used for testing this should be fine to
include as they are not used in nor are they intended to be used in any
production system. The expiry has been set for 365 days and the file
perms are set to o600 like the original issue mentioned. The tests pass
with this restriction.
I've verified that the API works via curl with the self signed certs as
I did *not* need to pass in the -k option to bypass checking the certs
were valid. The same goes for our tests. They use the rootCA.pem file
to verify the self signed cert when connecting and reject it otherwise.
With this users can be confident that their queries are safely encrypted
during transport.
Note that TLS works for both FlightSQL and our normal APIs.
Closes#25774
* feat: generate persistable admin token
- this commit allows admin token creation using `influxdb3 create token
--admin` and also allows regeneration of admin token by `influxdb3
create token --admin --regenerate`
- `influxdb3_authz` crate hosts all low level token types and behaviour
- catalog log and snapshot types updated to use the token repo
- tests that relied on auth have been updated to use the new token
generation mechanism and new admin token generation/regeneration tests
have been added
* feat: list admin tokens
- allows listing admin tokens
- uses _internal db for token system table
- mostly test fixes due to _internal db
* chore: couple of updates to fix cargo audit job
- remove humantime ignore in deny.toml
- update pyo3 to use 0.24.1 (https://rustsec.org/advisories/RUSTSEC-2025-0020.html)
* chore: moved pyo3 version to root cargo.toml
* feat: add influxdb3_shutdown crate
provides basic wait methods for unix/windows OS's
* feat: graceful shutdown
* docs: add rust docs and test to influxdb3_shutdown
Added rustdoc comments to types and methods in the influxdb3_shutdown
crate as well as a test that shows the ordering of a shutdown.
This creates a CatalogUpdateMessage type that is used to send
CatalogUpdates; this type performs the send on the oneshot Sender so
that the consumer of the message does not need to do so.
Subscribers to the catalog get a CatalogSubscription, which uses the
CatalogUpdateMessage type to ACK the message broadcast from the catalog.
This means that catalog message broadcast can fail, but this commit does
not provide any means of rolling back a catalog update.
A test was added to check that it works.
* refactor: use repository in catalog
The catalog was refactored to use identifiers on everything, and store
everything in a consistent structure. This structure makes use of the
`Repository` type that holds a `SerdeVecMap` of Id to Resource, along
with the next Id, and a bi-map of Id to resource name.
The `Repository` type is used at each level of the catalog where a
resource is stored.
This simplified repeated logic for snapshot'ing, insert and update of
resources in the catalog, as well as accessor methods for getting by id
or name, and mapping names to ids and vice-versa.
In addition, the process for catalog batch verification and permit was
altered so that the permit process induces a retry if the catalog was
updated while the catalog batch function was producing the batch, i.e, if
the catalog sequence incremented while the caller was waiting for a permit.
This eliminated the need for verifying the catalog batch after it had been
generated, and allows for a single path to apply a catalog batch after it
has been persisted to object store.
This assumes that the generation of the catalog batch implies validity.
Irelevant tests were removed.
Last and Distinct cache's now rely more heavily on Ids, though the proc-
essing engine still needs to switch over to use Ids for starting/stopping
triggers.
In #25927 we missed that JSON queries were broken despite having some
tests use the format. This fixes JSON queries such that they now
properly contain a comma between RecordBatches. This commit also
includes tests for the formats that now stream data back (CSV, JSON, and
JSON Lines) so that we won't run into this issue again.
* deduplicate QueryParams->QueryRequest and Format->QueryFormat
* move WriteParams into influxdb3_types crate
* DRY up client HTTP request handling code in *RequestBuilder.send
methods.
* DRY up a bunch of other non-Builder http request handling
Partially fixes https://github.com/influxdata/influxdb/issues/24672
* move most HTTP req/resp types into `influxdb3_types` crate
* removes the use of locally-scoped request type structs from the `influxdb3_client` crate
* fix plugin dependency/package install bug
* it looks like the `DELETE` http method was being used where `POST` was expected for `/api/v3/configure/plugin_environment/install_packages` and `/api/v3/configure/plugin_environment/install_requirements`
This commit allows us to stream data back for CSV and JSON formatted
queries. Prior to this we would buffer up all of the data in memory
before sending it back. Now we can make it so that we only buffer in
one RecordBatch at a time to reduce memory overhead.
Note that due to the way the APIs for writers work and for how Body in
hyper 0.14 works we can't use a streaming body that we can write too.
This in turn means we have to use a manually written Future state
machine, which works but is far from ideal.
Note this does not include the pretty and parquet files as streamable.
I'm attempting to get the pretty one to be streamable, but I don't think
that this one and parquet are as likely to be streamable back to the
user. In general we might want to discourage these formats from being
used.
This updates trigger creation to load the plugin file before creating the trigger.
Another small change is to make Github references use filenames and paths identical to what they would be in the plugin-dir. This makes it a little easier to have the plugins repo local and develop against it and then be able to reference the same file later with gh: once it's up on the repo.
This refactors plugins and triggers so that plugins no longer need to be "created". Since plugins exist in either the configured local directory or on the Github repo, a user now only needs to create a trigger and reference the plugin filename.
Closes#25876
praveen-influx
Trevor Hilton
Michael Gattozzi
Stuart Carnie
Jackson Newhouse
wayne
Paul Dix