* chore: update to latest core
* chore: allow CDLA permissive 2 license
* chore: update insta snapshot for new internal df tables
* test: update assertion in flightsql test
* fix: object store size hinting workaround in clap_blocks
Applied a workaround from upstream to strip size hinting from the object
store get request options. See:
https://github.com/influxdata/influxdb_iox/issues/13771
* fix: query_executor tests use object store size hinting workaround
* fix: insta snapshot test for show system summary command
* chore: update windows- crates for advisories
* chore: update to latest sha on influxdb3_core branch
* chore: update to latest influxdb3_core rev
* refactor: pr feedback
* refactor: do not use object store size hint layer
Instead of using the ObjectStoreStripSizeHint layer, just provide the
configuration to datafusion to disable the use of size hinting from
iox_query.
This is used in IOx and not relevant to Monolith.
* fix: use parquet cache for get_opts requests
* test: that the parquet cache is being hit from write buffer
* chore: Ensure Parquet sort key is serialised with snapshots
* chore: PR feedback, rename state variable to match intent
* chore: Use `Default` trait to implement `TableBuffer::new`
* chore: Fix change in file size with extra metadata
* chore: Add rustdoc for `sort_key` field
* feat: `/ping` API contains versioning headers
Further, the product version can be modified by updating the metadata in
the `influxdb3_process` `Cargo.toml`.
* chore: PR feedback
* chore: placate linter
* fix: do not allow operator token from being deleted
closes: https://github.com/influxdata/influxdb_pro/issues/819
* refactor: address PR feedback
* fix: add a word and clarifying colon
* fix: failing test
---------
Co-authored-by: Peter Barnett <peter.barnett03@gmail.com>
* feat: allow health,ping,metrics to opt out of auth
This commit introduces `--disable-authz <DISABLE_AUTHZ_RESOURCES>`. The
options for `DISABLE_AUTHZ_RESOURCES` are health, ping and metrics. By
default all these resources will be guarded
closes: https://github.com/influxdata/influxdb_pro/issues/774
* chore: update influxdb3/src/commands/helpers.rs
space after comma in help text
Co-authored-by: Trevor Hilton <thilton@influxdata.com>
* chore: update influxdb3/src/help/serve.txt
space after comma in help text
Co-authored-by: Trevor Hilton <thilton@influxdata.com>
* chore: update influxdb3/src/help/serve_all.txt
space after comma in help text
Co-authored-by: Trevor Hilton <thilton@influxdata.com>
* refactor: use statics to reduce clones/copies
---------
Co-authored-by: Trevor Hilton <thilton@influxdata.com>
* feat: support `Basic $TOKEN` for all apis
closes: https://github.com/influxdata/influxdb/issues/25833
* refactor: address PR feedback to return MalformedRequest error when `:` is used more than once in user-pass pair
* refactor: change the message sent back for malformed auth header
This commit adds support for CORS by modifying our requests to make
preflight checks valid and to handle responses containing the necessary
headers for browsers to access the data they need. We keep what we
accept as open as this is essentially what requests to the server are
normally like and we gate the requests with an auth token.
Closes#26313
This commit allows users to set a minimum TLS version. The default is
1.2. The choices are TLS 1.2 or TLS 1.3 which can be set via env var:
INFLUXDB3_TLS_MINIMUM_VERSION="tls-1.2"
or
INFLUXDB3_TLS_MINIMUM_VERSION="tls-1.3"
and for the command line flag for the serve command:
--tls-minimum-version tls-1.2
or
--tls-minimum-version tls-1.3
With this users have more fine grained control over what tls version
they require.
Closes#26255
Adds a metric to track total retried catalog operations due to the catalog
being updated elsewhere. Includes a test to check the counter increments
on basic catalog operations.
Catalog update APIs were returning an Option that was not necessary. It
was always Some, so this removes the Option from the API to make the
intent clear - if there is an update made by the requested change, there
is a Batch; whereas if the requested change is erroneous, or would not
produce a change, the response is Err.
* feat: enable auth by default
- Removes `--bearer-token` support and starts the server with auth by
default.
- Adds `--without-auth` switch to start the server without any auth
* feat: changes for auth being turned off
when auth is turned off,
- disallow token endpoints (returns 405)
- remove hash column when querying tokens system table
* refactor: address PR feedback
This commit allows deletion of tokens by name. Below is an example,
`influxdb3 delete token --token-name _admin --token $CURRENT_ADMIN_TOKEN`
It needs user confirmation before proceeding with the delete
This commit adds TLS support to influxdb3 and allows users to pass in a
path to a key and cert file with the --tls-key and --tls-cert flags in
the serve command. It also adds the ability for every command to specify
a certificate authority for requests. This is mostly needed when the
cert is self signed, but there are other use cases for this.
The big thing is that most of our tests now use TLS by default. Included
are self signed certs for localhost and the the CA cert included in the
commit. Since these are *only* used for testing this should be fine to
include as they are not used in nor are they intended to be used in any
production system. The expiry has been set for 365 days and the file
perms are set to o600 like the original issue mentioned. The tests pass
with this restriction.
I've verified that the API works via curl with the self signed certs as
I did *not* need to pass in the -k option to bypass checking the certs
were valid. The same goes for our tests. They use the rootCA.pem file
to verify the self signed cert when connecting and reject it otherwise.
With this users can be confident that their queries are safely encrypted
during transport.
Note that TLS works for both FlightSQL and our normal APIs.
Closes#25774
* feat: generate persistable admin token
- this commit allows admin token creation using `influxdb3 create token
--admin` and also allows regeneration of admin token by `influxdb3
create token --admin --regenerate`
- `influxdb3_authz` crate hosts all low level token types and behaviour
- catalog log and snapshot types updated to use the token repo
- tests that relied on auth have been updated to use the new token
generation mechanism and new admin token generation/regeneration tests
have been added
* feat: list admin tokens
- allows listing admin tokens
- uses _internal db for token system table
- mostly test fixes due to _internal db
* chore: couple of updates to fix cargo audit job
- remove humantime ignore in deny.toml
- update pyo3 to use 0.24.1 (https://rustsec.org/advisories/RUSTSEC-2025-0020.html)
* chore: moved pyo3 version to root cargo.toml
* feat: add influxdb3_shutdown crate
provides basic wait methods for unix/windows OS's
* feat: graceful shutdown
* docs: add rust docs and test to influxdb3_shutdown
Added rustdoc comments to types and methods in the influxdb3_shutdown
crate as well as a test that shows the ordering of a shutdown.
This creates a CatalogUpdateMessage type that is used to send
CatalogUpdates; this type performs the send on the oneshot Sender so
that the consumer of the message does not need to do so.
Subscribers to the catalog get a CatalogSubscription, which uses the
CatalogUpdateMessage type to ACK the message broadcast from the catalog.
This means that catalog message broadcast can fail, but this commit does
not provide any means of rolling back a catalog update.
A test was added to check that it works.
* refactor: use repository in catalog
The catalog was refactored to use identifiers on everything, and store
everything in a consistent structure. This structure makes use of the
`Repository` type that holds a `SerdeVecMap` of Id to Resource, along
with the next Id, and a bi-map of Id to resource name.
The `Repository` type is used at each level of the catalog where a
resource is stored.
This simplified repeated logic for snapshot'ing, insert and update of
resources in the catalog, as well as accessor methods for getting by id
or name, and mapping names to ids and vice-versa.
In addition, the process for catalog batch verification and permit was
altered so that the permit process induces a retry if the catalog was
updated while the catalog batch function was producing the batch, i.e, if
the catalog sequence incremented while the caller was waiting for a permit.
This eliminated the need for verifying the catalog batch after it had been
generated, and allows for a single path to apply a catalog batch after it
has been persisted to object store.
This assumes that the generation of the catalog batch implies validity.
Irelevant tests were removed.
Last and Distinct cache's now rely more heavily on Ids, though the proc-
essing engine still needs to switch over to use Ids for starting/stopping
triggers.
In #25927 we missed that JSON queries were broken despite having some
tests use the format. This fixes JSON queries such that they now
properly contain a comma between RecordBatches. This commit also
includes tests for the formats that now stream data back (CSV, JSON, and
JSON Lines) so that we won't run into this issue again.
* deduplicate QueryParams->QueryRequest and Format->QueryFormat
* move WriteParams into influxdb3_types crate
* DRY up client HTTP request handling code in *RequestBuilder.send
methods.
* DRY up a bunch of other non-Builder http request handling
Trevor Hilton
Stuart Carnie
praveen-influx
Michael Gattozzi
Jackson Newhouse
wayne