* refactor: Use iox_http_util::Request instead of hyper::Request
* refactor: Use iox_http_util::RequestBuilder instead of hyper::Request::builder
* refactor: Use iox_http_util::empty_request_body instead of Body::empty
* refactor: Use iox_http_util::bytes_to_request_body instead of Body::from
* refactor: Use http_body::Body instead of hyper::body::HttpBody
* refactor: Use iox_http_util::Response instead of hyper::Response
* refactor: Use iox_http_util::ResponseBuilder instead of hyper::Response::builder
* refactor: Use iox_http_util::empty_response_body instead of Body::empty
* refactor: Use iox_http_util::bytes_to_response_body instead of Body::from
* refactor: Use iox_http_util::stream_results_to_response_body instead of Body::wrap_stream
* refactor: Use the read_body_bytes_for_tests helper fn
* chore: update to latest core
* chore: allow CDLA permissive 2 license
* chore: update insta snapshot for new internal df tables
* test: update assertion in flightsql test
* fix: object store size hinting workaround in clap_blocks
Applied a workaround from upstream to strip size hinting from the object
store get request options. See:
https://github.com/influxdata/influxdb_iox/issues/13771
* fix: query_executor tests use object store size hinting workaround
* fix: insta snapshot test for show system summary command
* chore: update windows- crates for advisories
* chore: update to latest sha on influxdb3_core branch
* chore: update to latest influxdb3_core rev
* refactor: pr feedback
* refactor: do not use object store size hint layer
Instead of using the ObjectStoreStripSizeHint layer, just provide the
configuration to datafusion to disable the use of size hinting from
iox_query.
This is used in IOx and not relevant to Monolith.
* fix: use parquet cache for get_opts requests
* test: that the parquet cache is being hit from write buffer
* feat: `/ping` API contains versioning headers
Further, the product version can be modified by updating the metadata in
the `influxdb3_process` `Cargo.toml`.
* chore: PR feedback
* chore: placate linter
* feat: allow health,ping,metrics to opt out of auth
This commit introduces `--disable-authz <DISABLE_AUTHZ_RESOURCES>`. The
options for `DISABLE_AUTHZ_RESOURCES` are health, ping and metrics. By
default all these resources will be guarded
closes: https://github.com/influxdata/influxdb_pro/issues/774
* chore: update influxdb3/src/commands/helpers.rs
space after comma in help text
Co-authored-by: Trevor Hilton <thilton@influxdata.com>
* chore: update influxdb3/src/help/serve.txt
space after comma in help text
Co-authored-by: Trevor Hilton <thilton@influxdata.com>
* chore: update influxdb3/src/help/serve_all.txt
space after comma in help text
Co-authored-by: Trevor Hilton <thilton@influxdata.com>
* refactor: use statics to reduce clones/copies
---------
Co-authored-by: Trevor Hilton <thilton@influxdata.com>
This will have revision info displayed with `-nightly` added to the version, which
leads the most recent released version by 1 minor release.
For example,
* Using the `/ping` API:
```
❯ curl localhost:8181/ping
{"version":"3.1.0-nightly","revision":"f6aa0c1b29","process_id":"ccbd5879-a83b-4091-b68c-92c6b704cf8a"}
```
* Checking the binary version:
```
❯ influxdb3 --version
influxdb3 3.1.0-nightly, revision f6aa0c1b29
```
This commit allows users to set a minimum TLS version. The default is
1.2. The choices are TLS 1.2 or TLS 1.3 which can be set via env var:
INFLUXDB3_TLS_MINIMUM_VERSION="tls-1.2"
or
INFLUXDB3_TLS_MINIMUM_VERSION="tls-1.3"
and for the command line flag for the serve command:
--tls-minimum-version tls-1.2
or
--tls-minimum-version tls-1.3
With this users have more fine grained control over what tls version
they require.
Closes#26255
This commit is a follow up to #26246 and generates test certs on the fly
for our test suite. In practice this will only need to be done once with
a fresh repo check out as the certs will expire long after anyone would
reasonably be working on this code in the year 4096! This could be
extended in the future to generate negative tls tests where the file
should be expired.
Closes#26254
Adds a metric to track total retried catalog operations due to the catalog
being updated elsewhere. Includes a test to check the counter increments
on basic catalog operations.
This commit adds TLS support to influxdb3 and allows users to pass in a
path to a key and cert file with the --tls-key and --tls-cert flags in
the serve command. It also adds the ability for every command to specify
a certificate authority for requests. This is mostly needed when the
cert is self signed, but there are other use cases for this.
The big thing is that most of our tests now use TLS by default. Included
are self signed certs for localhost and the the CA cert included in the
commit. Since these are *only* used for testing this should be fine to
include as they are not used in nor are they intended to be used in any
production system. The expiry has been set for 365 days and the file
perms are set to o600 like the original issue mentioned. The tests pass
with this restriction.
I've verified that the API works via curl with the self signed certs as
I did *not* need to pass in the -k option to bypass checking the certs
were valid. The same goes for our tests. They use the rootCA.pem file
to verify the self signed cert when connecting and reject it otherwise.
With this users can be confident that their queries are safely encrypted
during transport.
Note that TLS works for both FlightSQL and our normal APIs.
Closes#25774
This changes the help messages for the serve command and the influxdb3
top level help output. This is to provide better help output and options
for users compared to the clap defaults. After many iterations this code
was the final design I landed on. In order to make our custom help work
we:
1. Roughly parse the args to check for a given command and help message
2. If there is no subcommand we print out our custom help message for
the top level command
3. If there is a subcommand we only print it out for serve currently
4. --help-all prints a more detailed message
This lets us upgrade our help messages in place over time as the
subcommands themselves have subcommands that also need their own help
messages. For now we only include these two.
We could not use something like `clap-help` or derive the output from
the derived clap struct automatically due to multiple issues, but
suffice to say the main thing was that we could not parse the args into
the `Config` struct and then figure out our help message and `clap`'s
message template function was too underpowered. We also wanted to create
a `help-all` flag and this was quite hard to make work with `clap`'s
short and long help.
In the end the best option was to roll our own. While a bit hacky in
terms of us maintaining it, the overall outcome should be a much nicer
user experience in which they can have much better introductions to the
tool. `serve` was quite egregious with a lot of output and very little
actionable options to use for most users.
Closes#26201
* feat: generate persistable admin token
- this commit allows admin token creation using `influxdb3 create token
--admin` and also allows regeneration of admin token by `influxdb3
create token --admin --regenerate`
- `influxdb3_authz` crate hosts all low level token types and behaviour
- catalog log and snapshot types updated to use the token repo
- tests that relied on auth have been updated to use the new token
generation mechanism and new admin token generation/regeneration tests
have been added
* feat: list admin tokens
- allows listing admin tokens
- uses _internal db for token system table
- mostly test fixes due to _internal db
* chore: couple of updates to fix cargo audit job
- remove humantime ignore in deny.toml
- update pyo3 to use 0.24.1 (https://rustsec.org/advisories/RUSTSEC-2025-0020.html)
* chore: moved pyo3 version to root cargo.toml
* feat: add influxdb3_shutdown crate
provides basic wait methods for unix/windows OS's
* feat: graceful shutdown
* docs: add rust docs and test to influxdb3_shutdown
Added rustdoc comments to types and methods in the influxdb3_shutdown
crate as well as a test that shows the ordering of a shutdown.
* refactor: use repository in catalog
The catalog was refactored to use identifiers on everything, and store
everything in a consistent structure. This structure makes use of the
`Repository` type that holds a `SerdeVecMap` of Id to Resource, along
with the next Id, and a bi-map of Id to resource name.
The `Repository` type is used at each level of the catalog where a
resource is stored.
This simplified repeated logic for snapshot'ing, insert and update of
resources in the catalog, as well as accessor methods for getting by id
or name, and mapping names to ids and vice-versa.
In addition, the process for catalog batch verification and permit was
altered so that the permit process induces a retry if the catalog was
updated while the catalog batch function was producing the batch, i.e, if
the catalog sequence incremented while the caller was waiting for a permit.
This eliminated the need for verifying the catalog batch after it had been
generated, and allows for a single path to apply a catalog batch after it
has been persisted to object store.
This assumes that the generation of the catalog batch implies validity.
Irelevant tests were removed.
Last and Distinct cache's now rely more heavily on Ids, though the proc-
essing engine still needs to switch over to use Ids for starting/stopping
triggers.
This commit restores the old behavior we had where new tags can be added
to a schema. To do this we made tags nullable and brings us in line with
our other products. These changes were made in this PR:
https://github.com/influxdata/influxdb3_core/pull/41.
Changes to accomplish this new behavior were:
- Queries now do not return an empty string for null tags instead they
are returned as null, or in many formats not at all.
- References to v1 for parsing and validating lines were removed as we
only have one path for doing so these days shared amongst all the
write_lp endpoints.
- We fixed failing tests that expected us to not be able to have new
tags or depended on that functionality indirectly
- Tests had their snapshot files updated to reflect that tags are
nullable by default
- Behavior for making a schema and checking whether a column can be null
were updated in a separate repo and integrated here
- The series_key is updated whenever we get a new tag added to the
schema
- New tests were added to show that you can add a new tag and that the
series key is updated as part of that
With the above changes we can now allow tags to be added again by users
like they would expect, especially with v1 and v2 apis and Telegraf
plugins.
The distinct cache info for tables was not serialized in the catalog.
This fixes it, but also updates the catalog serialization to use the
snapshot type serialization from the Catalog type all the way down.
The Eq and PartialEq impls were removed from Catalog and InnerCatalog
as they were only used in tests, and wer replaced by pure insta snapshot
tests.
A test was added to check that the distinct cache serializes/deserializes
* deduplicate QueryParams->QueryRequest and Format->QueryFormat
* move WriteParams into influxdb3_types crate
* DRY up client HTTP request handling code in *RequestBuilder.send
methods.
* DRY up a bunch of other non-Builder http request handling
Partially fixes https://github.com/influxdata/influxdb/issues/24672
* move most HTTP req/resp types into `influxdb3_types` crate
* removes the use of locally-scoped request type structs from the `influxdb3_client` crate
* fix plugin dependency/package install bug
* it looks like the `DELETE` http method was being used where `POST` was expected for `/api/v3/configure/plugin_environment/install_packages` and `/api/v3/configure/plugin_environment/install_requirements`
* feat: first stab at locally updating parquet cache
closes: https://github.com/influxdata/influxdb/issues/25887
* refactor: use enums to separate out the modes
This commit introduced the `Immediate` and `Eventual` modes for
fulfilling the cache request. In immediate mode since the data is
readily available to be cached, we can avoid extra requests to object
store.
part of: https://github.com/influxdata/influxdb/issues/25887
* feat: Add request plugin capability
Adds the request plugin type. Triggers can be bound to an API endpoint at /api/v3/engine/<path>. Requests will get yielded to the plugin with the query parameters, request parameters, and request body.
I didn't implement the test endpoint for this plugin type as it seems much more natural for users to save the file and make a new request. Once #25863 is done it'll make it very easy.
Closes#25862
* chore: fix spelling in error message
* feat(processing_engine): Add cron plugins and triggers to the processing engine.
* feat(processing_engine): switch from 'cron plugin' to 'schedule plugin', use TimeProvider.
* feat(processing_engine): add test for test scheduled plugin.
Carol (Nichols || Goulding)
Trevor Hilton
Stuart Carnie
praveen-influx
peterbarnett03
Michael Gattozzi
Paul Dix
Jackson Newhouse
wayne