* feat: additional server setup for admin token recovery
- new server to only serve admin token regeneration without an admin
token has been added
- minor refactors to allow reuse of some of the utilities like trace
layer for metrics moved to their own functions to allow them to be
instantiated for both servers
- tests added to check if both the new server works right for
regenerating token and also ensure none of the other functionalities
are available on the admin token recovery server
closes: https://github.com/influxdata/influxdb/issues/26330
* refactor: tidy ups + extra logging
* refactor: address PR feedback
- recovery server now only starts when `--admin-token-recovery-http-bind` is passed in
- as soon as regeneration is done, the recovery server shuts itself down
- the select! macro logic has been changed such that shutting down
recovery server does not shutdown the main server
* refactor: host url updates when regenerating token
- when `--regenerate` is passed in, `--host` still defaults to the main
server. To get to the recovery server, `--host` with the recovery
server address should be passed in
This commit fixes queries that could come back as failures due to
improperly quoted table names in queries. It also fixes issues in
Enterprise where compaction would fail due to double escaped names.
The fix is relatively simple:
- Use the parse not from function for the Path type in the object_store
crate
- Quote escape table names in queries
Add a new system table that allows users to inspect the arguments
configured for processing engine triggers. The table has three columns:
- trigger_name: name of the trigger
- argument_key: key of the argument
- argument_value: value of the argument
Each trigger argument appears as a separate row in the table, making
it easy to query specific triggers or arguments.
Update snapshot files to include processing_engine_trigger_arguments table
Update test snapshots to include the new processing_engine_trigger_arguments
system table in:
- Table listing outputs
- Error messages showing valid table names
- Table summaries
This ensures tests properly reflect the new system table in their
expected outputs.
* feat: Allow hard_deleted date of deleted schema to be updated
* feat: Include hard_deletion_date in `_internal` `databases` and `tables`
* feat: Unit tests for testing deleted databases
* chore: Default is now to hard-delete with default duration
* test: Update test names and assertions for new default hard deletion behavior
- Renamed delete_table_defaults_to_hard_delete_never to delete_table_defaults_to_hard_delete_default
- Renamed delete_database_defaults_to_hard_delete_never to delete_database_defaults_to_hard_delete_default
- Updated assertions to expect default deletion duration instead of None/Never
- Aligns with the change of HardDeletionTime default from Never to Default
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* chore: Remove TODO
* chore: PR feedback and other improvements
* Ensure system databases and tables schema specify a timezone for the
`hard_deletion_time` Timestamp columns (otherwise they display without
a timezone)
* `DELETE` using `default` delay is idempotent, so multiple requests
will not continue to update the `hard_deletion_time`
* Improved test coverage for these behaviours
---------
Co-authored-by: Claude <noreply@anthropic.com>
Before this commit, although `--object-store` is mandatory it is not
reflected in the error messages. The examples are listed in the issue
976.
This commit makes `object_store` explicitly required which means error
messages include `--object-store` listed as mandatory
closes: https://github.com/influxdata/influxdb_pro/issues/976
This commit touches quite a few things, but the main changes that need
to be taken into account are:
- An update command has been added to the CLI. This could be further
extended in the future to update more than just Database retention
periods. The API call for that has been written in such a
way as to allow other qualities of the database to be updated
at runtime from one API call. For now it only allows the retention
period to be updated, but it could in theory allow us to rename a
database without needing to wipe things, especially with a stable ID
underlying everything.
- The create database command has been extended to allow
its creation with a retention period. In tandem with the update
command users can now assign or delete retention periods at will
- The ability to query catalog data about both databases and tables has
been added as well. This has been used in tests added in this commit,
but is also a fairly useful query when wanting to look at things such
as the series key. This could be extended to a CLI command as well if
we want to allow users to look at this data, but for now it's in the
_internal table.
With these changes a nice UX has been created to allow our customers to
work with retention periods.
* Tracks the generation duration configuration for the write buffer
in the catalog.
* Still leverages the CLI arguments to set it on initial start up of
the server.
* Exposes a system table on the _internal database to view the configured
generation durations.
* This doesn't change how the gen1 duration is used by the write buffer.
* Adds several tests to check things work as intended.
* feat: add retention period to catalog
* fix: handle humantime parsing error properly
* refactor: use new iox_http_util types
---------
Co-authored-by: Michael Gattozzi <mgattozzi@influxdata.com>
* chore: update to latest core
* chore: allow CDLA permissive 2 license
* chore: update insta snapshot for new internal df tables
* test: update assertion in flightsql test
* fix: object store size hinting workaround in clap_blocks
Applied a workaround from upstream to strip size hinting from the object
store get request options. See:
https://github.com/influxdata/influxdb_iox/issues/13771
* fix: query_executor tests use object store size hinting workaround
* fix: insta snapshot test for show system summary command
* chore: update windows- crates for advisories
* chore: update to latest sha on influxdb3_core branch
* chore: update to latest influxdb3_core rev
* refactor: pr feedback
* refactor: do not use object store size hint layer
Instead of using the ObjectStoreStripSizeHint layer, just provide the
configuration to datafusion to disable the use of size hinting from
iox_query.
This is used in IOx and not relevant to Monolith.
* fix: use parquet cache for get_opts requests
* test: that the parquet cache is being hit from write buffer
* feat: `/ping` API contains versioning headers
Further, the product version can be modified by updating the metadata in
the `influxdb3_process` `Cargo.toml`.
* chore: PR feedback
* chore: placate linter
* fix: do not allow operator token from being deleted
closes: https://github.com/influxdata/influxdb_pro/issues/819
* refactor: address PR feedback
* fix: add a word and clarifying colon
* fix: failing test
---------
Co-authored-by: Peter Barnett <peter.barnett03@gmail.com>
* feat: allow health,ping,metrics to opt out of auth
This commit introduces `--disable-authz <DISABLE_AUTHZ_RESOURCES>`. The
options for `DISABLE_AUTHZ_RESOURCES` are health, ping and metrics. By
default all these resources will be guarded
closes: https://github.com/influxdata/influxdb_pro/issues/774
* chore: update influxdb3/src/commands/helpers.rs
space after comma in help text
Co-authored-by: Trevor Hilton <thilton@influxdata.com>
* chore: update influxdb3/src/help/serve.txt
space after comma in help text
Co-authored-by: Trevor Hilton <thilton@influxdata.com>
* chore: update influxdb3/src/help/serve_all.txt
space after comma in help text
Co-authored-by: Trevor Hilton <thilton@influxdata.com>
* refactor: use statics to reduce clones/copies
---------
Co-authored-by: Trevor Hilton <thilton@influxdata.com>
* feat: support `Basic $TOKEN` for all apis
closes: https://github.com/influxdata/influxdb/issues/25833
* refactor: address PR feedback to return MalformedRequest error when `:` is used more than once in user-pass pair
* refactor: change the message sent back for malformed auth header
* feat: remove limit on LVC size
* fix: bad test case and incorrect info
* fix: more clarity and default value
* fix: light CLI polishes
* fix: bad snapshot
* feat: Add a negative cert test
This adds a test that will panic on server startup because connections
to said server are invalid. We add a bad expired cert to our cert
generation for usage in our tests.
Note that this test is only really valid if other tests pass as it
depends on waiting for the server start checks to fail. If other
tests run then their server started fine and so did this one, the
only difference being that connections will error due to a bad tls cert.
Closes#26256
* feat: Add minimum TLS version test
This is a follow on to #26307. In this commit we add a test where we
check that connections only pass if TLS is set to v1.3. The default is
1.2 and other tests connect with that just fine. In this test we spin
up a server using only v1.3 as the minimum and try to connect with v1.2
which we expect to fail and then v1.3 which should pass.
Closes#26308
This commit is a follow up to #26246 and generates test certs on the fly
for our test suite. In practice this will only need to be done once with
a fresh repo check out as the certs will expire long after anyone would
reasonably be working on this code in the year 4096! This could be
extended in the future to generate negative tls tests where the file
should be expired.
Closes#26254
* feat: enable auth by default
- Removes `--bearer-token` support and starts the server with auth by
default.
- Adds `--without-auth` switch to start the server without any auth
* feat: changes for auth being turned off
when auth is turned off,
- disallow token endpoints (returns 405)
- remove hash column when querying tokens system table
* refactor: address PR feedback
This commit allows deletion of tokens by name. Below is an example,
`influxdb3 delete token --token-name _admin --token $CURRENT_ADMIN_TOKEN`
It needs user confirmation before proceeding with the delete
This commit adds TLS support to influxdb3 and allows users to pass in a
path to a key and cert file with the --tls-key and --tls-cert flags in
the serve command. It also adds the ability for every command to specify
a certificate authority for requests. This is mostly needed when the
cert is self signed, but there are other use cases for this.
The big thing is that most of our tests now use TLS by default. Included
are self signed certs for localhost and the the CA cert included in the
commit. Since these are *only* used for testing this should be fine to
include as they are not used in nor are they intended to be used in any
production system. The expiry has been set for 365 days and the file
perms are set to o600 like the original issue mentioned. The tests pass
with this restriction.
I've verified that the API works via curl with the self signed certs as
I did *not* need to pass in the -k option to bypass checking the certs
were valid. The same goes for our tests. They use the rootCA.pem file
to verify the self signed cert when connecting and reject it otherwise.
With this users can be confident that their queries are safely encrypted
during transport.
Note that TLS works for both FlightSQL and our normal APIs.
Closes#25774
* feat: generate persistable admin token
- this commit allows admin token creation using `influxdb3 create token
--admin` and also allows regeneration of admin token by `influxdb3
create token --admin --regenerate`
- `influxdb3_authz` crate hosts all low level token types and behaviour
- catalog log and snapshot types updated to use the token repo
- tests that relied on auth have been updated to use the new token
generation mechanism and new admin token generation/regeneration tests
have been added
* feat: list admin tokens
- allows listing admin tokens
- uses _internal db for token system table
- mostly test fixes due to _internal db
* refactor: make ShutdownManager Clone
ShutdownManager can be clone since its underlying types from tokio are
all shareable via clone.
* refactor: make ShutdownToken not Clone
Alters the API so that the ShutdownToken is not cloneable. This will help
ensure that the Drop implementation is invoked from the correct place.
* feat: trigger shutdown if wal has been overwritten
WAL persist uses PutMode::Create in order to invoke shutdown if another
process writes to the WAL ahead of it.
A test was added to check that it works from CLI test suite.
* chore: clippy
praveen-influx
Michael Gattozzi
Trevor Hilton
Stuart Carnie
wayne
peterbarnett03
Jackson Newhouse