Commit Graph

37 Commits (pk/write-fixed-load-gen)

Author SHA1 Message Date
Trevor Hilton d1bf17017d
refactor: move jemalloc to `influxdb3 serve` and address workspace lints (#26466)
* refactor: move jemalloc code to influxdb3 serve

* refactor: enable workspace lints in influxdb3

* feat: add FromStr impl to catalog identifier types
2025-05-27 12:51:23 -04:00
Trevor Hilton 4dc61df77f
chore: update to latest influxdb3_core (#26429)
* chore: update to latest core

* chore: allow CDLA permissive 2 license

* chore: update insta snapshot for new internal df tables

* test: update assertion in flightsql test

* fix: object store size hinting workaround in clap_blocks

Applied a workaround from upstream to strip size hinting from the object
store get request options. See:

https://github.com/influxdata/influxdb_iox/issues/13771

* fix: query_executor tests use object store size hinting workaround

* fix: insta snapshot test for show system summary command

* chore: update windows- crates for advisories

* chore: update to latest sha on influxdb3_core branch

* chore: update to latest influxdb3_core rev

* refactor: pr feedback

* refactor: do not use object store size hint layer

Instead of using the ObjectStoreStripSizeHint layer, just provide the
configuration to datafusion to disable the use of size hinting from
iox_query.

This is used in IOx and not relevant to Monolith.

* fix: use parquet cache for get_opts requests

* test: that the parquet cache is being hit from write buffer
2025-05-26 14:11:06 -04:00
Trevor Hilton d1c10f4b29
fix: backfill new tags with NULL instead of empty string (#26446)
* fix: backfill new tags with NULL instead of empty string

* refactor: use helper for append_null

* test: add a test to check null back/forward fill
2025-05-21 17:23:51 -04:00
praveen-influx 1ec063b0c4
feat: support named admin tokens (#26434)
* feat: support named admin tokens

- `--name` and `--expiry` are now allowed for `--admin` subcommand
- `--regenerate` is restricted to operator token only
- the endpoint is not allowed if auth is disabled

closes: https://github.com/influxdata/influxdb_pro/issues/854

This is a port of https://github.com/influxdata/influxdb_pro/pull/850 (hash:156981e4a1)

* refactor: address PR feedback
2025-05-20 15:30:19 +01:00
praveen-influx b404e8475c
fix: do not allow operator token from being deleted (#26418)
* fix: do not allow operator token from being deleted

closes: https://github.com/influxdata/influxdb_pro/issues/819

* refactor: address PR feedback

* fix: add a word and clarifying colon

* fix: failing test

---------

Co-authored-by: Peter Barnett <peter.barnett03@gmail.com>
2025-05-15 09:10:37 +01:00
peterbarnett03 f6aa0c1b29
fix: update token creation output (#26336)
* fix: clarify token generation output for easier readibility

* fix: linter error

* fix: additional linter error

* fix: linter error on output

* fix: switch to owo_colorize
2025-04-30 12:11:33 -05:00
praveen-influx 08f2f857da
feat: additional info when started --without-auth (#26338)
- return 405 message body to indicate the endpoints are disabled
- extra log to say server has been started without auth
2025-04-29 08:06:44 -05:00
Trevor Hilton e977d263f8
chore: back-port changes from influxdb_pro#762 (#26323) 2025-04-25 10:02:08 -04:00
Stuart Carnie 8507bdc766
fix: Ensure `iox` table schema is sorted for SHOW TABLES (#26302)
* fix: Ensure `iox` table schema is sorted for SHOW TABLES

Closes #25860

* chore: Add insta snapshot

* chore: placate clippy
2025-04-23 09:05:06 +10:00
praveen-influx b41a2d9bc3
feat: expose `--format json` for token creation cmd (#26286)
* feat: expose `--format json` for token creation cmd

closes: https://github.com/influxdata/influxdb/issues/25913

* refactor: address feedback
2025-04-18 08:06:02 +01:00
Trevor Hilton e4cfbf71f7
chore: back-port catalog limit refactor from enterprise (#26278)
This back-ports some changes to how limits are enforced on the catalog
from enterprise. There were some changes that would catch some potential
bugs.
2025-04-16 14:03:24 -04:00
Stuart Carnie 5da3850319
fix: `--tags` argument is optional and now requires at least one tag (#26275)
* fix: `--tags` argument is optional and now requires at least one tag

Closes #25824

* chore: Added CLI unit tests
2025-04-16 11:46:13 -04:00
praveen-influx 0cb5f9077c
feat: ask for confirmation when regenerating token (#26258) 2025-04-13 09:07:09 +01:00
praveen-influx e5af9f95d2
feat: enable auth by default (#26243)
* feat: enable auth by default

- Removes `--bearer-token` support and starts the server with auth by
  default.
- Adds `--without-auth` switch to start the server without any auth

* feat: changes for auth being turned off

when auth is turned off,
- disallow token endpoints (returns 405)
- remove hash column when querying tokens system table

* refactor: address PR feedback
2025-04-11 16:44:27 +01:00
Trevor Hilton a1923d9304
fix: build after some merge regressions (#26248) 2025-04-10 16:20:11 -04:00
praveen-influx 64eeb09687
feat: allow token deletes (#26247)
This commit allows deletion of tokens by name. Below is an example,

`influxdb3 delete token --token-name _admin --token $CURRENT_ADMIN_TOKEN`

It needs user confirmation before proceeding with the delete
2025-04-10 20:31:19 +01:00
Michael Gattozzi fe69793418
feat: Add TLS support for influxdb3 (#26246)
This commit adds TLS support to influxdb3 and allows users to pass in a
path to a key and cert file with the --tls-key and --tls-cert flags in
the serve command. It also adds the ability for every command to specify
a certificate authority for requests. This is mostly needed when the
cert is self signed, but there are other use cases for this.

The big thing is that most of our tests now use TLS by default. Included
are self signed certs for localhost and the the CA cert included in the
commit. Since these are *only* used for testing this should be fine to
include as they are not used in nor are they intended to be used in any
production system. The expiry has been set for 365 days and the file
perms are set to o600 like the original issue mentioned. The tests pass
with this restriction.

I've verified that the API works via curl with the self signed certs as
I did *not* need to pass in the -k option to bypass checking the certs
were valid. The same goes for our tests. They use the rootCA.pem file
to verify the self signed cert when connecting and reject it otherwise.

With this users can be confident that their queries are safely encrypted
during transport.

Note that TLS works for both FlightSQL and our normal APIs.

Closes #25774
2025-04-10 13:45:47 -04:00
praveen-influx 1983818e36
feat: porting token work from enterprise (#26239)
* feat: generate persistable admin token

- this commit allows admin token creation using `influxdb3 create token
  --admin` and also allows regeneration of admin token by `influxdb3
  create token --admin --regenerate`
- `influxdb3_authz` crate hosts all low level token types and behaviour
- catalog log and snapshot types updated to use the token repo
- tests that relied on auth have been updated to use the new token
  generation mechanism and new admin token generation/regeneration tests
  have been added

* feat: list admin tokens

- allows listing admin tokens
- uses _internal db for token system table
- mostly test fixes due to _internal db
2025-04-09 16:31:59 +01:00
Trevor Hilton c7854363c4
chore: back-port changes to shutdown code from enterprise (#26206)
* refactor: make ShutdownManager Clone

ShutdownManager can be clone since its underlying types from tokio are
all shareable via clone.

* refactor: make ShutdownToken not Clone

Alters the API so that the ShutdownToken is not cloneable. This will help
ensure that the Drop implementation is invoked from the correct place.
2025-04-01 11:32:23 -04:00
Jackson Newhouse d245b2e886
fix(processing_engine): properly extract data from DictionaryArrays (#26204) 2025-03-31 11:38:23 -07:00
Trevor Hilton 24887770ef
feat: shutdown on WAL overwritten (#26203)
* feat: trigger shutdown if wal has been overwritten

WAL persist uses PutMode::Create in order to invoke shutdown if another
process writes to the WAL ahead of it.

A test was added to check that it works from CLI test suite.

* chore: clippy
2025-03-31 12:58:18 -04:00
Jackson Newhouse 7c02593fef
chore: Overhaul CLI tests for better ergonomics. (#26168)
* chore: Overhaul CLI tests for better ergonomics.

* chore: switch with_query() to with_sql().
2025-03-20 09:19:12 -07:00
Trevor Hilton 3ad0670452
test: small change to distinct cache cli test (#26152)
Added one check to the distinct cache create test that checks duplicate
creation behaviour while investigating influxdb_pro#601
2025-03-17 11:05:25 -04:00
Jackson Newhouse e49f0e89fa
fix(processing_engine): Change how temporary plugin files are managed. (#26141) 2025-03-14 10:52:38 -07:00
Jackson Newhouse e1015f15f7
feat(processing_engine): proper escaping of LineBuilder arguments. (#26134) 2025-03-14 09:31:58 -07:00
Michael Gattozzi 97b8c471f8
fix: Make sure JSON fully closes on empty records (#26130) 2025-03-12 14:01:40 -04:00
Jackson Newhouse a455598be6
feat(processing_engine): Implement Processing Engine Cache. (#26111) 2025-03-11 15:48:47 -07:00
Trevor Hilton 503819468e
feat: catalog checkpoints (#26126) 2025-03-11 18:20:36 -04:00
Trevor Hilton 72dc4458fd
chore: backport changes to catalog from enterprise (#26116)
* chore: backport changes to influxdb3_catalog crate

* chore: backport changes to influxdb3_cache crate

* chore: backport changes to influxdb3_write crate

* chore: backport changes to influxdb3_proc_eng crate

* chore: backport influxdb3 crate changes for catalog

* chore: backport changes to influxdb3_id crate

* chore: backport changes to influxdb3_wal crate

* chore: backport changes to influxdb3_clap_blocks crate

* chore: backport changes to influxdb3_client crate

* chore: backport influxdb3_server crate changes

* chore: fix after full backport

* fix: ordering of catalog broadcast
2025-03-11 12:11:51 -04:00
Jackson Newhouse 5fa417c3f0
feat: remove system-py (#26087)
* feat: remove system-py

* chore: allow Apache-2.0 WITH LLVM-exception license.
2025-03-10 11:10:33 -07:00
Jackson Newhouse 357c05f3d0
feat(processing_engine): Support Flask semantics for responses from request plugins. (#26095) 2025-03-04 09:00:59 -08:00
wayne e2c3e7c8c8
feat: improve CLI list arg consistency (always use commas) (#26088)
* chore: eliminate SeparateList, use comma-delimited flag list options everywhere
* fix: remove num_args attribute from comma-delimited args
2025-03-03 12:52:13 -07:00
Jackson Newhouse 208b82a32d
fix(processing_engine): Use default globals so that builtins are automatically included. (#26072) 2025-02-27 16:11:08 -08:00
Michael Gattozzi 1f72bfcc33
feat: Update to Rust 1.85 and 2024 Edition (#26046) 2025-02-20 14:58:07 -05:00
Jackson Newhouse fbcb9403c4
feat(processing_engine): log processing engine logging calls to sys events. (#25939) 2025-02-04 15:16:04 -08:00
Jackson Newhouse d9dd8a32a2
fix(processing_engine): Use the configured request path for Request plugins. (#25945) 2025-01-31 10:36:47 -08:00
Michael Gattozzi b7d2475ea6
chore: move separate cli and server tests (#25934)
This commit restructures our tests to look like Enterprise in their
layout. We break cli.rs into it's own module, combine the server tests
and cli tests under one lib.rs file and handle the changes for
visibility and import paths needed to make things work. the packages
tests have been cfged out as a module so that it would not need to be
added on a per test basis. Note that those tests fail locally for me
currently, but it seems like we weren't testing these in CI at the
moment.

There is no issue for this.
2025-01-31 11:41:44 -05:00