Commit Graph

236 Commits (main)

Author SHA1 Message Date
Trevor Hilton e4cfbf71f7
chore: back-port catalog limit refactor from enterprise (#26278)
This back-ports some changes to how limits are enforced on the catalog
from enterprise. There were some changes that would catch some potential
bugs.
2025-04-16 14:03:24 -04:00
Trevor Hilton 5d7cb88f87
feat: track catalog retries as prometheus metric (#26251)
Adds a metric to track total retried catalog operations due to the catalog
being updated elsewhere. Includes a test to check the counter increments
on basic catalog operations.
2025-04-11 15:24:10 -04:00
Trevor Hilton b11cf81dc6
refactor: remove Option from catalog update APIs (#26250)
Catalog update APIs were returning an Option that was not necessary. It
was always Some, so this removes the Option from the API to make the
intent clear - if there is an update made by the requested change, there
is a Batch; whereas if the requested change is erroneous, or would not
produce a change, the response is Err.
2025-04-11 12:34:18 -04:00
praveen-influx e5af9f95d2
feat: enable auth by default (#26243)
* feat: enable auth by default

- Removes `--bearer-token` support and starts the server with auth by
  default.
- Adds `--without-auth` switch to start the server without any auth

* feat: changes for auth being turned off

when auth is turned off,
- disallow token endpoints (returns 405)
- remove hash column when querying tokens system table

* refactor: address PR feedback
2025-04-11 16:44:27 +01:00
praveen-influx 64eeb09687
feat: allow token deletes (#26247)
This commit allows deletion of tokens by name. Below is an example,

`influxdb3 delete token --token-name _admin --token $CURRENT_ADMIN_TOKEN`

It needs user confirmation before proceeding with the delete
2025-04-10 20:31:19 +01:00
Michael Gattozzi fe69793418
feat: Add TLS support for influxdb3 (#26246)
This commit adds TLS support to influxdb3 and allows users to pass in a
path to a key and cert file with the --tls-key and --tls-cert flags in
the serve command. It also adds the ability for every command to specify
a certificate authority for requests. This is mostly needed when the
cert is self signed, but there are other use cases for this.

The big thing is that most of our tests now use TLS by default. Included
are self signed certs for localhost and the the CA cert included in the
commit. Since these are *only* used for testing this should be fine to
include as they are not used in nor are they intended to be used in any
production system. The expiry has been set for 365 days and the file
perms are set to o600 like the original issue mentioned. The tests pass
with this restriction.

I've verified that the API works via curl with the self signed certs as
I did *not* need to pass in the -k option to bypass checking the certs
were valid. The same goes for our tests. They use the rootCA.pem file
to verify the self signed cert when connecting and reject it otherwise.

With this users can be confident that their queries are safely encrypted
during transport.

Note that TLS works for both FlightSQL and our normal APIs.

Closes #25774
2025-04-10 13:45:47 -04:00
praveen-influx 1983818e36
feat: porting token work from enterprise (#26239)
* feat: generate persistable admin token

- this commit allows admin token creation using `influxdb3 create token
  --admin` and also allows regeneration of admin token by `influxdb3
  create token --admin --regenerate`
- `influxdb3_authz` crate hosts all low level token types and behaviour
- catalog log and snapshot types updated to use the token repo
- tests that relied on auth have been updated to use the new token
  generation mechanism and new admin token generation/regeneration tests
  have been added

* feat: list admin tokens

- allows listing admin tokens
- uses _internal db for token system table
- mostly test fixes due to _internal db
2025-04-09 16:31:59 +01:00
Trevor Hilton 7d95c48ecf
fix: empty writes do not corrupt wal (#26223)
* test: reproducer for empty write issue

* fix: empty writes do not corrupt wal
2025-04-05 16:44:34 -04:00
Jackson Newhouse f881c5844b
chore: remove generic term from HttpApi (#26210) 2025-04-01 10:17:56 -07:00
praveen-influx e653f736d9
chore: couple of updates to fix cargo audit job (#26209)
* chore: couple of updates to fix cargo audit job

- remove humantime ignore in deny.toml
- update pyo3 to use 0.24.1 (https://rustsec.org/advisories/RUSTSEC-2025-0020.html)

* chore: moved pyo3 version to root cargo.toml
2025-04-01 15:36:01 +01:00
Trevor Hilton 9401137825
feat: handle graceful shutdown (#26197)
* feat: add influxdb3_shutdown crate

provides basic wait methods for unix/windows OS's

* feat: graceful shutdown

* docs: add rust docs and test to influxdb3_shutdown

Added rustdoc comments to types and methods in the influxdb3_shutdown
crate as well as a test that shows the ordering of a shutdown.
2025-03-31 09:58:40 -04:00
Jackson Newhouse 033e1176d8
fix: correctly handle multiple empty batches in JSON format results. (#26192) 2025-03-25 13:13:05 -07:00
Trevor Hilton 863a6d0b4a
feat: ack catalog update broadcast (#26118)
This creates a CatalogUpdateMessage type that is used to send
CatalogUpdates; this type performs the send on the oneshot Sender so
that the consumer of the message does not need to do so.

Subscribers to the catalog get a CatalogSubscription, which uses the
CatalogUpdateMessage type to ACK the message broadcast from the catalog.

This means that catalog message broadcast can fail, but this commit does
not provide any means of rolling back a catalog update.

A test was added to check that it works.
2025-03-17 20:20:07 -04:00
Trevor Hilton 0c8d17fb89
refactor: use repositories in catalog (#26135)
* refactor: use repository in catalog

The catalog was refactored to use identifiers on everything, and store
everything in a consistent structure. This structure makes use of the
`Repository` type that holds a `SerdeVecMap` of Id to Resource, along
with the next Id, and a bi-map of Id to resource name.

The `Repository` type is used at each level of the catalog where a
resource is stored.

This simplified repeated logic for snapshot'ing, insert and update of
resources in the catalog, as well as accessor methods for getting by id
or name, and mapping names to ids and vice-versa.

In addition, the process for catalog batch verification and permit was
altered so that the permit process induces a retry if the catalog was
updated while the catalog batch function was producing the batch, i.e, if
the catalog sequence incremented while the caller was waiting for a permit.
This eliminated the need for verifying the catalog batch after it had been
generated, and allows for a single path to apply a catalog batch after it
has been persisted to object store.

This assumes that the generation of the catalog batch implies validity.

Irelevant tests were removed.

Last and Distinct cache's now rely more heavily on Ids, though the proc-
essing engine still needs to switch over to use Ids for starting/stopping
triggers.
2025-03-13 22:42:18 -04:00
Jackson Newhouse 331f88533c
feat(processing_engine): Attach a node id to the processing engine and triggers for multi-node support compatibility. (#26136) 2025-03-13 16:10:02 -07:00
Michael Gattozzi 97b8c471f8
fix: Make sure JSON fully closes on empty records (#26130) 2025-03-12 14:01:40 -04:00
Trevor Hilton 72dc4458fd
chore: backport changes to catalog from enterprise (#26116)
* chore: backport changes to influxdb3_catalog crate

* chore: backport changes to influxdb3_cache crate

* chore: backport changes to influxdb3_write crate

* chore: backport changes to influxdb3_proc_eng crate

* chore: backport influxdb3 crate changes for catalog

* chore: backport changes to influxdb3_id crate

* chore: backport changes to influxdb3_wal crate

* chore: backport changes to influxdb3_clap_blocks crate

* chore: backport changes to influxdb3_client crate

* chore: backport influxdb3_server crate changes

* chore: fix after full backport

* fix: ordering of catalog broadcast
2025-03-11 12:11:51 -04:00
Jackson Newhouse 5fa417c3f0
feat: remove system-py (#26087)
* feat: remove system-py

* chore: allow Apache-2.0 WITH LLVM-exception license.
2025-03-10 11:10:33 -07:00
praveen-influx c724e06e3f
feat: query path instrumentation (#26106)
- spans added for buffer, parquet chunks along with number of files that
  are already in parquet cache along with the sql
2025-03-06 17:24:34 +00:00
Jackson Newhouse c930d9eef8
feat(processing_engine): error handling for triggers. (#26086) 2025-03-04 09:32:58 -08:00
Michael Gattozzi 1f72bfcc33
feat: Update to Rust 1.85 and 2024 Edition (#26046) 2025-02-20 14:58:07 -05:00
wayne 8daccb7ee8
fix: add SpanContext with TraceCollector to query_sql call (#26006) 2025-02-13 18:43:25 -07:00
Jackson Newhouse b0a24220c0
feat(processing_engine): Allow async plugin execution. (#25994) 2025-02-13 09:08:19 -08:00
Michael Gattozzi cf1dd5c831
fix: broken format for JSON queries and add tests (#25980)
In #25927 we missed that JSON queries were broken despite having some
tests use the format. This fixes JSON queries such that they now
properly contain a comma between RecordBatches. This commit also
includes tests for the formats that now stream data back (CSV, JSON, and
JSON Lines) so that we won't run into this issue again.
2025-02-06 20:25:18 -05:00
Trevor Hilton 1ac67a3ca1
fix: remove request uri from error log (#25981) 2025-02-06 14:47:22 -05:00
praveen-influx 6ca3411862
chore: remove log line (#25976) 2025-02-05 22:23:46 +00:00
praveen-influx fa69045f5b
chore: fixes for the build (#25973)
- missing clone/copy impl for NoProcessingEngine
- scope the visibility of a method
2025-02-05 17:44:24 +00:00
Jackson Newhouse fbcb9403c4
feat(processing_engine): log processing engine logging calls to sys events. (#25939) 2025-02-04 15:16:04 -08:00
wayne fa18b6d8da
refactor: DRY up influxdb3_client code (#25962)
* deduplicate QueryParams->QueryRequest and Format->QueryFormat
* move WriteParams into influxdb3_types crate
* DRY up client HTTP request handling code in *RequestBuilder.send
  methods.
* DRY up a bunch of other non-Builder http request handling
2025-02-04 08:54:37 -07:00
wayne 27653f5a76
fix: enable workspace lints on all crates, fix all lints (#25961) 2025-02-03 17:38:20 -07:00
wayne ca25ab6f70
fix: remove accidentally-commited dbg! macro (#25958) 2025-02-03 15:48:35 -07:00
wayne 0fffcc8c37
refactor: introduce influxdb3_types crate (#25946)
Partially fixes https://github.com/influxdata/influxdb/issues/24672

* move most HTTP req/resp types into `influxdb3_types` crate
* removes the use of locally-scoped request type structs from the `influxdb3_client` crate
* fix plugin dependency/package install bug
  * it looks like the `DELETE` http method was being used where `POST` was expected for `/api/v3/configure/plugin_environment/install_packages` and `/api/v3/configure/plugin_environment/install_requirements`
2025-02-03 11:28:47 -07:00
Michael Gattozzi 20fdc7b51b
feat: stream data back for CSV and JSON queries (#25927)
This commit allows us to stream data back for CSV and JSON formatted
queries. Prior to this we would buffer up all of the data in memory
before sending it back. Now we can make it so that we only buffer in
one RecordBatch at a time to reduce memory overhead.

Note that due to the way the APIs for writers work and for how Body in
hyper 0.14 works we can't use a streaming body that we can write too.
This in turn means we have to use a manually written Future state
machine, which works but is far from ideal.

Note this does not include the pretty and parquet files as streamable.
I'm attempting to get the pretty one to be streamable, but I don't think
that this one and parquet are as likely to be streamable back to the
user. In general we might want to discourage these formats from being
used.
2025-01-31 13:25:30 -05:00
wayne 05da40fa9b
fix: clarify table creation conflict error message (#25936)
Also include a basic CLI integration test to exemplify the new error message.
2025-01-30 13:08:35 -07:00
Jackson Newhouse 8840d99e9d
feat(processing_engine): integration with virtual environments. (#25895)
* feat(processing_engine): integration with virtual environments.

* feat: Initial scaffolding for environment managers (pip, pipx, uv).

* feat(processing_engine): CLI for package management, remove pipx support.

* feat(processing_engine): test installations in virtualenvs.

* feat(processing_engine): Automatically setup virtual environment on startup.
2025-01-28 15:30:17 -08:00
Paul Dix 9a5424693c
feat: Update trigger creation to validate plugin file present (#25918)
This updates trigger creation to load the plugin file before creating the trigger.

Another small change is to make Github references use filenames and paths identical to what they would be in the plugin-dir. This makes it a little easier to have the plugins repo local and develop against it and then be able to reference the same file later with gh: once it's up on the repo.
2025-01-27 12:31:49 -05:00
Paul Dix d49276a7fb
feat: Refactor plugins to only require creating trigger (#25914)
This refactors plugins and triggers so that plugins no longer need to be "created". Since plugins exist in either the configured local directory or on the Github repo, a user now only needs to create a trigger and reference the plugin filename.

Closes #25876
2025-01-27 11:26:46 -05:00
Michael Gattozzi 43e186d761
feat: add no_sync write_lp param for fast writes (#25902) 2025-01-24 13:34:38 -05:00
praveen-influx 4ef972eab4
feat: first stab at locally updating parquet cache (#25904)
* feat: first stab at locally updating parquet cache

closes: https://github.com/influxdata/influxdb/issues/25887

* refactor: use enums to separate out the modes

This commit introduced the `Immediate` and `Eventual` modes for
fulfilling the cache request. In immediate mode since the data is
readily available to be cached, we can avoid extra requests to object
store.

part of: https://github.com/influxdata/influxdb/issues/25887
2025-01-24 14:36:06 +00:00
Trevor Hilton d451ef0de6
refactor: writer-id to node-id (#25905) 2025-01-23 18:09:24 -05:00
Michael Gattozzi 63bd5096f5
feat: loosen 72 hour query/write restriction (#25890)
This commit does a few key things:

- Removes the 72 hour query and write restrictions in Core
- Limits the queries to a default number of parquet files. We chose 432
  as this is about 72 hours using default settings for the gen1
  timeblock
- The file limit can be increased, but the help text and error message
  when exceeded note that query performance will likely be degraded as
  a result.
- We warn users to use smaller time ranges if possible if they hit this
  query error

With this we eliminate the hard restriction we have in place, but
instead create a soft one that users can choose to take the performance
hit with. If they can't take that hit then it's recomended that they
upgrade to Enterprise which has the compactor built in to make
performant historical queries.
2025-01-23 10:02:26 -05:00
Trevor Hilton 44ca7a4d36
refactor: reduce catalog locks when getting chunks (#25896)
* refactor: reduce catalog locks when getting chunks

The main refactor was to change the ChunkContainer trait to use the
DatabaseSchema and TableDefinition types directly in the signature, vs.
the names, which then required an additional catalog lock and lookups for
both entities. This was already handled upstream in the QueryTable, so
there was no need to do the lookups again.

This required the addition of a test helper in influxdb3_write::test_helpers
that provides convenience methods for getting record batches from the
WriteBuffer. We have been implementing such a method manually in several
places, so this is nice to have it unified. This provides a blanket impl
so that anything implementing WriteBuffer gets the method.

Some other house cleaning was included.

* refactor: clean up test helpers in influxdb3_write

* refactor: pass original df filters forward with ChunkFilter

* chore: clippy
2025-01-22 14:38:46 -05:00
Paul Dix e87f8862b3
feat: Add request plugin capability (#25864)
* feat: Add request plugin capability

Adds the request plugin type. Triggers can be bound to an API endpoint at /api/v3/engine/<path>. Requests will get yielded to the plugin with the query parameters, request parameters, and request body.

I didn't implement the test endpoint for this plugin type as it seems much more natural for users to save the file and make a new request. Once #25863 is done it'll make it very easy.

Closes #25862

* chore: fix spelling in error message
2025-01-21 20:22:27 -08:00
praveen-influx 447f66d9a7
fix: allow default browser header (#25885)
Although the `format` in the request is used, the value coming
through the header is parsed earlier. So, when that lookup in
the header fails an error is returned (`InvalidMimeType`).

In this commit, there are extra checks to allow the default `Accept`
header values that come from the browser by defaulting it to `json`

closes: https://github.com/influxdata/influxdb/issues/25874
2025-01-21 13:43:07 +00:00
Jackson Newhouse 1d8d3d66fc
feat(processing_engine): Add cron plugins and triggers to the processing engine. (#25852)
* feat(processing_engine): Add cron plugins and triggers to the processing engine.

* feat(processing_engine): switch from 'cron plugin' to 'schedule plugin', use TimeProvider.

* feat(processing_engine): add test for test scheduled plugin.
2025-01-18 07:18:18 -05:00
Trevor Hilton 8ea6d65f8e
chore: bring back enterprise changes (#25858) 2025-01-17 13:24:32 -05:00
Jackson Newhouse 244ac567a8
fix(processing_engine): start triggers when server is started. (#25848) 2025-01-16 16:39:42 -08:00
Trevor Hilton b8a94488b5
feat: support v1 query API GROUP BY semantics (#25845)
This updates the v1 /query API hanlder to handle InfluxDB v1's unique
query response structure when GROUP BY clauses are provided.

The distinction is in the addition of a "tags" field to the emitted series
data that contains a map of the GROUP BY tags along with their distinct
values associated with the data in the "values" field.

This required splitting the QueryExecutor into two query paths for InfluxQL
and SQL, as this allowed for handling InfluxQL query parsing in advance
of query planning.

A set of snapshot tests were added to check that it all works.
2025-01-16 11:59:01 -05:00
Trevor Hilton db24a62658
refactor: change host-id to writer-id (#25804)
This changes the CLI arg `host-id` to `writer-id` to more accurately
indicate meaning.

This changes also goes through the codebase and changes struct fields,
methods, and variables to use the term `writer_id` or `writer_identifier_prefix`
instead of `host_id` etc., to make the meaning clear in the code.

This also changes the catalog serialization to use the field `writer_id`
instead of `host_id`, which is breaking change.
2025-01-12 11:40:47 -05:00
Paul Dix 491a37b0d4
feat: Update create plugin to use server file (#25803)
This updates the create plugin API and CLI so that it doesn't take the pugin code, but instead takes a file name of a file that must be in the plugin-dir of the server. It returns an error if the plugin-dir is not configured or if the file isn't there.

Also updates the WAL and catalog so that it doesn't store the plugin code directly. The code is read from disk one time when the plugin runs.

Closes #25797
2025-01-11 21:02:51 -05:00
praveen-influx 50963443a4
feat: introduce num wal files to keep (#25801)
* feat: introduce num wal files to keep

This commit allows a configurable number of wal files to be left behind
in OS. This is necessary as enterprise replicas rely on these files.

closes: https://github.com/influxdata/influxdb/issues/25788

* refactor: address PR feedback

* refactor: address PR comment
2025-01-12 00:33:13 +00:00
Trevor Hilton 0bdc2fa953
chore: patch enterprise back to core (#25798) 2025-01-11 17:26:41 -05:00
Paul Dix e421baf0bc
feat: include specific plugin error to http response (#25794) 2025-01-11 14:32:36 -05:00
Paul Dix bdcdb4f296
refactor: rename plugin activate/deactivate to enable/disable (#25793)
Closes #25789
2025-01-11 13:40:24 -05:00
Paul Dix e8422a240a
feat: Wire up arguments to wal plugin trigger (#25783)
This allows the user to specify arguments that will be passed to each execution of a wal plugin trigger. The CLI test was updated to check this end to end.

Closes #25655
2025-01-10 16:58:18 -05:00
Jackson Newhouse 954416d4d9
fix(processing_engine): enable downstream system-py feature (#25784) 2025-01-10 13:44:24 -08:00
Paul Dix 0da0785960
feat: Finish wiring up WAL plugin with trigger (#25781)
This updates the WAL so that it can have new file notifiers added that will get updated when the wal flushes. The processing engine now implements the WALNotifier trait.

I've updated the CLI test for creating a trigger to run and end-to-end test that defines a plugin, creates a trigger, writes data into the database, triggering the plugin, which writes summary statistics back into the database in a different table. The test queries the destination table to confirm that the plugin worked.
2025-01-10 12:56:49 -05:00
Trevor Hilton c71dafc313
refactor: rename metadata cache to distinct value cache (#25775) 2025-01-10 08:48:51 -05:00
Paul Dix 7230148b58
feat: Update WAL plugin for new structure (#25777)
* feat: Update WAL plugin for new structure

This ended up being a very large change set. In order to get around circular dependencies, the processing engine had to be moved into its own crate, which I think is ultimately much cleaner.

Unfortunately, this required changing a ton of things. There's more testing and things to add on to this, but I think it's important to get this through and build on it.

Importantly, the processing engine no longer resides inside the write buffer. Instead, it is attached to the HTTP server. It is now able to take a query executor, write buffer, and WAL so that the full range of functionality of the server can be exposed to the plugin API.

There are a bunch of system-py feature flags littered everywhere, which I'm hoping we can remove soon.

* refactor: PR feedback
2025-01-10 05:52:33 -05:00
Paul Dix 2d18a61949
feat: Add query API to Python plugins (#25766)
This ended up being a couple things rolled into one. In order to add a query API to the Python plugin, I had to pull the QueryExecutor trait out of server into a place so that the python crate could use it.

This implements the query API, but also fixes up the WAL plugin test CLI a bit. I've added a test in the CLI section so that it shows end-to-end operation of the WAL plugin test API and exercise of the entire Plugin API.

Closes #25757
2025-01-09 20:13:20 -05:00
Trevor Hilton 23866ef3d1
fix: /query API emits empty result as JSON (#25769) 2025-01-09 09:26:30 -05:00
praveen-influx 6e2e39cd4c
feat: snapshot when wal buffer is empty (#25765)
* feat: snapshot when wal buffer is empty

- This commit changes the functionality to allow snapshots to happen even when
  wal buffer is empty. For snapshots wal periods are still required but
  not the wal buffer. To allow this, we write a no-op into wal file with
  snapshot details. This enables force snapshotting functionality

closes: https://github.com/influxdata/influxdb/issues/25685

* refactor: address PR feedback
2025-01-09 12:12:37 +00:00
Trevor Hilton dfc853d903
feat: handle params in request body for /query API (#25762)
Closes #25749

This changes the `/query` API handler so that the parameters can be passed in either the request URI or in the request body for either a `GET` or `POST` request.

Parameters can be specified in the URI, the body, or both; if they are specified in both places, those in the body will take precedent.

Error variants in the HTTP server code related to missing request parameters were updated to return `400` status.
2025-01-07 19:53:29 -05:00
Trevor Hilton 3efd49066e
fix: bind to correct port for e2e tests (#25758)
* fix: bind to correct port for e2e tests

This also fixes up some log messages on server start for naming

* chore: do notpass value in TEST_LOG env var to CI tests
2025-01-07 12:23:59 -05:00
Trevor Hilton 6524f383ba
feat: show databases CLI/API (#25748)
_Follows #25737 (keeping in draft until that merges)_

Closes #25745 

This PR provides both a CLI and underlying API for listing databases in the InfluxDB 3 Core server. Details are below.

There was already a method to list databases for the query executor for InfluxQL; this works by exposing that via the `HttpApi` in `influxdb3_server`.

However, one thing that we may address is that the query result for that uses `iox::database` as the column name. If we are removing references to `iox`, then we may want to just have it as `database`. I left it as is, for now, because I wanted to keep code churn down and wasn't sure why we use that prefix in the first place for the `SHOW DATABASES` and `SHOW RETENTION POLICIES` InfluxQL queries.

## Details

### CLI

This PR provides the `influxdb3 show` CLI:
```
influxdb3 show -h
List resources on the InfluxDB 3 Core server

Usage: influxdb3 show <COMMAND>

Commands:
  databases  List databases
  help       Print this message or the help of the given subcommand(s)

Options:
  -h, --help  Print help information
```
with the ability to list databases:
```
influxdb3 show databases -h
List databases

Usage: influxdb3 show databases [OPTIONS]

Options:
  -H, --host <HOST_URL>         The host URL of the running InfluxDB 3 Core server [env: INFLUXDB3_HOST_URL=] [default: http://127.0.0.1:8181]
      --token <AUTH_TOKEN>      The token for authentication with the InfluxDB 3 Core server [env: INFLUXDB3_AUTH_TOKEN=]
      --show-deleted            Include databases that were marked as deleted in the output
      --format <OUTPUT_FORMAT>  The format in which to output the list of databases [default: pretty] [possible values: pretty, json, json_lines, csv]
  -h, --help                    Print help information
```
Since this uses the query executor, we can pass a `--format` argument to get the output as JSON, CSV, or JSONL, but by default, it uses the `pretty` format:
```
influxdb3 show databases
+---------------+
| iox::database |
+---------------+
| bar           |
+---------------+
```
The `--show-deleted` flag will have the `deleted` column displayed as well as any databases that have been marked as deleted:
```
influxdb3 show databases --show-deleted
+---------------------+---------+
| iox::database       | deleted |
+---------------------+---------+
| bar                 | false   |
| foo-20250105T202949 | true    |
+---------------------+---------+
```

### API

The API to list databases can be invoked via:
```
GET /api/v3/configure/database
```
with optional parameters:
* `format`: `pretty`, `json`, `csv`, `parquet`, or `jsonl`
* `show_deleted`: `bool`, defaults to `false`

Note that `database` is singular in the API endpoint, to be consistent with the other database related create/delete API endpoints. We could change it to be plural `databases` if that is the convention we want to go with.
2025-01-06 21:08:12 -05:00
Michael Gattozzi f793d31f63
feat: Cleanup CLI flags for InfluxDB 3 Core (#25737)
This makes quite a few major changes to our CLI and how users interact
with it:

1. All commands are now in the form <verb> <noun> this was to make the
   commands consistent. We had last-cache as a noun, but serve as a
   verb in the top level. Given that we could only create or delete
   All noun based commands have been move under a create and delete
   command
2. --host short form is now -H not -h which is reassigned to -h/--help
   for shorter help text and is in line with what users would expect
   for a CLI
3. Only the needed items from clap_blocks have been moved into
   `influxdb3_clap_blocks` and any IOx specific references were changed
   to InfluxDB 3 specific ones
4. References to InfluxDB 3.0 OSS have been changed to InfluxDB 3 Core
   in our CLI tools
5. --dbname has been changed to --database to be consistent with --table
   in many commands. The short -d flag still remains. In the create/
   delete command for the database however the name of the database is
   a positional arg

   e.g. `influxbd3 create database foo` rather than
        `influxdb3 database create --dbname foo`
6. --table has been removed from the delete/create command for tables
   and is now a positional arg much like database
7. clap_blocks was removed as dependency to avoid having IOx specific
   env vars
8. --cache-name is now an optional positional arg for last_cache and meta_cache
9. last-cache/meta-cache commands are now last_cache and meta_cache respectively

Unfortunately we have quite a few options to run the software and I
couldn't cut down on them, but at least with this commands and options
will be more discoverable and we have full control over our CLI options
now.

Closes #25646
2025-01-06 18:51:55 -05:00
Paul Dix 1ce6a24c3f
feat: Implement WAL plugin test API (#25704)
* feat: Implement WAL plugin test API

This implements the WAL plugin test API. It also introduces a new API for the Python plugins to be called, get their data, and call back into the database server.

There are some things that I'll want to address in follow on work:
* CLI tests, but will wait on #25737 to land for a refactor of the CLI here
* Would be better to hook the Python logging to call back into the plugin return state like here: https://pyo3.rs/v0.23.3/ecosystem/logging.html#the-python-to-rust-direction
* We should only load the LineBuilder interface once in a module, rather than on every execution of a WAL plugin
* More tests all around

But I want to get this in so that the actual plugin and trigger system can get udated to build around this model.

* refactor: PR feedback
2025-01-06 17:32:17 -05:00
Michael Gattozzi ccda3dd3a9
feat: remove required field restriction for tables (#25738)
This commit removes the required fields restriction when using the CLI
or the API to create a new table. As users can't write via the line
protocol without a field this is fine and the schema will be updated on
write. This expands the test to check for the correct response code now
and make sure that we can both query the empty table and write new data
to it.

Closes #25735
2025-01-03 18:10:56 -05:00
Jackson Newhouse 7aa8f41268
feat(processing_engine): Add CLI support for plugins and triggers (#25731) 2025-01-03 12:11:52 -08:00
Trevor Hilton 7c7e5d77df
feat: remove table_name clause requirement for parquet files system table (#25733)
Updated the tests to check for correct behaviour on queries to the
system.parquet_files table.
2025-01-02 16:59:32 -05:00
Michael Gattozzi 636503ff46
fix: Return a 422 for limits on a partial write (#25729)
Prior to this change we would error correctly with a 422 if a limit was
hit. However, we would not send back the correct error in the case of a
limit being hit that caused a partial write. This change fixes that by
checking the error messages for failed lines and if one is found that
caused a limit to be hit, then a 422 is returned rather than a 400 as
we would have been able to process the line otherwise, but the limit was
hit instead

Closes #25208
2025-01-02 14:01:23 -05:00
Jackson Newhouse 29dacc318a
feat(processing_engine): Add REST API endpoints for activating and deactivating triggers. (#25711) 2025-01-02 09:23:18 -08:00
Trevor Hilton de227b95d9
refactor: cleanup v3 write API and series key method on catalog (#25723)
Store the series key column names on the TableDefinitin in catalog so
looking up the series key by column names is more efficient

Remove the /api/v3/write API and related code/tests
2024-12-30 09:32:54 -05:00
praveen-influx 96a580f365
refactor: porting changes in pro to oss (#25712)
- query_executor is moved to it's own module so that it's easier to swap
  the setup for pro.
- system_tables setup is also modified
2024-12-27 15:02:22 +00:00
Michael Gattozzi c764d37636
feat: Add json lines support to query output (#25698) 2024-12-20 14:57:19 -05:00
Michael Gattozzi 048e45fff9
fix: change 200 to 204 for compatibility of v1/v2 (#25699) 2024-12-20 14:56:18 -05:00
Michael Gattozzi 2a132f1edf
fix: change failed query with missing db to 404 (#25693) 2024-12-20 11:53:13 -05:00
Trevor Hilton d10ad87f2c
feat: write metrics (#25692)
Added prometheus metrics to track lines written and bytes written per
database. The write buffer does the tracking after validation of incoming
line protocol.

Tests added to verify.
2024-12-20 11:02:36 -05:00
Michael Gattozzi e51bea65b4
feat: create DB and Tables via REST and CLI (#25687)
* feat: create DB and Tables via REST and CLI

This commit does a few things:

1. It brings the database command naming scheme for types inline with
   the rest of the CLI types
2. It brings the table command naming scheme for types inline with
   the rest of the CLI types
3. Adds tests to check that the num of dbs is not exceeded and that you
   cannot create more than one database with a given name.
4. Adds tests to check that you can create a table and put data into it
   and querying it
5. Adds tests for the CLI for both the database and table commands
6. It creates an endpoint to create databases given a JSON blob
7. It creates an endpoint to create tables given a JSON blob

With this users can now create a database or table without first needing
to write to the database via the line protocol!

Closes #25640
Closes #25641
2024-12-19 16:01:34 -05:00
Jackson Newhouse 8bfccb74ab
feat(processing_engine): Runtime and write-back improvements (#25672)
* Move processing engine invocation to a seperate tokio task.
* Support writing back line protocol from python via insert_line_protocol().
* Update structs to work with bincode.
2024-12-17 16:38:12 -08:00
Michael Gattozzi 90b9071297
fix: Change references of Edge into OSS (#25661)
This changes the code to reference InfluxDB 3 OSS rather than Edge which
had been it's original name when we first started the project. With this
we now have the code reflect what we are actually calling it. On top of
this the long help text has been changed to give advice about how to
actually run the code now with the bare minimum set of flags needed now
as `influxdb serve` is no longer a viable command on it's own.

Closes #25649
2024-12-16 11:24:35 -05:00
Jackson Newhouse 486d79d801
feat(processing_engine): initial implementation of Processing Engine plugins and triggers (#25639) 2024-12-13 14:11:38 -08:00
Michael Gattozzi 9292a3213d
feat: Significantly decrease startup times for WAL (#25643)
* feat: add startup time to logging output

This change adds a startup time counter to the output when starting up
a server. The main purpose of this is to verify whether the impact of
changes actually speeds up the loading of the server.

* feat: Significantly decrease startup times for WAL

This commit does a few important things to speedup startup times:
1. We avoid changing an Arc<str> to a String with the series key as the
   From<String> impl will call with_column which will then turn it into
   an Arc<str> again. Instead we can just call `with_column` directly
   and pass in the iterator without also collecting into a Vec<String>
2. We switch to using bitcode as the serialization format for the WAL.
   This significantly reduces startup time as this format is faster to
   use instead of JSON, which was eating up massive amounts of time.
   Part of this change involves not using the tag feature of serde as
   it's currently not supported by bincode
3. We also parallelize reading and deserializing the WAL files before
   we then apply them in order. This reduces time waiting on IO and we
   eagerly evaluate each spawned task in order as much as possible.

This gives us about a 189% speedup over what we were doing before.

Closes #25534
2024-12-12 11:27:51 -05:00
Trevor Hilton 37219af9d4
feat: track parquet cache metrics (#25632)
* feat: parquet cache metrics

* feat: track parquet cache metrics

Adds metrics to track the following in the in-memory parquet cache:
* cache size in bytes (also included a fix in the calculation of that)
* cache size in n files
* cache hits
* cache misses
* cache misses while the oracle is fetching a file

A test was added to check this functionality

* refactor: clean up logic and fix cache removal tracking error

Some logic and naming was cleaned up and the boolean to optionally track
metrics on entry removal was removed, as it was incorrect in the first place:
a fetching entry still has a size, which counts toward the size of the
cache. So, this makes is such that anytime an entry is removed, whether
its state is success or fetching, its size will be decremented from
the cache size metrics.

The sizing caclulations were made to be correct, and the cache metrics
test was updated with more thurough assertions
2024-12-10 09:32:15 -05:00
Trevor Hilton 0bfef47ff9
refactor: move parquet cache to influxdb3_cache crate (#25630) 2024-12-09 11:56:52 -05:00
Trevor Hilton 161c8b4eda
refactor: remove query concurrency limit (#25629) 2024-12-09 11:06:14 -05:00
Trevor Hilton 154ff7da23
feat: LastCacheExec to track predicate pushdown in last cache queries (#25621) 2024-12-06 10:53:19 -08:00
Trevor Hilton 9b87cd7a65
refactor: move last cache to influxdb3_cache crate (#25620)
Moved all of the last cache implementation into the `influxdb3_cache`
crate. This also splits out the implementation into three modules:
- `cache.rs`: the core cache implementation
- `provider.rs`: the cache provider used by the database to hold multiple
  caches.
- `table_function.rs`: same as before, holds the DataFusion impls

Tests were preserved and moved to `mod.rs`, however, they were updated to
not rely on the WriteBuffer implementation, and instead use the types in
the `influxdb3_cache::last_cache` module directly. This simplified the
test code, while not changing any of the test assertions at all.
2024-12-05 14:04:25 -05:00
Trevor Hilton dbb1f55b5e
chore: update core for latest sync (#25617) 2024-12-04 14:11:13 -05:00
Michael Gattozzi d2fbd65a44
feat: Deny extra tags on write APIs (#25596)
This commit does three important major changes:

1. We will deny writes to the v1, v2, and v3 write apis that add new tags in
   subsequent writes after the first write
2. We make every table have a series key by default now
3. We enfore sorting order by the series key which is the order the keys came in

With these changes we have consistentcy across the various write apis and can
make optimizations and future features with the assumption we have a series key.

Closes #25585
2024-12-03 12:10:26 -05:00
praveen-influx 43755c2d9c
feat: sys events store added (#25603)
This commit introduces basic store for sys events and the backing ring
buffer. Since the buffer needs to hold arbitrary data, it uses `Box<dyn
Any>`

closes: https://github.com/influxdata/influxdb/issues/25581
2024-12-02 10:55:37 +00:00
Trevor Hilton a01fd16d4d
fix: plan queries on DF threadpool to not block IO in REST API (#25604) 2024-11-29 16:03:17 -05:00
Trevor Hilton 9ead1dfe4b
feat: meta_caches system table (#25593)
This adds a new system table "meta_caches" that allows users to view the
state of their metadata caches on a per-db basis

An integration test was added to verify that it works.
2024-11-28 08:57:02 -05:00
Trevor Hilton 234d37329a
feat: metacache REST APIs to create and delete (#25587) 2024-11-27 08:41:46 -05:00
praveen-influx bfa0e71558
feat: make query executor as trait object (#25591)
* feat: make query executor as trait object

This commit moves `QueryExecutorImpl` behind a `dyn` (trait object) as
we have other impls in core for `QueryExecutor` and this will keep both
pro and OSS traits in sync

* chore: fix cargo audit failures

- address https://rustsec.org/advisories/RUSTSEC-2024-0399.html by
running `cargo update --precise 0.23.18 --package rustls@0.23.14`

- address yanked version of `url` crate (2.5.3) by running
`cargo update -p url`
2024-11-26 17:18:22 +00:00
praveen-influx 3cde24feb4
feat: delete table (#25572)
This commit allows deleting (soft) a table. For an user, following
command will allow soft deleting a table (bar) in db (foo)

```
influxdb3 table delete --dbname foo --table bar --host $host
```

- Added `soft_delete_table` to `DatabaseManager` trait, which already
  hosts `soft_delete_database` method. The code roughly follows the same
  flow as db delete. Although like db schema, it does clone on write
  because the reference is behind an Arc, `Arc::make_mut` is used in
  this change.
- Moved db delete related cli parser under "manage" module that has both
  db and table delete functionality
- Some minor tidyups (removing unused methods, renaming method so that
  the order in name matches actual return type eg. `table_id_and_schema`,
  should return (id, schema) and not (schema, id))

closes: https://github.com/influxdata/influxdb/issues/25561
2024-11-22 08:42:45 +00:00
praveen-influx 33c2d47ba9
feat: drop/delete database (#25549)
* feat: drop/delete database

This commit allows soft deletion of database using `influxdb3 database
delete <db_name>` command. The write buffer and last value cache are
cleared as well.

closes: https://github.com/influxdata/influxdb/issues/25523

* feat: reuse same code path when deleting database

- In previous commit, the deletion of database immediately triggered
  clearing last cache and query buffer. But on restarts same logic had
  to be repeated to allow deleting database when starting up. This
  commit removes immediate deletion by explicitly calling necessary
  methods and moves the logic to `apply_catalog_batch` which already
  applies `CatalogOp` and also clearing cache and buffer in
  `buffer_ops` method which has hooks to call other places.

closes: https://github.com/influxdata/influxdb/issues/25523

* feat: use reqwest query api for query param

Co-authored-by: Trevor Hilton <thilton@influxdata.com>

* feat: include deleted flag in DatabaseSnapshot

- `DatabaseSchema` serialization/deserialization is delegated to
 `DatabaseSnapshot`, so the `deleted` flag should be included in
 `DatabaseSnapshot` as well.
- insta test snapshots fixed

closes: https://github.com/influxdata/influxdb/issues/25523

* feat: address PR comments + tidy ups

---------

Co-authored-by: Trevor Hilton <thilton@influxdata.com>
2024-11-19 16:08:14 +00:00
praveen-influx 814eb31309
chore: update core deps (#25532)
* chore: update core deps

- arrow/parquet deps are patched (as in core)
- three specific code changes to cope with changes in core crates
  - TransitionPartitionId, use `from_parts` instead of `new`
  - arrow buffers can take &[u8] directly without `to_vec()`/`vec!`
    (used only in tests)
  - `schema` and `influxdb_line_protocol` crates need `v3` feature enabled

* chore: update deny.toml

* chore: formatting and deny toml changes

Unicode-3.0 license is added to allowed licenses list, without it
end up with 19 errors (`zerovec`, `zerovec-derive` etc.)

* chore: address PR feedback

- move enabling v3 feature to root Cargo.toml
- added the upstream PR for datafusion-common that introduced RUSTSEC-2024-0384
2024-11-12 16:07:31 +00:00
praveen-influx c2b8a3a355
feat: add column names to last cache sys table (#25521)
* feat: add column names to last cache sys table

closes: https://github.com/influxdata/influxdb/issues/25511

* feat: move all `get_by_id` methods to take reference in schema
2024-11-08 16:08:30 +00:00
Trevor Hilton 391b67f9ab
feat: configurable last cache eviction (#25520)
* refactor: make last cache eviction optional

This changes how the last cache is evicted. It will no longer run eviction
on writes to the cache, instead, there is an optional method to create a
last cache provider that will run eviction in a background task on a specified
interval.

Otherwise, when records are produced from the cache, only those that have
not expired will be produced.

This should reduce locks on the cache and hopefully improve performance.

* feat: configurable last cache eviction interval

* docs: clean up var names, code docs, and comments
2024-11-06 09:59:17 -05:00