Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
influxdb/influxdb3_server/Cargo.toml

101 lines
2.7 KiB
TOML
Raw Normal View History

feat: Add basic Edge server structure (#24552) * WIP: basic influxdb3 command and http server * WIP: write lp, buffer, query out * WIP: test write & query on influxdb3_server, fix warnings * WIP: pull write buffer and catalog into separate crate * WIP: sketch out types used for write: buffer, wal, persister * WIP: remove a bunch of old IOx stuff and fmt
2024-01-08 16:50:59 +00:00
[package]
name = "influxdb3_server"
version.workspace = true
authors.workspace = true
edition.workspace = true
license.workspace = true
[dependencies]
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
# Core Crates
authz.workspace = true
data_types.workspace = true
datafusion_util.workspace = true
influxdb-line-protocol.workspace = true
iox_catalog.workspace = true
feat: support v1 and v2 write APIs (#24793) feat: support v1 and v2 write APIs This adds support for two APIs: /write and /api/v2/write. These implement the v1 and v2 write APIs, respectively. In general, the difference between these and the new /api/v3/write_lp API is in the request parsing. We leverage the WriteRequestUnifier trait from influxdb3_core to handle parsing of v1 and v2 HTTP requests, to keep the error handling at that level consistent with distributed versions of InfluxDB 3.0. Specifically, we use the SingleTenantRequestUnifier implementation of the trait. Changes: - Addition of two new routes to the route_request method in influxdb3_server::http to serve /write and /api/v2/write requests. - Database name validation was updated to handle cases where retention policies may be passed in /write requests, and to also reject empty names. A unit test was added to verify the validate_db_name function. - HTTP request authorization in the router will extract the full Authorization header value, and store it in the request extensions; this is used in the write request parsing from the core iox_http crate to authorize write requests. - E2E tests to verify correct HTTP request parsing / response behaviour for both /write and /api/v2/write APIs - E2E tests to check that data sent in through /write and /api/v2/write can be queried back
2024-03-28 17:33:17 +00:00
iox_http.workspace = true
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
iox_query.workspace = true
feat: add the `api/v3/query_influxql` API (#24696) feat: add query_influxql api This PR adds support for the /api/v3/query_influxql API. This re-uses code from the existing query_sql API, but some refactoring was done to allow for code re-use between the two. The main change to the original code from the existing query_sql API was that the format is determined up front, in the event that the user provides some incorrect Accept header, so that the 400 BAD REQUEST is returned before performing the query. Support of several InfluxQL queries that previously required a bridge to be executed in 3.0 was added: SHOW MEASUREMENTS SHOW TAG KEYS SHOW TAG VALUES SHOW FIELD KEYS SHOW DATABASES Handling of qualified measurement names in SELECT queries (see below) This is accomplished with the newly added iox_query_influxql_rewrite crate, which provides the means to re-write an InfluxQL statement to strip out a database name and retention policy, if provided. Doing so allows the query_influxql API to have the database parameter optional, as it may be provided in the query string. Handling qualified measurement names in SELECT The implementation in this PR will inspect all measurements provided in a FROM clause and extract the database (DB) name and retention policy (RP) name (if not the default). If multiple DB/RP's are provided, an error is thrown. Testing E2E tests were added for performing basic queries against a running server on both the query_sql and query_influxql APIs. In addition, the test for query_influxql includes some of the InfluxQL-specific queries, e.g., SHOW MEASUREMENTS. Other Changes The influxdb3_client now has the api_v3_query_influxql method (and a basic test was added for this)
2024-03-01 17:27:38 +00:00
iox_query_params.workspace = true
iox_query_influxql.workspace = true
feat: add the `system.queries` table (#24992) The system.queries table is now accessible, when queries are initiated in debug mode, which is not currently enabled via the HTTP API, therefore this is not yet accessible unless via the gRPC interface. The system.queries table lists all queries in the QueryLog on the QueryExecutorImpl.
2024-05-17 16:04:25 +00:00
iox_system_tables.workspace = true
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
iox_time.workspace = true
metric.workspace = true
metric_exporters.workspace = true
observability_deps.workspace = true
schema.workspace = true
service_common.workspace = true
service_grpc_flight.workspace = true
trace.workspace = true
trace_exporters.workspace = true
trace_http.workspace = true
tracker.workspace = true
# Local Deps
feat: metacache REST APIs to create and delete (#25587)
2024-11-27 13:41:46 +00:00
influxdb3_cache = { path = "../influxdb3_cache" }
refactor: Move Catalog into influxdb3_catalog crate (#25210) * refactor: Move Catalog into influxdb3_catalog crate This moves the catalog and its serialization logic into its own crate. This is a precursor to recording more catalog modifications into the WAL. Fixes #25204 * fix: cargo update * fix: add version = 2 to deny.toml * fix: update deny.toml * fix: add CCO to deny.toml
2024-08-02 20:04:12 +00:00
influxdb3_catalog = { path = "../influxdb3_catalog" }
feat: Implement WAL plugin test API (#25704) * feat: Implement WAL plugin test API This implements the WAL plugin test API. It also introduces a new API for the Python plugins to be called, get their data, and call back into the database server. There are some things that I'll want to address in follow on work: * CLI tests, but will wait on #25737 to land for a refactor of the CLI here * Would be better to hook the Python logging to call back into the plugin return state like here: https://pyo3.rs/v0.23.3/ecosystem/logging.html#the-python-to-rust-direction * We should only load the LineBuilder interface once in a module, rather than on every execution of a WAL plugin * More tests all around But I want to get this in so that the actual plugin and trigger system can get udated to build around this model. * refactor: PR feedback
2025-01-06 22:32:17 +00:00
influxdb3_client = { path = "../influxdb3_client" }
feat: Add u32 ID for Databases (#25302) * feat: Remove lock for FileId tests Since we now are using cargo-nextest in CI we can remove the locks used in the FileId tests to make sure that we have no race conditions * feat: Add u32 ID for Databases This commit adds a new DbId for databases. It also updates paths to use that id as part of the name. When starting up the WriteBuffer we apply the DbId from the persisted snapshot much like we do for ParquetFileId's This introduces the influxdb3_id crate to avoid circular deps with ids. The ParquetFileId should also be moved into this crate, but it's outside the scope of this change. Closes #25301
2024-09-18 15:44:04 +00:00
influxdb3_id = { path = "../influxdb3_id" }
feat: Add query API to Python plugins (#25766) This ended up being a couple things rolled into one. In order to add a query API to the Python plugin, I had to pull the QueryExecutor trait out of server into a place so that the python crate could use it. This implements the query API, but also fixes up the WAL plugin test CLI a bit. I've added a test in the CLI section so that it shows end-to-end operation of the WAL plugin test API and exercise of the entire Plugin API. Closes #25757
2025-01-10 01:13:20 +00:00
influxdb3_internal_api = { path = "../influxdb3_internal_api" }
feat: `/ping` API to serve version and revision (#24864) * feat: /ping API to serve version The /ping API was added, which is served at GET and POST methods. The API responds with a JSON body containing the version and revision of the build. A new crate was added, influxdb3_process, which takes the process_info.rs module from the influxdb3 crate, and puts it in a separate crate so that other crates (influxdb3_server) can depend on it. This was needed in order to have access to the version and revision values, which are generated at build time, in the HTTP API code of influxdb3_server. A E2E test was added to check that /ping works. E2E TestServer can now have logs emitted using the TEST_LOG environment variable.
2024-04-01 20:57:10 +00:00
influxdb3_process = { path = "../influxdb3_process", default-features = false }
refactor: implement new wal and refactor write buffer (#25196) * feat: refactor WAL and WriteBuffer There is a ton going on here, but here are the high level things. This implements a new WAL, which is backed entirely by object store. It then updates the WriteBuffer to be able to work with how the new WAL works, which also required an update to how the Catalog is modified and persisted. The concept of Segments has been removed. Previously there was a separate WAL per segment of time. Instead, there is now a single WAL that all writes and updates flow into. Data within the write buffer is organized by Chunk(s) within tables, which is based on the timestamp of the row data. These are known as the Level0 files, which will be persisted as Parquet into object store. The default chunk duration for level 0 files is 10 minutes. The WAL is written as single files that get created at the configured WAL flush interval (1s by default). After a certain number of files have been created, the server will attempt to snapshot the WAL (default is to snapshot the first 600 files of the WAL after we have 900 total, i.e. snapshot 10 minutes of WAL data). The design goal with this is to persist 10 minute chunks of data that are no longer receiving writes, while clearing out old WAL files. This works if data getting written in around "now" with no more than 5 minutes of delay. If we continue to have delayed writes, a snapshot of all data will be forced in order to clear out the WAL and free up memory in the buffer. Overall, this structure of a single wal, with flushes and snapshots and chunks in the queryable buffer led to a simpler setup for the write buffer overall. I was able to clear out quite a bit of code related to the old segment organization. Fixes #25142 and fixes #25173 * refactor: address PR feedback * refactor: wal to replay and background flush on new * chore: remove stray println
2024-08-01 19:04:15 +00:00
influxdb3_wal = { path = "../influxdb3_wal"}
feat: Add u32 ID for Databases (#25302) * feat: Remove lock for FileId tests Since we now are using cargo-nextest in CI we can remove the locks used in the FileId tests to make sure that we have no race conditions * feat: Add u32 ID for Databases This commit adds a new DbId for databases. It also updates paths to use that id as part of the name. When starting up the WriteBuffer we apply the DbId from the persisted snapshot much like we do for ParquetFileId's This introduces the influxdb3_id crate to avoid circular deps with ids. The ParquetFileId should also be moved into this crate, but it's outside the scope of this change. Closes #25301
2024-09-18 15:44:04 +00:00
influxdb3_write = { path = "../influxdb3_write" }
feat: add the `api/v3/query_influxql` API (#24696) feat: add query_influxql api This PR adds support for the /api/v3/query_influxql API. This re-uses code from the existing query_sql API, but some refactoring was done to allow for code re-use between the two. The main change to the original code from the existing query_sql API was that the format is determined up front, in the event that the user provides some incorrect Accept header, so that the 400 BAD REQUEST is returned before performing the query. Support of several InfluxQL queries that previously required a bridge to be executed in 3.0 was added: SHOW MEASUREMENTS SHOW TAG KEYS SHOW TAG VALUES SHOW FIELD KEYS SHOW DATABASES Handling of qualified measurement names in SELECT queries (see below) This is accomplished with the newly added iox_query_influxql_rewrite crate, which provides the means to re-write an InfluxQL statement to strip out a database name and retention policy, if provided. Doing so allows the query_influxql API to have the database parameter optional, as it may be provided in the query string. Handling qualified measurement names in SELECT The implementation in this PR will inspect all measurements provided in a FROM clause and extract the database (DB) name and retention policy (RP) name (if not the default). If multiple DB/RP's are provided, an error is thrown. Testing E2E tests were added for performing basic queries against a running server on both the query_sql and query_influxql APIs. In addition, the test for query_influxql includes some of the InfluxQL-specific queries, e.g., SHOW MEASUREMENTS. Other Changes The influxdb3_client now has the api_v3_query_influxql method (and a basic test was added for this)
2024-03-01 17:27:38 +00:00
iox_query_influxql_rewrite = { path = "../iox_query_influxql_rewrite" }
feat: sys events store added (#25603) This commit introduces basic store for sys events and the backing ring buffer. Since the buffer needs to hold arbitrary data, it uses `Box<dyn Any>` closes: https://github.com/influxdata/influxdb/issues/25581
2024-12-02 10:55:37 +00:00
influxdb3_sys_events = { path = "../influxdb3_sys_events" }
feat(telemetry): adds reads and writes (#25409) - instrumented code to get read and write measurement - introduced EventsBucket for collection of reads/writes - sampler now samples every minute for all metrics (including reads/writes) - other tidy ups closes: https://github.com/influxdata/influxdb/issues/25372
2024-10-01 17:34:00 +00:00
influxdb3_telemetry = { path = "../influxdb3_telemetry" }
feat: Add basic Edge server structure (#24552) * WIP: basic influxdb3 command and http server * WIP: write lp, buffer, query out * WIP: test write & query on influxdb3_server, fix warnings * WIP: pull write buffer and catalog into separate crate * WIP: sketch out types used for write: buffer, wal, persister * WIP: remove a bunch of old IOx stuff and fmt
2024-01-08 16:50:59 +00:00
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
# crates.io Dependencies
feat: support v1 query API (#24746) feat: support the v1 query API This PR adds support for the `/api/v1/query` API, which is meant to serve the original InfluxDB v1 query API, to serve single statement `SELECT` and `SHOW` queries. The response, which is returned as JSON, can be chunked via the `chunked` and optional `chunk_size` parameters. An optional `epoch` parameter can be supplied to have `time` column timestamps converted to a UNIX epoch with the given precision. ## Buffering The response is buffered by default, but if the `chunked` parameter is not supplied, or is passed as `false`, then the entire query result will be buffered into memory before being returned in the response. This is how the original API behaves, so we are replicating that here. When `chunked` is passed as `true`, then the response will be a stream of chunks, where each chunk is a self-contained response, with the same structure as that of the non-chunked response. Chunks are split up by the provided `chunk_size`, or by series, i.e., measurement, which ever comes first. The default chunk size is 10,000 rows. Buffering is implemented with the `QueryResponseStream` and `ChunkBuffer` types, the former implements the `Stream` trait, which allows it to be streamed in the HTTP response directly with `hyper`'s `Body::wrap_stream`. The `QueryResponseStream` is a wrapper around the inner arrow `RecordBatchStream`, which buffers the streamed `RecordBatch`es according to the requested chunking parameters. ## Testing Two new E2E tests were added to test basic query functionality and chunking behaviour, respectively. In addition, some manual testing was done to verify that the InfluxDB Grafana plugin works with this API.
2024-03-15 17:38:15 +00:00
anyhow.workspace = true
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
arrow.workspace = true
refactor: v1 recordbatch to json (#25085) * refactor: refactor serde json to use recordbatch * fix: cargo audit with cargo update * fix: add timestamp datatype * fix: add timestamp datatype * fix: apply feedbacks * fix: cargo audit with cargo update * fix: add timestamp datatype * fix: apply feedbacks * refactor: test data conversion
2024-07-05 13:21:40 +00:00
arrow-array.workspace = true
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
arrow-csv.workspace = true
feat: support FlightSQL in 3.0 (#24678) * feat: support FlightSQL by serving gRPC requests on same port as HTTP This commit adds support for FlightSQL queries via gRPC to the influxdb3 service. It does so by ensuring the QueryExecutor implements the QueryNamespaceProvider trait, and the underlying QueryDatabase implements QueryNamespace. Satisfying those requirements allows the construction of a FlightServiceServer from the service_grpc_flight crate. The FlightServiceServer is a gRPC server that can be served via tonic at the API surface; however, enabling this required some tower::Service wrangling. The influxdb3_server/src/server.rs module was introduced to house this code. The objective is to serve both gRPC (via the newly introduced tonic server) and standard REST HTTP requests (via the existing HTTP server) on the same port. This is accomplished by the HybridService which can handle either gRPC or non-gRPC HTTP requests. The HybridService is wrapped in a HybridMakeService which allows us to serve it via hyper::Server on a single bind address. End-to-end tests were added in influxdb3/tests/flight.rs. These cover some basic FlightSQL cases. A common.rs module was added that introduces some fixtures to aid in end-to-end tests in influxdb3.
2024-02-26 20:07:48 +00:00
arrow-flight.workspace = true
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
arrow-json.workspace = true
arrow-schema.workspace = true
async-trait.workspace = true
feat: Change Bearer Auth Token to use random bits (#24733) This changes the 'influxdb3 create token' command so that it will just automatically generate a completely random base64 encoded token prepended with 'apiv3_' that is then fed into a Sha512 algorithm instead of Sha256. The user can no longer pass in a token to be turned into the proper output. This also changes the server code to handle the change to Sha512 as well. Closes #24704
2024-03-06 17:43:00 +00:00
base64.workspace = true
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
bytes.workspace = true
chrono.workspace = true
feat: add csv influx v1 (#25030) * feat: add csv influx v1 * fix: clippy error * fix: cargo.lock * fix: apply feedbacks * test: add csv integration test * fix: cargo audit
2024-06-25 12:45:55 +00:00
csv.workspace = true
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
datafusion.workspace = true
flate2.workspace = true
futures.workspace = true
hex.workspace = true
hyper.workspace = true
feat: metacache REST APIs to create and delete (#25587)
2024-11-27 13:41:46 +00:00
humantime.workspace = true
feat: API to create last caches (#25147) Closes #25096 - Adds a new HTTP API that allows the creation of a last cache, see the issue for details - An E2E test was added to check success/failure behaviour of the API - Adds the mime crate, for parsing request MIME types, but this is only used in the code I added - we may adopt it in other APIs / parts of the HTTP server in future PRs
2024-07-16 14:32:26 +00:00
mime.workspace = true
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
object_store.workspace = true
parking_lot.workspace = true
pin-project-lite.workspace = true
feat: support v1 query API (#24746) feat: support the v1 query API This PR adds support for the `/api/v1/query` API, which is meant to serve the original InfluxDB v1 query API, to serve single statement `SELECT` and `SHOW` queries. The response, which is returned as JSON, can be chunked via the `chunked` and optional `chunk_size` parameters. An optional `epoch` parameter can be supplied to have `time` column timestamps converted to a UNIX epoch with the given precision. ## Buffering The response is buffered by default, but if the `chunked` parameter is not supplied, or is passed as `false`, then the entire query result will be buffered into memory before being returned in the response. This is how the original API behaves, so we are replicating that here. When `chunked` is passed as `true`, then the response will be a stream of chunks, where each chunk is a self-contained response, with the same structure as that of the non-chunked response. Chunks are split up by the provided `chunk_size`, or by series, i.e., measurement, which ever comes first. The default chunk size is 10,000 rows. Buffering is implemented with the `QueryResponseStream` and `ChunkBuffer` types, the former implements the `Stream` trait, which allows it to be streamed in the HTTP response directly with `hyper`'s `Body::wrap_stream`. The `QueryResponseStream` is a wrapper around the inner arrow `RecordBatchStream`, which buffers the streamed `RecordBatch`es according to the requested chunking parameters. ## Testing Two new E2E tests were added to test basic query functionality and chunking behaviour, respectively. In addition, some manual testing was done to verify that the InfluxDB Grafana plugin works with this API.
2024-03-15 17:38:15 +00:00
secrecy.workspace = true
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
serde.workspace = true
serde_json.workspace = true
serde_urlencoded.workspace = true
sha2.workspace = true
thiserror.workspace = true
tokio.workspace = true
tokio-util.workspace = true
tonic.workspace = true
tower.workspace = true
unicode-segmentation.workspace = true
feat: Add basic Edge server structure (#24552) * WIP: basic influxdb3 command and http server * WIP: write lp, buffer, query out * WIP: test write & query on influxdb3_server, fix warnings * WIP: pull write buffer and catalog into separate crate * WIP: sketch out types used for write: buffer, wal, persister * WIP: remove a bunch of old IOx stuff and fmt
2024-01-08 16:50:59 +00:00
feat: Implement WAL plugin test API (#25704) * feat: Implement WAL plugin test API This implements the WAL plugin test API. It also introduces a new API for the Python plugins to be called, get their data, and call back into the database server. There are some things that I'll want to address in follow on work: * CLI tests, but will wait on #25737 to land for a refactor of the CLI here * Would be better to hook the Python logging to call back into the plugin return state like here: https://pyo3.rs/v0.23.3/ecosystem/logging.html#the-python-to-rust-direction * We should only load the LineBuilder interface once in a module, rather than on every execution of a WAL plugin * More tests all around But I want to get this in so that the actual plugin and trigger system can get udated to build around this model. * refactor: PR feedback
2025-01-06 22:32:17 +00:00
[dependencies.pyo3]
version = "0.23.3"
# this is necessary to automatically initialize the Python interpreter
features = ["auto-initialize"]
optional = true
[features]
system-py = ["pyo3"]
feat: Add basic Edge server structure (#24552) * WIP: basic influxdb3 command and http server * WIP: write lp, buffer, query out * WIP: test write & query on influxdb3_server, fix warnings * WIP: pull write buffer and catalog into separate crate * WIP: sketch out types used for write: buffer, wal, persister * WIP: remove a bunch of old IOx stuff and fmt
2024-01-08 16:50:59 +00:00
[dev-dependencies]
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
# Core Crates
feat: Add different output support to queries (#24616) This commit adds the ability to choose the output format of a query via the v3 api so that a user can choose, whether by Accept headers or the format url param, how the data will be returned to them. Prior to this commit the default was a pretty printed text format, but that instead has been changed to json as the default. There are multiple formats one can choose: 1. json 2. csv 3. pretty printed text 4. parquet I've tested each of these out and it works well. In particular the parquet output is exciting as users will be able to perform a query and receive back parquet data that they can then load into say a Python script or something else to work on and operate it. As we extend what data can be queried, as well as persisting it, what people will be able to do with Edge will be really cool and I'm interested to see how users will end up using this functionality in the future.
2024-02-12 17:04:05 +00:00
parquet.workspace = true
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
parquet_file.workspace = true
test_helpers.workspace = true
# crates.io crates
http.workspace = true
hyper.workspace = true
feat: add `system.parquet_files` table (#25225) This extends the system tables available with a new `parquet_files` table which will list the parquet files associated with a given table in a database. Queries to system.parquet_files must provide a table_name predicate to specify the table name of interest. The files are accessed through the QueryableBuffer. In addition, a test was added to check success and failure modes of the new system table query. Finally, the Persister trait had its associated error type removed. This was somewhat of a consequence of how I initially implemented this change, but I felt cleaned the code up a bit, so I kept it in the commit.
2024-08-08 12:46:26 +00:00
test-log.workspace = true
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
urlencoding.workspace = true
pretty_assertions.workspace = true