Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
influxdb/influxdb3/Cargo.toml

94 lines
3.0 KiB
TOML
Raw Normal View History

chore: move `influxdb_iox` into a proper workspace package
2021-10-26 08:46:57 +00:00
[package]
feat: Add basic Edge server structure (#24552) * WIP: basic influxdb3 command and http server * WIP: write lp, buffer, query out * WIP: test write & query on influxdb3_server, fix warnings * WIP: pull write buffer and catalog into separate crate * WIP: sketch out types used for write: buffer, wal, persister * WIP: remove a bunch of old IOx stuff and fmt
2024-01-08 16:50:59 +00:00
name = "influxdb3"
feat: Use workspace package metadata to reduce differences and repetition
2022-09-26 14:43:00 +00:00
version.workspace = true
authors.workspace = true
edition.workspace = true
license.workspace = true
chore: move `influxdb_iox` into a proper workspace package
2021-10-26 08:46:57 +00:00
[dependencies]
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
# Core Crates
clap_blocks.workspace = true
iox_query.workspace = true
iox_time.workspace = true
ioxd_common.workspace = true
metric.workspace = true
object_store.workspace = true
observability_deps.workspace = true
panic_logging.workspace = true
parquet_file.workspace = true
tokio_metrics_bridge.workspace = true
trace.workspace = true
trace_exporters.workspace = true
trogging.workspace = true
# Local Crates
feat: add write and query CLI sub-commands (#24671) * feat: add query and write cli for influxdb3 Adds two new sub-commands to the influxdb3 CLI: - query: perform queries against the running server - write: perform writes against the running server Both share a common set of parameters for connecting to the database which are managed in influxdb3/src/commands/common.rs. Currently, query supports all underlying output formats, and can write the output to a file on disk. It only supports SQL as the query language, but will eventually also support InfluxQL. Write supports line protocol for input and expects the source of data to be from a file.
2024-02-20 21:14:19 +00:00
influxdb3_client = { path = "../influxdb3_client" }
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
influxdb3_server = { path = "../influxdb3_server" }
feat: Add basic Edge server structure (#24552) * WIP: basic influxdb3 command and http server * WIP: write lp, buffer, query out * WIP: test write & query on influxdb3_server, fix warnings * WIP: pull write buffer and catalog into separate crate * WIP: sketch out types used for write: buffer, wal, persister * WIP: remove a bunch of old IOx stuff and fmt
2024-01-08 16:50:59 +00:00
influxdb3_write = { path = "../influxdb3_write" }
chore: move `influxdb_iox` into a proper workspace package
2021-10-26 08:46:57 +00:00
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
# Crates.io dependencies
backtrace.workspace = true
feat: Change Bearer Auth Token to use random bits (#24733) This changes the 'influxdb3 create token' command so that it will just automatically generate a completely random base64 encoded token prepended with 'apiv3_' that is then fed into a Sha512 algorithm instead of Sha256. The user can no longer pass in a token to be turned into the proper output. This also changes the server code to handle the change to Sha512 as well. Closes #24704
2024-03-06 17:43:00 +00:00
base64.workspace = true
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
clap.workspace = true
dotenvy.workspace = true
hex.workspace = true
libc.workspace = true
num_cpus.workspace = true
once_cell.workspace = true
parking_lot.workspace = true
feat: Change Bearer Auth Token to use random bits (#24733) This changes the 'influxdb3 create token' command so that it will just automatically generate a completely random base64 encoded token prepended with 'apiv3_' that is then fed into a Sha512 algorithm instead of Sha256. The user can no longer pass in a token to be turned into the proper output. This also changes the server code to handle the change to Sha512 as well. Closes #24704
2024-03-06 17:43:00 +00:00
rand.workspace = true
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
secrecy.workspace = true
sha2.workspace = true
thiserror.workspace = true
tokio.workspace = true
tokio-util.workspace = true
url.workspace = true
uuid.workspace = true
# Optional Dependencies
chore(deps): Bump console-subscriber from 0.1.9 to 0.1.10 (#8142) Bumps [console-subscriber](https://github.com/tokio-rs/console) from 0.1.9 to 0.1.10. - [Release notes](https://github.com/tokio-rs/console/releases) - [Commits](https://github.com/tokio-rs/console/compare/console-subscriber-v0.1.9...console-subscriber-v0.1.10) --- updated-dependencies: - dependency-name: console-subscriber dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-04 12:13:49 +00:00
console-subscriber = { version = "0.1.10", optional = true, features = ["parking_lot"] }
chore(deps): Bump tikv-jemalloc-ctl from 0.5.0 to 0.5.4 (#8359) Bumps [tikv-jemalloc-ctl](https://github.com/tikv/jemallocator) from 0.5.0 to 0.5.4. - [Release notes](https://github.com/tikv/jemallocator/releases) - [Changelog](https://github.com/tikv/jemallocator/blob/main/CHANGELOG.md) - [Commits](https://github.com/tikv/jemallocator/compare/0.5.0...0.5.4) --- updated-dependencies: - dependency-name: tikv-jemalloc-ctl dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Dom <dom@itsallbroken.com>
2023-07-28 12:45:43 +00:00
tikv-jemalloc-ctl = { version = "0.5.4", optional = true }
feat: Add basic Edge server structure (#24552) * WIP: basic influxdb3 command and http server * WIP: write lp, buffer, query out * WIP: test write & query on influxdb3_server, fix warnings * WIP: pull write buffer and catalog into separate crate * WIP: sketch out types used for write: buffer, wal, persister * WIP: remove a bunch of old IOx stuff and fmt
2024-01-08 16:50:59 +00:00
tikv-jemalloc-sys = { version = "0.5.4", optional = true, features = ["unprefixed_malloc_on_supported_platforms"] }
chore: move `influxdb_iox` into a proper workspace package
2021-10-26 08:46:57 +00:00
[features]
feat: Enable Obj Storage Support for Azure/GCP/S3 (#24724) In order for Edge to support other object stores besides the local file system we just needed to turn on the features in clap_blocks which handles all of the configuration needed to create an `Arc<dyn ObjectStore>` for us. We already were calling it's `make_object_store` function that did this and so it's a simple switch flip to turn it on. Closes: #24553
2024-03-04 15:51:57 +00:00
default = ["jemalloc_replacing_malloc", "azure", "gcp", "aws"]
chore: move `influxdb_iox` into a proper workspace package
2021-10-26 08:46:57 +00:00
build: remove unused dependencies from crates This commit fixes loads of crates (47!) had unused dependencies, or mis-configured dependencies (test deps as normal deps). I added the "unused_crate_dependencies" to all crates to help prevent this mess from growing again! https://doc.rust-lang.org/beta/nightly-rustc/rustc_lint_defs/builtin/static.UNUSED_CRATE_DEPENDENCIES.html This has the minor downside of false-positives when specifying dev-dependencies for test/bench binaries - these are files in /test or /benches (not normal tests). This commit includes a workaround, importing them in lib.rs (gated by a feature flag). I think the trade-off of better dependency management is worth it!
2023-05-22 13:08:20 +00:00
azure = ["clap_blocks/azure"] # Optional Azure Object store support
gcp = ["clap_blocks/gcp"] # Optional GCP object store support
aws = ["clap_blocks/aws"] # Optional AWS / S3 object store support
fix: fix pprof (#4261)
2022-04-08 17:46:01 +00:00
pprof = ["ioxd_common/pprof"] # Optional http://localhost:8080/debug/pprof/profile support
fix: fix heappy + update docs (#4917) * docs: Update heap profiling documentation * fix: fix heappy builds * fix: do not run cli tests with heappy Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-21 19:53:28 +00:00
heappy = ["ioxd_common/heappy"] # Optional http://localhost:8080/debug/pproc/alloc support
chore: move `influxdb_iox` into a proper workspace package
2021-10-26 08:46:57 +00:00
feat: tokio-console support This commit adds an optional feature (disabled by default) to the IOx server binary named "tokio_console". When enabled, it adds support for the tokio-console to IOx: https://github.com/tokio-rs/console Enabling this feature drops the compile-time log filter to TRACE, and enables the necessary dependencies to support the instrumentation needed for the console to function. Unfortunately, this feature uses tonic 0.7 (latest) while we use tonic 0.6, so we wind up with two tonic versions being compiled in when this feature is enabled.
2022-04-12 10:33:27 +00:00
# Enable tokio_console support (https://github.com/tokio-rs/console)
#
# Requires enabling trace level tracing events for [tokio,runtime].
tokio_console = ["console-subscriber", "tokio/tracing", "observability_deps/release_max_level_trace"]
chore: move `influxdb_iox` into a proper workspace package
2021-10-26 08:46:57 +00:00
# heappy is an optional feature; Not on by default as it
# runtime overhead on all allocations (calls to malloc).
# Cargo cannot currently implement mutually exclusive features so let's force every build
# to pick either heappy or jemalloc_replacing_malloc feature at least until we figure out something better.
fix: include git sha (again) in release build (#4193) * fix: error if git-sha can not be found * refactor: move main to influxdb_iox * fix: fmt
2022-03-31 19:14:21 +00:00
jemalloc_replacing_malloc = ["tikv-jemalloc-sys", "tikv-jemalloc-ctl"]
ci: run clippy for all features This indeed fixes some issues in our object store implementation. Closes #537.
2021-11-11 09:48:21 +00:00
# Implicit feature selected when running under `clippy --all-features` to accept mutable exclusive features during
# linting
clippy = []
feat: Add All or Nothing Bearer token auth support (#24666) This commit adds basic authorization support to Edge. Up to this point we didn't need have authorization at all and so the server would receive and accept requests from anyone. This isn't exactly secure or ideal for a deployment and so we add a basic form of authentication. The way this works is that a user passes in a hex encoded sha256 hash of a given token to the '--bearer-token' flag of the serve command. When the server starts with this flag it will now check a header of the form 'Authorization: Bearer <token>' by making sure it is valid in the sense that it is not malformed and that when token is hashed it matches the value passed in on the command line. The request is denied with either a 400 Bad Request if the header is malformed or a 401 Unauthorized if the hash does not match or the header is missing. The user is provided a new subcommand of the form: 'influxdb3 create token <token>' where the output contains the command to run the server with and what the header should look like to make requests. I can see future work including multiple tokens and rotating between them or adding new ones to a live service, but for now this shall suffice. As part of the commit end-to-end tests are included to run the server and make requests against the HTTP API and to make sure that requests are denied for being unauthorized, accepted for having the right header, or denied for being malformed. Also as part of this commit a small fix is included for 'Accept: */*' headers. We were not checking for them and if this header was included we were denying it instead of sending back the default payload return value.
2024-02-20 20:34:39 +00:00
[dev-dependencies]
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
# Core Crates
arrow_util.workspace = true
influxdb_iox_client.workspace = true
feat: support FlightSQL in 3.0 (#24678) * feat: support FlightSQL by serving gRPC requests on same port as HTTP This commit adds support for FlightSQL queries via gRPC to the influxdb3 service. It does so by ensuring the QueryExecutor implements the QueryNamespaceProvider trait, and the underlying QueryDatabase implements QueryNamespace. Satisfying those requirements allows the construction of a FlightServiceServer from the service_grpc_flight crate. The FlightServiceServer is a gRPC server that can be served via tonic at the API surface; however, enabling this required some tower::Service wrangling. The influxdb3_server/src/server.rs module was introduced to house this code. The objective is to serve both gRPC (via the newly introduced tonic server) and standard REST HTTP requests (via the existing HTTP server) on the same port. This is accomplished by the HybridService which can handle either gRPC or non-gRPC HTTP requests. The HybridService is wrapped in a HybridMakeService which allows us to serve it via hyper::Server on a single bind address. End-to-end tests were added in influxdb3/tests/flight.rs. These cover some basic FlightSQL cases. A common.rs module was added that introduces some fixtures to aid in end-to-end tests in influxdb3.
2024-02-26 20:07:48 +00:00
# Crates.io dependencies in alphabetical order:
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
arrow.workspace = true
arrow-array.workspace = true
arrow-flight.workspace = true
assert_cmd.workspace = true
futures.workspace = true
hyper.workspace = true
feat: add the `api/v3/query_influxql` API (#24696) feat: add query_influxql api This PR adds support for the /api/v3/query_influxql API. This re-uses code from the existing query_sql API, but some refactoring was done to allow for code re-use between the two. The main change to the original code from the existing query_sql API was that the format is determined up front, in the event that the user provides some incorrect Accept header, so that the 400 BAD REQUEST is returned before performing the query. Support of several InfluxQL queries that previously required a bridge to be executed in 3.0 was added: SHOW MEASUREMENTS SHOW TAG KEYS SHOW TAG VALUES SHOW FIELD KEYS SHOW DATABASES Handling of qualified measurement names in SELECT queries (see below) This is accomplished with the newly added iox_query_influxql_rewrite crate, which provides the means to re-write an InfluxQL statement to strip out a database name and retention policy, if provided. Doing so allows the query_influxql API to have the database parameter optional, as it may be provided in the query string. Handling qualified measurement names in SELECT The implementation in this PR will inspect all measurements provided in a FROM clause and extract the database (DB) name and retention policy (RP) name (if not the default). If multiple DB/RP's are provided, an error is thrown. Testing E2E tests were added for performing basic queries against a running server on both the query_sql and query_influxql APIs. In addition, the test for query_influxql includes some of the InfluxQL-specific queries, e.g., SHOW MEASUREMENTS. Other Changes The influxdb3_client now has the api_v3_query_influxql method (and a basic test was added for this)
2024-03-01 17:27:38 +00:00
pretty_assertions.workspace = true
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
reqwest.workspace = true
feat: add the `api/v3/query_influxql` API (#24696) feat: add query_influxql api This PR adds support for the /api/v3/query_influxql API. This re-uses code from the existing query_sql API, but some refactoring was done to allow for code re-use between the two. The main change to the original code from the existing query_sql API was that the format is determined up front, in the event that the user provides some incorrect Accept header, so that the 400 BAD REQUEST is returned before performing the query. Support of several InfluxQL queries that previously required a bridge to be executed in 3.0 was added: SHOW MEASUREMENTS SHOW TAG KEYS SHOW TAG VALUES SHOW FIELD KEYS SHOW DATABASES Handling of qualified measurement names in SELECT queries (see below) This is accomplished with the newly added iox_query_influxql_rewrite crate, which provides the means to re-write an InfluxQL statement to strip out a database name and retention policy, if provided. Doing so allows the query_influxql API to have the database parameter optional, as it may be provided in the query string. Handling qualified measurement names in SELECT The implementation in this PR will inspect all measurements provided in a FROM clause and extract the database (DB) name and retention policy (RP) name (if not the default). If multiple DB/RP's are provided, an error is thrown. Testing E2E tests were added for performing basic queries against a running server on both the query_sql and query_influxql APIs. In addition, the test for query_influxql includes some of the InfluxQL-specific queries, e.g., SHOW MEASUREMENTS. Other Changes The influxdb3_client now has the api_v3_query_influxql method (and a basic test was added for this)
2024-03-01 17:27:38 +00:00
test_helpers.workspace = true
feat: support FlightSQL in 3.0 (#24678) * feat: support FlightSQL by serving gRPC requests on same port as HTTP This commit adds support for FlightSQL queries via gRPC to the influxdb3 service. It does so by ensuring the QueryExecutor implements the QueryNamespaceProvider trait, and the underlying QueryDatabase implements QueryNamespace. Satisfying those requirements allows the construction of a FlightServiceServer from the service_grpc_flight crate. The FlightServiceServer is a gRPC server that can be served via tonic at the API surface; however, enabling this required some tower::Service wrangling. The influxdb3_server/src/server.rs module was introduced to house this code. The objective is to serve both gRPC (via the newly introduced tonic server) and standard REST HTTP requests (via the existing HTTP server) on the same port. This is accomplished by the HybridService which can handle either gRPC or non-gRPC HTTP requests. The HybridService is wrapped in a HybridMakeService which allows us to serve it via hyper::Server on a single bind address. End-to-end tests were added in influxdb3/tests/flight.rs. These cover some basic FlightSQL cases. A common.rs module was added that introduces some fixtures to aid in end-to-end tests in influxdb3.
2024-02-26 20:07:48 +00:00
tonic.workspace = true
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
tower.workspace = true