2021-10-26 08:46:57 +00:00
|
|
|
[workspace]
|
|
|
|
|
# In alphabetical order
|
2020-05-01 16:52:00 +00:00
|
|
|
members = [
|
2024-01-08 16:50:59 +00:00
|
|
|
"influxdb3",
|
2024-02-16 20:02:16 +00:00
|
|
|
"influxdb3_client",
|
2024-03-25 12:26:24 +00:00
|
|
|
"influxdb3_load_generator",
|
2024-04-01 20:57:10 +00:00
|
|
|
"influxdb3_process",
|
2024-01-08 16:50:59 +00:00
|
|
|
"influxdb3_server",
|
|
|
|
|
"influxdb3_write",
|
feat: add the `api/v3/query_influxql` API (#24696)
feat: add query_influxql api
This PR adds support for the /api/v3/query_influxql API. This re-uses code from the existing query_sql API, but some refactoring was done to allow for code re-use between the two.
The main change to the original code from the existing query_sql API was that the format is determined up front, in the event that the user provides some incorrect Accept header, so that the 400 BAD REQUEST is returned before performing the query.
Support of several InfluxQL queries that previously required a bridge to be executed in 3.0 was added:
SHOW MEASUREMENTS
SHOW TAG KEYS
SHOW TAG VALUES
SHOW FIELD KEYS
SHOW DATABASES
Handling of qualified measurement names in SELECT queries (see below)
This is accomplished with the newly added iox_query_influxql_rewrite crate, which provides the means to re-write an InfluxQL statement to strip out a database name and retention policy, if provided. Doing so allows the query_influxql API to have the database parameter optional, as it may be provided in the query string.
Handling qualified measurement names in SELECT
The implementation in this PR will inspect all measurements provided in a FROM clause and extract the database (DB) name and retention policy (RP) name (if not the default). If multiple DB/RP's are provided, an error is thrown.
Testing
E2E tests were added for performing basic queries against a running server on both the query_sql and query_influxql APIs. In addition, the test for query_influxql includes some of the InfluxQL-specific queries, e.g., SHOW MEASUREMENTS.
Other Changes
The influxdb3_client now has the api_v3_query_influxql method (and a basic test was added for this)
2024-03-01 17:27:38 +00:00
|
|
|
"iox_query_influxql_rewrite",
|
2020-05-01 16:52:00 +00:00
|
|
|
]
|
2024-01-08 16:50:59 +00:00
|
|
|
default-members = ["influxdb3"]
|
2021-10-26 08:46:57 +00:00
|
|
|
|
2021-11-18 22:26:33 +00:00
|
|
|
resolver = "2"
|
|
|
|
|
|
2021-10-26 08:46:57 +00:00
|
|
|
exclude = [
|
|
|
|
|
"*.md",
|
|
|
|
|
"*.txt",
|
|
|
|
|
".circleci/",
|
|
|
|
|
".editorconfig",
|
|
|
|
|
".git*",
|
|
|
|
|
".github/",
|
|
|
|
|
".kodiak.toml",
|
|
|
|
|
"LICENSE*",
|
|
|
|
|
]
|
2019-11-22 21:59:04 +00:00
|
|
|
|
2022-09-26 14:43:00 +00:00
|
|
|
[workspace.package]
|
|
|
|
|
version = "0.1.0"
|
2024-02-29 21:21:41 +00:00
|
|
|
authors = ["influxdata Edge Developers"]
|
2022-09-26 14:43:00 +00:00
|
|
|
edition = "2021"
|
|
|
|
|
license = "MIT OR Apache-2.0"
|
|
|
|
|
|
2022-10-24 17:47:45 +00:00
|
|
|
[workspace.dependencies]
|
feat: support v1 query API (#24746)
feat: support the v1 query API
This PR adds support for the `/api/v1/query` API, which is meant to
serve the original InfluxDB v1 query API, to serve single statement
`SELECT` and `SHOW` queries. The response, which is returned as JSON,
can be chunked via the `chunked` and optional `chunk_size` parameters.
An optional `epoch` parameter can be supplied to have `time` column
timestamps converted to a UNIX epoch with the given precision.
## Buffering
The response is buffered by default, but if the `chunked` parameter
is not supplied, or is passed as `false`, then the entire query
result will be buffered into memory before being returned in the
response. This is how the original API behaves, so we are replicating
that here.
When `chunked` is passed as `true`, then the response will be a
stream of chunks, where each chunk is a self-contained response,
with the same structure as that of the non-chunked response. Chunks
are split up by the provided `chunk_size`, or by series, i.e.,
measurement, which ever comes first. The default chunk size is 10,000
rows.
Buffering is implemented with the `QueryResponseStream` and
`ChunkBuffer` types, the former implements the `Stream` trait,
which allows it to be streamed in the HTTP response directly with
`hyper`'s `Body::wrap_stream`. The `QueryResponseStream` is a wrapper
around the inner arrow `RecordBatchStream`, which buffers the
streamed `RecordBatch`es according to the requested chunking parameters.
## Testing
Two new E2E tests were added to test basic query functionality and
chunking behaviour, respectively. In addition, some manual testing
was done to verify that the InfluxDB Grafana plugin works with this
API.
2024-03-15 17:38:15 +00:00
|
|
|
anyhow = "1.0"
|
feat: add the `api/v3/query_influxql` API (#24696)
feat: add query_influxql api
This PR adds support for the /api/v3/query_influxql API. This re-uses code from the existing query_sql API, but some refactoring was done to allow for code re-use between the two.
The main change to the original code from the existing query_sql API was that the format is determined up front, in the event that the user provides some incorrect Accept header, so that the 400 BAD REQUEST is returned before performing the query.
Support of several InfluxQL queries that previously required a bridge to be executed in 3.0 was added:
SHOW MEASUREMENTS
SHOW TAG KEYS
SHOW TAG VALUES
SHOW FIELD KEYS
SHOW DATABASES
Handling of qualified measurement names in SELECT queries (see below)
This is accomplished with the newly added iox_query_influxql_rewrite crate, which provides the means to re-write an InfluxQL statement to strip out a database name and retention policy, if provided. Doing so allows the query_influxql API to have the database parameter optional, as it may be provided in the query string.
Handling qualified measurement names in SELECT
The implementation in this PR will inspect all measurements provided in a FROM clause and extract the database (DB) name and retention policy (RP) name (if not the default). If multiple DB/RP's are provided, an error is thrown.
Testing
E2E tests were added for performing basic queries against a running server on both the query_sql and query_influxql APIs. In addition, the test for query_influxql includes some of the InfluxQL-specific queries, e.g., SHOW MEASUREMENTS.
Other Changes
The influxdb3_client now has the api_v3_query_influxql method (and a basic test was added for this)
2024-03-01 17:27:38 +00:00
|
|
|
arrow = { version = "50.0.0", features = ["prettyprint", "chrono-tz"] }
|
|
|
|
|
arrow-array = "50.0.0"
|
|
|
|
|
arrow-buffer = "50.0.0"
|
|
|
|
|
arrow-csv = "50.0.0"
|
|
|
|
|
arrow-flight = { version = "50.0.0", features = ["flight-sql-experimental"] }
|
|
|
|
|
arrow-json = "50.0.0"
|
|
|
|
|
arrow-schema = "50.0.0"
|
2024-02-29 21:21:41 +00:00
|
|
|
assert_cmd = "2.0.14"
|
|
|
|
|
async-trait = "0.1"
|
|
|
|
|
backtrace = "0.3"
|
2024-03-06 17:43:00 +00:00
|
|
|
base64 = "0.22.0"
|
2024-02-29 21:21:41 +00:00
|
|
|
byteorder = "1.3.4"
|
|
|
|
|
bytes = "1.5"
|
|
|
|
|
chrono = "0.4"
|
|
|
|
|
clap = { version = "4", features = ["derive", "env", "string"] }
|
|
|
|
|
crc32fast = "1.2.0"
|
|
|
|
|
crossbeam-channel = "0.5.11"
|
2024-04-03 13:36:28 +00:00
|
|
|
datafusion = { git = "https://github.com/influxdata/arrow-datafusion.git", rev = "581e74785b876615d6a63db8c2e5ba372bf78828" }
|
|
|
|
|
datafusion-proto = { git = "https://github.com/influxdata/arrow-datafusion.git", rev = "581e74785b876615d6a63db8c2e5ba372bf78828" }
|
2024-03-25 12:26:24 +00:00
|
|
|
csv = "1.3.0"
|
2024-02-29 21:21:41 +00:00
|
|
|
dotenvy = "0.15.7"
|
|
|
|
|
flate2 = "1.0.27"
|
|
|
|
|
futures = "0.3.28"
|
|
|
|
|
futures-util = "0.3.30"
|
|
|
|
|
hashbrown = "0.14.3"
|
|
|
|
|
hex = "0.4.3"
|
|
|
|
|
http = "0.2.9"
|
2024-03-25 12:26:24 +00:00
|
|
|
humantime = "2.1.0"
|
2024-02-29 21:21:41 +00:00
|
|
|
hyper = "0.14"
|
|
|
|
|
libc = { version = "0.2" }
|
|
|
|
|
mockito = { version = "1.2.0", default-features = false }
|
|
|
|
|
num_cpus = "1.16.0"
|
2024-03-21 14:29:56 +00:00
|
|
|
object_store = "0.9.1"
|
2024-02-29 21:21:41 +00:00
|
|
|
once_cell = { version = "1.18", features = ["parking_lot"] }
|
|
|
|
|
parking_lot = "0.12.1"
|
feat: add the `api/v3/query_influxql` API (#24696)
feat: add query_influxql api
This PR adds support for the /api/v3/query_influxql API. This re-uses code from the existing query_sql API, but some refactoring was done to allow for code re-use between the two.
The main change to the original code from the existing query_sql API was that the format is determined up front, in the event that the user provides some incorrect Accept header, so that the 400 BAD REQUEST is returned before performing the query.
Support of several InfluxQL queries that previously required a bridge to be executed in 3.0 was added:
SHOW MEASUREMENTS
SHOW TAG KEYS
SHOW TAG VALUES
SHOW FIELD KEYS
SHOW DATABASES
Handling of qualified measurement names in SELECT queries (see below)
This is accomplished with the newly added iox_query_influxql_rewrite crate, which provides the means to re-write an InfluxQL statement to strip out a database name and retention policy, if provided. Doing so allows the query_influxql API to have the database parameter optional, as it may be provided in the query string.
Handling qualified measurement names in SELECT
The implementation in this PR will inspect all measurements provided in a FROM clause and extract the database (DB) name and retention policy (RP) name (if not the default). If multiple DB/RP's are provided, an error is thrown.
Testing
E2E tests were added for performing basic queries against a running server on both the query_sql and query_influxql APIs. In addition, the test for query_influxql includes some of the InfluxQL-specific queries, e.g., SHOW MEASUREMENTS.
Other Changes
The influxdb3_client now has the api_v3_query_influxql method (and a basic test was added for this)
2024-03-01 17:27:38 +00:00
|
|
|
parquet = { version = "50.0.0", features = ["object_store"] }
|
2024-02-29 21:21:41 +00:00
|
|
|
pbjson = "0.6.0"
|
|
|
|
|
pbjson-build = "0.6.2"
|
|
|
|
|
pbjson-types = "0.6.0"
|
|
|
|
|
pin-project-lite = "0.2"
|
|
|
|
|
pretty_assertions = "1.4.0"
|
|
|
|
|
prost = "0.12.3"
|
|
|
|
|
prost-build = "0.12.2"
|
|
|
|
|
prost-types = "0.12.3"
|
2024-03-06 17:43:00 +00:00
|
|
|
rand = "0.8.5"
|
feat: support v1 query API (#24746)
feat: support the v1 query API
This PR adds support for the `/api/v1/query` API, which is meant to
serve the original InfluxDB v1 query API, to serve single statement
`SELECT` and `SHOW` queries. The response, which is returned as JSON,
can be chunked via the `chunked` and optional `chunk_size` parameters.
An optional `epoch` parameter can be supplied to have `time` column
timestamps converted to a UNIX epoch with the given precision.
## Buffering
The response is buffered by default, but if the `chunked` parameter
is not supplied, or is passed as `false`, then the entire query
result will be buffered into memory before being returned in the
response. This is how the original API behaves, so we are replicating
that here.
When `chunked` is passed as `true`, then the response will be a
stream of chunks, where each chunk is a self-contained response,
with the same structure as that of the non-chunked response. Chunks
are split up by the provided `chunk_size`, or by series, i.e.,
measurement, which ever comes first. The default chunk size is 10,000
rows.
Buffering is implemented with the `QueryResponseStream` and
`ChunkBuffer` types, the former implements the `Stream` trait,
which allows it to be streamed in the HTTP response directly with
`hyper`'s `Body::wrap_stream`. The `QueryResponseStream` is a wrapper
around the inner arrow `RecordBatchStream`, which buffers the
streamed `RecordBatch`es according to the requested chunking parameters.
## Testing
Two new E2E tests were added to test basic query functionality and
chunking behaviour, respectively. In addition, some manual testing
was done to verify that the InfluxDB Grafana plugin works with this
API.
2024-03-15 17:38:15 +00:00
|
|
|
reqwest = { version = "0.11.24", default-features = false, features = ["rustls-tls", "stream"] }
|
2024-02-29 21:21:41 +00:00
|
|
|
secrecy = "0.8.0"
|
|
|
|
|
serde = { version = "1.0", features = ["derive"] }
|
2024-03-05 20:40:58 +00:00
|
|
|
serde_arrow = { version = "0.10", features = ["arrow-50"] }
|
2024-02-29 21:21:41 +00:00
|
|
|
serde_json = "1.0"
|
|
|
|
|
serde_urlencoded = "0.7.0"
|
|
|
|
|
sha2 = "0.10.8"
|
|
|
|
|
snap = "1.0.0"
|
|
|
|
|
sqlparser = "0.41.0"
|
2024-04-02 21:16:17 +00:00
|
|
|
sysinfo = "0.30.8"
|
2024-02-29 21:21:41 +00:00
|
|
|
thiserror = "1.0"
|
|
|
|
|
tokio = { version = "1.35", features = ["full"] }
|
|
|
|
|
tokio-util = "0.7.9"
|
2024-02-01 00:18:51 +00:00
|
|
|
tonic = { version = "0.10.2", features = ["tls", "tls-roots"] }
|
2024-02-29 21:21:41 +00:00
|
|
|
tonic-build = "0.10.2"
|
|
|
|
|
tonic-health = "0.10.2"
|
|
|
|
|
tonic-reflection = "0.10.2"
|
|
|
|
|
tower = "0.4.13"
|
|
|
|
|
unicode-segmentation = "1.11.0"
|
|
|
|
|
url = "2.5.0"
|
|
|
|
|
urlencoding = "1.1"
|
|
|
|
|
uuid = { version = "1", features = ["v4"] }
|
|
|
|
|
|
|
|
|
|
# Core.git crates we depend on
|
2024-04-23 16:55:30 +00:00
|
|
|
arrow_util = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a"}
|
|
|
|
|
authz = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a", features = ["http"] }
|
|
|
|
|
clap_blocks = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
data_types = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
datafusion_util = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
influxdb-line-protocol = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
influxdb_influxql_parser = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
influxdb_iox_client = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
iox_catalog = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
ioxd_common = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
iox_http = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
iox_query = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
iox_query_params = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
iox_query_influxql = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
iox_time = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
metric = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
metric_exporters = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
observability_deps = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
panic_logging = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
parquet_file = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
schema = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
service_common = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
service_grpc_flight = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
test_helpers = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
test_helpers_end_to_end = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
tokio_metrics_bridge = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
trace = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
trace_exporters = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
trace_http = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
tracker = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a" }
|
|
|
|
|
trogging = { git = "https://github.com/influxdata/influxdb3_core", rev = "b546e7f86ee9adbff0dd3c5e687140848397604a", default-features = true, features = ["clap"] }
|
2023-07-11 13:36:38 +00:00
|
|
|
|
2024-02-01 00:18:51 +00:00
|
|
|
[workspace.lints.rust]
|
|
|
|
|
rust_2018_idioms = "deny"
|
|
|
|
|
unreachable_pub = "deny"
|
|
|
|
|
missing_debug_implementations = "deny"
|
|
|
|
|
missing_copy_implementations = "deny"
|
|
|
|
|
|
|
|
|
|
[workspace.lints.clippy]
|
|
|
|
|
dbg_macro = "deny"
|
|
|
|
|
todo = "deny"
|
|
|
|
|
clone_on_ref_ptr = "deny"
|
|
|
|
|
future_not_send = "deny"
|
|
|
|
|
|
|
|
|
|
[workspace.lints.rustdoc]
|
|
|
|
|
broken_intra_doc_links = "deny"
|
|
|
|
|
bare_urls = "deny"
|
2023-04-12 16:07:19 +00:00
|
|
|
|
2021-12-06 21:08:41 +00:00
|
|
|
# This profile optimizes for runtime performance and small binary size at the expense of longer
|
|
|
|
|
# build times. It's most suitable for final release builds.
|
2020-03-27 20:39:22 +00:00
|
|
|
[profile.release]
|
2024-05-03 20:39:42 +00:00
|
|
|
lto = "fat"
|
2020-03-27 20:39:22 +00:00
|
|
|
|
2020-12-04 14:48:19 +00:00
|
|
|
[profile.bench]
|
|
|
|
|
debug = true
|
2021-12-06 16:47:52 +00:00
|
|
|
|
2021-12-06 21:08:41 +00:00
|
|
|
# This profile optimizes for short build times at the expense of larger binary size and slower
|
|
|
|
|
# runtime performance. It's most suitable for development iterations.
|
2021-12-06 16:47:52 +00:00
|
|
|
[profile.quick-release]
|
|
|
|
|
inherits = "release"
|
|
|
|
|
codegen-units = 16
|
|
|
|
|
lto = false
|
2021-12-08 14:51:27 +00:00
|
|
|
incremental = true
|
2022-10-13 22:37:49 +00:00
|
|
|
|
|
|
|
|
# Per insta docs: https://insta.rs/docs/quickstart/#optional-faster-runs
|
|
|
|
|
[profile.dev.package.insta]
|
|
|
|
|
opt-level = 3
|
|
|
|
|
|
|
|
|
|
[profile.dev.package.similar]
|
2023-01-19 15:16:13 +00:00
|
|
|
opt-level = 3
|
2024-04-03 13:36:28 +00:00
|
|
|
|
|
|
|
|
# patch arrow-flight crate to allow for prepared statement parameters
|
|
|
|
|
# see related arrow-rs PR https://github.com/apache/arrow-rs/pull/5433
|
|
|
|
|
[patch.crates-io]
|
|
|
|
|
arrow-array = { git = "https://github.com/erratic-pattern/arrow-rs.git", branch = "50.0.0" }
|
|
|
|
|
arrow-schema = { git = "https://github.com/erratic-pattern/arrow-rs.git", branch = "50.0.0" }
|
|
|
|
|
arrow-data = { git = "https://github.com/erratic-pattern/arrow-rs.git", branch = "50.0.0" }
|
|
|
|
|
arrow-buffer = { git = "https://github.com/erratic-pattern/arrow-rs.git", branch = "50.0.0" }
|
|
|
|
|
arrow-ipc = { git = "https://github.com/erratic-pattern/arrow-rs.git", branch = "50.0.0" }
|
|
|
|
|
arrow-flight = { git = "https://github.com/erratic-pattern/arrow-rs.git", branch = "50.0.0" }
|
