Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
influxdb/influxdb3_write/Cargo.toml

47 lines
1.1 KiB
TOML
Raw Normal View History

feat: Add basic Edge server structure (#24552) * WIP: basic influxdb3 command and http server * WIP: write lp, buffer, query out * WIP: test write & query on influxdb3_server, fix warnings * WIP: pull write buffer and catalog into separate crate * WIP: sketch out types used for write: buffer, wal, persister * WIP: remove a bunch of old IOx stuff and fmt
2024-01-08 16:50:59 +00:00
[package]
name = "influxdb3_write"
version.workspace = true
authors.workspace = true
edition.workspace = true
license.workspace = true
[dependencies]
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
# Core Crates
data_types.workspace = true
datafusion_util.workspace = true
influxdb-line-protocol.workspace = true
iox_catalog.workspace = true
iox_query.workspace = true
feat: Segment the write buffer on time (#24745) * Split WriteBuffer into segments * Add SegmentRange and SegmentDuration * Update WAL to store SegmentRange and to be able to open up multiple ranges * Remove Partitioner and PartitionBuffer * Update SegmentState and loader * Update SegmentState with current, next and outside * Update loader and tests to load up current, next and previous outside segments based on the passed in time and desired segment duration * Update WriteBufferImpl and Flusher * Update the flusher to flush to multiple segments * Update WriteBufferImpl to split data into segments getting written to * Update HTTP and WriteBuffer to use TimeProvider * Wire up outside segment writes and loading * Data outside current and next no longer go to a single segment, but to a segment based on that data's time. Limits to 100 segments of time that can be written to at any given time. * Refactor SegmentDuration add config option * Refactors SegmentDuration to be a new type over duration * Adds the clap block configuration to pass SegmentDuration, defaulting to 1h * refactor: SegmentState and loader * remove the current_segment and next_segment from the loader and segment state, instead having just a collection of segments * open up only the current_segment by default * keep current and next segments open if they exist, while others go into persisting or persisted * fix: cargo audit * refactor: fixup PR feedback
2024-03-11 17:54:09 +00:00
iox_time.workspace = true
feat: wire up query from parquet files (#24749) * feat: wire up query from parquet files This adds the functionality to query from Parquet files that have been persisted in object storage. Any segments that are loaded up on boot up will be included (limit of 1k segments at the time of this PR). In a follow on PR we should add a good end-to-end test that has persistence and query through the main API (might be tricky). * Move BufferChunk and ParquetChunk into chunk module * Add object_store_url to Persister * Register object_store on server startup * Add loaded persisted_segments to SegmentState * refactor: PR feedback
2024-03-12 13:47:32 +00:00
parquet_file.workspace = true
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
observability_deps.workspace = true
schema.workspace = true
feat: Add basic Edge server structure (#24552) * WIP: basic influxdb3 command and http server * WIP: write lp, buffer, query out * WIP: test write & query on influxdb3_server, fix warnings * WIP: pull write buffer and catalog into separate crate * WIP: sketch out types used for write: buffer, wal, persister * WIP: remove a bunch of old IOx stuff and fmt
2024-01-08 16:50:59 +00:00
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
# crates.io dependencies
arrow.workspace = true
async-trait.workspace = true
byteorder.workspace = true
feat: add `_series_id` to tables on write (#24842) feat: add _series_id to tables on write New _series_id column is added to tables; this stores a 32 byte SHA256 hash of the tag set of a line of Line Protocol. The tag set is checked for sort order, then sorted if not already, before producing the hash. Unit tests were added to check hashing and sorting functions work. Tests that performed queries needed to be modified to account for the new _series_id column; in general, SELECT * queries were altered to use a select clause with specific column names. The Column limit was increased to 501 internally, to account for the new _series_id column, but the user-facing limit is still 500
2024-03-26 19:22:19 +00:00
bytes.workspace = true
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
chrono.workspace = true
crc32fast.workspace = true
crossbeam-channel.workspace = true
datafusion.workspace = true
feat: add `_series_id` to tables on write (#24842) feat: add _series_id to tables on write New _series_id column is added to tables; this stores a 32 byte SHA256 hash of the tag set of a line of Line Protocol. The tag set is checked for sort order, then sorted if not already, before producing the hash. Unit tests were added to check hashing and sorting functions work. Tests that performed queries needed to be modified to account for the new _series_id column; in general, SELECT * queries were altered to use a select clause with specific column names. The Column limit was increased to 501 internally, to account for the new _series_id column, but the user-facing limit is still 500
2024-03-26 19:22:19 +00:00
futures-util.workspace = true
hex.workspace = true
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
object_store.workspace = true
parking_lot.workspace = true
parquet.workspace = true
thiserror.workspace = true
tokio.workspace = true
serde.workspace = true
serde_json.workspace = true
feat: add `_series_id` to tables on write (#24842) feat: add _series_id to tables on write New _series_id column is added to tables; this stores a 32 byte SHA256 hash of the tag set of a line of Line Protocol. The tag set is checked for sort order, then sorted if not already, before producing the hash. Unit tests were added to check hashing and sorting functions work. Tests that performed queries needed to be modified to account for the new _series_id column; in general, SELECT * queries were altered to use a select clause with specific column names. The Column limit was increased to 501 internally, to account for the new _series_id column, but the user-facing limit is still 500
2024-03-26 19:22:19 +00:00
sha2.workspace = true
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
snap.workspace = true
feat: wire up query from parquet files (#24749) * feat: wire up query from parquet files This adds the functionality to query from Parquet files that have been persisted in object storage. Any segments that are loaded up on boot up will be included (limit of 1k segments at the time of this PR). In a follow on PR we should add a good end-to-end test that has persistence and query through the main API (might be tricky). * Move BufferChunk and ParquetChunk into chunk module * Add object_store_url to Persister * Register object_store on server startup * Add loaded persisted_segments to SegmentState * refactor: PR feedback
2024-03-12 13:47:32 +00:00
url.workspace = true
feat: add basic wal implementation for Edge (#24570) * feat: add basic wal implementation for Edge This WAL implementation uses some of the code from the wal crate, but departs pretty significantly from it in many ways. For now it uses simple JSON encoding for the serialized ops, but we may want to switch that to Protobuf at some point in the future. This version of the wal doesn't have its own buffering. That will be implemented higher up in the BufferImpl, which will use the wal and SegmentWriter to make data in the buffer durable. The write flow will be that writes will come into the buffer and validate/update against an in memory Catalog. Once validated, writes will get buffered up in memory and then flushed into the WAL periodically (likely every 10-20ms). After being flushed to the wal, the entire batch of writes will be put into the in memory queryable buffer. After that responses will be sent back to the clients. This should reduce the write lock pressure on the in-memory buffer considerably. In this PR: - Update the Wal, WalSegmentWriter, and WalSegmentReader traits to line up with new design/understanding - Implement wal (mainly just a way to identify segment files in a directory) - Implement WalSegmentWriter (write header, op batch with crc, and track sequence number in segment, re-open existing file) - Implement WalSegmentReader * refactor: make Wal return impl reader/writer * refactor: clean up wal segment open * fix: WriteBuffer and Wal usage Turn wal and write buffer references into a concrete type, rather than dyn. * fix: have wal loading ignore invalid files
2024-01-12 16:52:28 +00:00
[dev-dependencies]
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
# Core Crates
arrow_util.workspace = true
feat: add the `api/v3/query_influxql` API (#24696) feat: add query_influxql api This PR adds support for the /api/v3/query_influxql API. This re-uses code from the existing query_sql API, but some refactoring was done to allow for code re-use between the two. The main change to the original code from the existing query_sql API was that the format is determined up front, in the event that the user provides some incorrect Accept header, so that the 400 BAD REQUEST is returned before performing the query. Support of several InfluxQL queries that previously required a bridge to be executed in 3.0 was added: SHOW MEASUREMENTS SHOW TAG KEYS SHOW TAG VALUES SHOW FIELD KEYS SHOW DATABASES Handling of qualified measurement names in SELECT queries (see below) This is accomplished with the newly added iox_query_influxql_rewrite crate, which provides the means to re-write an InfluxQL statement to strip out a database name and retention policy, if provided. Doing so allows the query_influxql API to have the database parameter optional, as it may be provided in the query string. Handling qualified measurement names in SELECT The implementation in this PR will inspect all measurements provided in a FROM clause and extract the database (DB) name and retention policy (RP) name (if not the default). If multiple DB/RP's are provided, an error is thrown. Testing E2E tests were added for performing basic queries against a running server on both the query_sql and query_influxql APIs. In addition, the test for query_influxql includes some of the InfluxQL-specific queries, e.g., SHOW MEASUREMENTS. Other Changes The influxdb3_client now has the api_v3_query_influxql method (and a basic test was added for this)
2024-03-01 17:27:38 +00:00
pretty_assertions.workspace = true
feat: Split out shared core crates from Edge (#24714) This commit is a major refactor for the code base. It mainly does four things: 1. Splits code shared between the internal IOx repository and this one into it's own repo over at https://github.com/influxdata/influxdb3_core 2. Removes any docs or anything else that did not relate to this project 3. Reorganizes the Cargo.toml files to use the top level Cargo.toml to declare dependencies and versions to keep all crates in sync and sets all others to use `<dep>.workspace = true` unless it's an optional dependency 4. Set the top level Cargo.toml to point to the core crates as git dependencies With this any changes specific to Edge will be contained here, updating deps will be a PR over in `influxdata/influxdb3_core`, and we can prove out the viability for this model to use for IOx.
2024-02-29 21:21:41 +00:00
test_helpers.workspace = true