Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
influxdb/influxdb3_cache/Cargo.toml

53 lines
1.2 KiB
TOML
Raw Normal View History

feat: metadata cache core impl (#25552) * feat: core metadata cache structs with basic tests Implement the base MetaCache type that holds the hierarchical structure of the metadata cache providing methods to create and push rows from the WAL into the cache. Added a prune method as well as a method for gathering record batches from a meta cache. A test was added to check the latter for various predicates and that the former works, though, pruning shows that we need to modify how record batches are produced such that expired entries are not emitted. * refactor: filter expired entries and do some clean up in the meta cache
2024-11-18 17:28:12 +00:00
[package]
name = "influxdb3_cache"
version.workspace = true
authors.workspace = true
edition.workspace = true
license.workspace = true
[dependencies]
# Core Crates
iox_time.workspace = true
feat: track parquet cache metrics (#25632) * feat: parquet cache metrics * feat: track parquet cache metrics Adds metrics to track the following in the in-memory parquet cache: * cache size in bytes (also included a fix in the calculation of that) * cache size in n files * cache hits * cache misses * cache misses while the oracle is fetching a file A test was added to check this functionality * refactor: clean up logic and fix cache removal tracking error Some logic and naming was cleaned up and the boolean to optionally track metrics on entry removal was removed, as it was incorrect in the first place: a fetching entry still has a size, which counts toward the size of the cache. So, this makes is such that anytime an entry is removed, whether its state is success or fetching, its size will be decremented from the cache size metrics. The sizing caclulations were made to be correct, and the cache metrics test was updated with more thurough assertions
2024-12-10 14:32:15 +00:00
metric.workspace = true
refactor: move last cache to influxdb3_cache crate (#25620) Moved all of the last cache implementation into the `influxdb3_cache` crate. This also splits out the implementation into three modules: - `cache.rs`: the core cache implementation - `provider.rs`: the cache provider used by the database to hold multiple caches. - `table_function.rs`: same as before, holds the DataFusion impls Tests were preserved and moved to `mod.rs`, however, they were updated to not rely on the WriteBuffer implementation, and instead use the types in the `influxdb3_cache::last_cache` module directly. This simplified the test code, while not changing any of the test assertions at all.
2024-12-05 19:04:25 +00:00
observability_deps.workspace = true
feat: metadata cache core impl (#25552) * feat: core metadata cache structs with basic tests Implement the base MetaCache type that holds the hierarchical structure of the metadata cache providing methods to create and push rows from the WAL into the cache. Added a prune method as well as a method for gathering record batches from a meta cache. A test was added to check the latter for various predicates and that the former works, though, pruning shows that we need to modify how record batches are produced such that expired entries are not emitted. * refactor: filter expired entries and do some clean up in the meta cache
2024-11-18 17:28:12 +00:00
schema.workspace = true
# Local deps
influxdb3_catalog = { path = "../influxdb3_catalog" }
influxdb3_id = { path = "../influxdb3_id" }
influxdb3_wal = { path = "../influxdb3_wal" }
# crates.io dependencies
anyhow.workspace = true
arrow.workspace = true
feat: add metadata cache provider with APIs for write and query (#25566) This adds the MetaDataCacheProvider for managing metadata caches in the influxdb3 instance. This includes APIs to create caches through the WAL as well as from a catalog on initialization, to write data into the managed caches, and to query data out of them. The query side is fairly involved, relying on Datafusion's TableFunctionImpl and TableProvider traits to make querying the cache using a user-defined table function (UDTF) possible. The predicate code was modified to only support two kinds of predicates: IN and NOT IN, which simplifies the code, and maps nicely with the DataFusion LiteralGuarantee which we leverage to derive the predicates from the incoming queries. A custom ExecutionPlan implementation was added specifically for the metadata cache that can report the predicates that are pushed down to the cache during query planning/execution. A big set of tests was added to to check that queries are working, and that predicates are being pushed down properly.
2024-11-22 15:57:26 +00:00
async-trait.workspace = true
refactor: move parquet cache to influxdb3_cache crate (#25630)
2024-12-09 16:56:52 +00:00
bytes.workspace = true
dashmap.workspace = true
feat: add metadata cache provider with APIs for write and query (#25566) This adds the MetaDataCacheProvider for managing metadata caches in the influxdb3 instance. This includes APIs to create caches through the WAL as well as from a catalog on initialization, to write data into the managed caches, and to query data out of them. The query side is fairly involved, relying on Datafusion's TableFunctionImpl and TableProvider traits to make querying the cache using a user-defined table function (UDTF) possible. The predicate code was modified to only support two kinds of predicates: IN and NOT IN, which simplifies the code, and maps nicely with the DataFusion LiteralGuarantee which we leverage to derive the predicates from the incoming queries. A custom ExecutionPlan implementation was added specifically for the metadata cache that can report the predicates that are pushed down to the cache during query planning/execution. A big set of tests was added to to check that queries are working, and that predicates are being pushed down properly.
2024-11-22 15:57:26 +00:00
datafusion.workspace = true
refactor: move parquet cache to influxdb3_cache crate (#25630)
2024-12-09 16:56:52 +00:00
futures.workspace = true
feat: add metadata cache provider with APIs for write and query (#25566) This adds the MetaDataCacheProvider for managing metadata caches in the influxdb3 instance. This includes APIs to create caches through the WAL as well as from a catalog on initialization, to write data into the managed caches, and to query data out of them. The query side is fairly involved, relying on Datafusion's TableFunctionImpl and TableProvider traits to make querying the cache using a user-defined table function (UDTF) possible. The predicate code was modified to only support two kinds of predicates: IN and NOT IN, which simplifies the code, and maps nicely with the DataFusion LiteralGuarantee which we leverage to derive the predicates from the incoming queries. A custom ExecutionPlan implementation was added specifically for the metadata cache that can report the predicates that are pushed down to the cache during query planning/execution. A big set of tests was added to to check that queries are working, and that predicates are being pushed down properly.
2024-11-22 15:57:26 +00:00
indexmap.workspace = true
parking_lot.workspace = true
refactor: move parquet cache to influxdb3_cache crate (#25630)
2024-12-09 16:56:52 +00:00
object_store.workspace = true
feat: metacache REST APIs to create and delete (#25587)
2024-11-27 13:41:46 +00:00
serde.workspace = true
feat: add metadata cache provider with APIs for write and query (#25566) This adds the MetaDataCacheProvider for managing metadata caches in the influxdb3 instance. This includes APIs to create caches through the WAL as well as from a catalog on initialization, to write data into the managed caches, and to query data out of them. The query side is fairly involved, relying on Datafusion's TableFunctionImpl and TableProvider traits to make querying the cache using a user-defined table function (UDTF) possible. The predicate code was modified to only support two kinds of predicates: IN and NOT IN, which simplifies the code, and maps nicely with the DataFusion LiteralGuarantee which we leverage to derive the predicates from the incoming queries. A custom ExecutionPlan implementation was added specifically for the metadata cache that can report the predicates that are pushed down to the cache during query planning/execution. A big set of tests was added to to check that queries are working, and that predicates are being pushed down properly.
2024-11-22 15:57:26 +00:00
thiserror.workspace = true
tokio.workspace = true
feat: metadata cache core impl (#25552) * feat: core metadata cache structs with basic tests Implement the base MetaCache type that holds the hierarchical structure of the metadata cache providing methods to create and push rows from the WAL into the cache. Added a prune method as well as a method for gathering record batches from a meta cache. A test was added to check the latter for various predicates and that the former works, though, pruning shows that we need to modify how record batches are produced such that expired entries are not emitted. * refactor: filter expired entries and do some clean up in the meta cache
2024-11-18 17:28:12 +00:00
[dev-dependencies]
refactor: move last cache to influxdb3_cache crate (#25620) Moved all of the last cache implementation into the `influxdb3_cache` crate. This also splits out the implementation into three modules: - `cache.rs`: the core cache implementation - `provider.rs`: the cache provider used by the database to hold multiple caches. - `table_function.rs`: same as before, holds the DataFusion impls Tests were preserved and moved to `mod.rs`, however, they were updated to not rely on the WriteBuffer implementation, and instead use the types in the `influxdb3_cache::last_cache` module directly. This simplified the test code, while not changing any of the test assertions at all.
2024-12-05 19:04:25 +00:00
# core deps
arrow_util.workspace = true
data_types.workspace = true
feat: metadata cache core impl (#25552) * feat: core metadata cache structs with basic tests Implement the base MetaCache type that holds the hierarchical structure of the metadata cache providing methods to create and push rows from the WAL into the cache. Added a prune method as well as a method for gathering record batches from a meta cache. A test was added to check the latter for various predicates and that the former works, though, pruning shows that we need to modify how record batches are produced such that expired entries are not emitted. * refactor: filter expired entries and do some clean up in the meta cache
2024-11-18 17:28:12 +00:00
# Local deps
influxdb3_write = { path = "../influxdb3_write" }
refactor: move parquet cache to influxdb3_cache crate (#25630)
2024-12-09 16:56:52 +00:00
influxdb3_test_helpers = { path = "../influxdb3_test_helpers" }
feat: metadata cache core impl (#25552) * feat: core metadata cache structs with basic tests Implement the base MetaCache type that holds the hierarchical structure of the metadata cache providing methods to create and push rows from the WAL into the cache. Added a prune method as well as a method for gathering record batches from a meta cache. A test was added to check the latter for various predicates and that the former works, though, pruning shows that we need to modify how record batches are produced such that expired entries are not emitted. * refactor: filter expired entries and do some clean up in the meta cache
2024-11-18 17:28:12 +00:00
refactor: move last cache to influxdb3_cache crate (#25620) Moved all of the last cache implementation into the `influxdb3_cache` crate. This also splits out the implementation into three modules: - `cache.rs`: the core cache implementation - `provider.rs`: the cache provider used by the database to hold multiple caches. - `table_function.rs`: same as before, holds the DataFusion impls Tests were preserved and moved to `mod.rs`, however, they were updated to not rely on the WriteBuffer implementation, and instead use the types in the `influxdb3_cache::last_cache` module directly. This simplified the test code, while not changing any of the test assertions at all.
2024-12-05 19:04:25 +00:00
# crates.io deps
bimap.workspace = true
insta.workspace = true
object_store.workspace = true
refactor: move parquet cache to influxdb3_cache crate (#25630)
2024-12-09 16:56:52 +00:00
pretty_assertions.workspace = true
refactor: move last cache to influxdb3_cache crate (#25620) Moved all of the last cache implementation into the `influxdb3_cache` crate. This also splits out the implementation into three modules: - `cache.rs`: the core cache implementation - `provider.rs`: the cache provider used by the database to hold multiple caches. - `table_function.rs`: same as before, holds the DataFusion impls Tests were preserved and moved to `mod.rs`, however, they were updated to not rely on the WriteBuffer implementation, and instead use the types in the `influxdb3_cache::last_cache` module directly. This simplified the test code, while not changing any of the test assertions at all.
2024-12-05 19:04:25 +00:00
test-log.workspace = true
feat: metadata cache core impl (#25552) * feat: core metadata cache structs with basic tests Implement the base MetaCache type that holds the hierarchical structure of the metadata cache providing methods to create and push rows from the WAL into the cache. Added a prune method as well as a method for gathering record batches from a meta cache. A test was added to check the latter for various predicates and that the former works, though, pruning shows that we need to modify how record batches are produced such that expired entries are not emitted. * refactor: filter expired entries and do some clean up in the meta cache
2024-11-18 17:28:12 +00:00
[lints]
workspace = true