Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
influxdb/ingester/Cargo.toml

48 lines
1.4 KiB
TOML
Raw Normal View History

feat: add ingester crate and a few basic data structures for its data lifecycle
2022-01-17 20:38:03 +00:00
[package]
name = "ingester"
version = "0.1.0"
authors = ["Nga Tran <nga-tran@live.com>"]
edition = "2021"
[dependencies]
chore: Update datafusion, upgrade to arrow/parqet/arrow-flight 8.0.0 (#3517) * chore: Update datafusion * chore: update to arrow 8 * fix: update to use new DataFusion APIs * fix: update case for sortedness * fix: cargo hakari
2022-01-27 13:33:27 +00:00
arrow = { version = "8.0", features = ["prettyprint"] }
feat: Ingester Flight gRPC API (#3623) * feat: Add a way to run ingester with an in-memory catalog from the CLI If you set the --catalog-dsn string to "mem", rather than using that as a Postgres connection URL, create an in-memory catalog. Planning on using this in tests, so not documenting. * fix: Set default topic to the same value as SHARED_KAFKA_TOPIC Namely, both should use an underscore. I don't think there's a way to directly share these values between a constant and an annotation. * feat: Add a flight API (handshake only) to ingester * fix: Create partitions if using file-based write buffer * fix: Change the server fixture to handle ingester server type For now, the ingester doesn't implement the deployment API. Not sure if it should or not. * feat: Start implementing ingester do_get, namely decoding the query Skip serialization of the predicate for the moment. * refactor: Rename ingest protos to ingester to match crate name * refactor: Rename QueryResults to QueryData * feat: Move ingester flight client to new querier crate * fix: Off by one error, different starting indexes in sequencers * fix: Create new CLI argument to pick the catalog type * fix: Create a CLI option to set the number of topics to auto-create in the write buffer * fix: Check the arrow flight service's health to tell that the ingester gRPC is up * fix: Set postgres as the default catalog type * fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
arrow-flight = "8.0"
feat: add but ignore a few compaction tests
2022-01-24 17:00:23 +00:00
arrow_util = { path = "../arrow_util" }
feat: Add lifecycle manager to ingester (#3645) This adds the lifecycle manager to the ingester. It will trigger based on a threshold for max partition size or age or based on keeping total memory under a certain threshold. It defines a new interface for a persister, which is stubbed out for IngesterData. I'm not sure yet how persistence errors should be handled. The assumption here is that the persister continues to retry persistence forever until it succeeds. There is one scenario I can think of that may cause this lifecycle manager problems. If a single partition is very high throughput, it could cause things to back up as persistence is not parallelized within a single partition. Any given partition can currently only run one persistence operation at a time. We can address this later. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 15:23:40 +00:00
async-trait = "0.1.42"
feat: Write parquet to object storage in a similar way as parquet_file::Storage
2022-01-26 20:46:03 +00:00
base64 = "0.13"
bytes = "1.0"
feat: initial implementation for ingester's compaction
2022-01-20 23:22:41 +00:00
datafusion = { path = "../datafusion" }
data_types = { path = "../data_types" }
feat: Write parquet to object storage in a similar way as parquet_file::Storage
2022-01-26 20:46:03 +00:00
futures = "0.3"
feat: Create a new proto definition for the new design's IoxMetadata
2022-01-27 19:44:20 +00:00
generated_types = { path = "../generated_types" }
feat: wire up the write buffer to the ingester process (#3533) This adds the scaffolding for the ingester server to consume data from Kafka. This ingests data in an in memory structure while creating records in the catalog for any partitions that don't yet exist. I've removed catalog_update.rs in ingester for now. That was mostly a placeholder and will be going in a combination of handler.rs and data.rs on my next PR which will have some primitive lifecycle wired up. There's one ugly bit here where the DML write is cloned because it's getting borrowed to output spans and metrics. I'll need to follow up with a refactor to make it so that the DML write's tables can be consumed without it gumming up the metrics stuff. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-03 11:47:28 +00:00
chrono = { version = "0.4", default-features = false }
db = { path = "../db" }
dml = { path = "../dml" }
feat: Add scaffolding for ingester server * Adds a new ingester command to start an ingester server * Moves previous ingester server over to handler * Skeleton for gRPC and HTTP handlers
2022-01-21 23:02:19 +00:00
hyper = "0.14"
feat: commit in order to pull and merge new commit from main
2022-01-18 21:11:25 +00:00
iox_catalog = { path = "../iox_catalog" }
feat: Add scaffolding for ingester server * Adds a new ingester command to start an ingester server * Moves previous ingester server over to handler * Skeleton for gRPC and HTTP handlers
2022-01-21 23:02:19 +00:00
metric = { path = "../metric" }
refactor: address review comments
2022-01-19 01:57:13 +00:00
mutable_batch = { path = "../mutable_batch"}
feat: Wire object store into ingester persistence
2022-01-26 19:26:01 +00:00
object_store = { path = "../object_store" }
chore: upgrade rskafka and parking_lot (#3592)
2022-02-01 11:50:42 +00:00
parking_lot = "0.12"
feat: Write parquet to object storage in a similar way as parquet_file::Storage
2022-01-26 20:46:03 +00:00
parquet = "8.0"
refactor: Move IoxMetadata to parquet_file
2022-01-28 16:56:48 +00:00
parquet_file = { path = "../parquet_file" }
feat: wire up the write buffer to the ingester process (#3533) This adds the scaffolding for the ingester server to consume data from Kafka. This ingests data in an in memory structure while creating records in the catalog for any partitions that don't yet exist. I've removed catalog_update.rs in ingester for now. That was mostly a placeholder and will be going in a combination of handler.rs and data.rs on my next PR which will have some primitive lifecycle wired up. There's one ugly bit here where the DML write is cloned because it's getting borrowed to output spans and metrics. I'll need to follow up with a refactor to make it so that the DML write's tables can be consumed without it gumming up the metrics stuff. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-03 11:47:28 +00:00
observability_deps = { path = "../observability_deps" }
feat: initial implementation for ingester's compaction
2022-01-20 23:22:41 +00:00
predicate = { path = "../predicate" }
feat: Write parquet to object storage in a similar way as parquet_file::Storage
2022-01-26 20:46:03 +00:00
prost = "0.9"
feat: initial implementation for ingester's compaction
2022-01-20 23:22:41 +00:00
query = { path = "../query" }
schema = { path = "../schema" }
refactor: address review comments
2022-01-19 01:57:13 +00:00
snafu = "0.7"
feat: Add scaffolding for ingester server * Adds a new ingester command to start an ingester server * Moves previous ingester server over to handler * Skeleton for gRPC and HTTP handlers
2022-01-21 23:02:19 +00:00
thiserror = "1.0"
feat: IoxMetadata for parquet file (#3547) * feat: IoxMetadata for parquet file * fix: typos * refactor: address review comments Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-28 14:41:59 +00:00
time = { path = "../time" }
tokio = { version = "1.13", features = ["macros", "parking_lot", "rt-multi-thread", "sync", "time"] }
feat: Ingester Flight gRPC API (#3623) * feat: Add a way to run ingester with an in-memory catalog from the CLI If you set the --catalog-dsn string to "mem", rather than using that as a Postgres connection URL, create an in-memory catalog. Planning on using this in tests, so not documenting. * fix: Set default topic to the same value as SHARED_KAFKA_TOPIC Namely, both should use an underscore. I don't think there's a way to directly share these values between a constant and an annotation. * feat: Add a flight API (handshake only) to ingester * fix: Create partitions if using file-based write buffer * fix: Change the server fixture to handle ingester server type For now, the ingester doesn't implement the deployment API. Not sure if it should or not. * feat: Start implementing ingester do_get, namely decoding the query Skip serialization of the predicate for the moment. * refactor: Rename ingest protos to ingester to match crate name * refactor: Rename QueryResults to QueryData * feat: Move ingester flight client to new querier crate * fix: Off by one error, different starting indexes in sequencers * fix: Create new CLI argument to pick the catalog type * fix: Create a CLI option to set the number of topics to auto-create in the write buffer * fix: Check the arrow flight service's health to tell that the ingester gRPC is up * fix: Set postgres as the default catalog type * fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
tonic = { version = "0.6" }
refactor: add persisting struct
2022-01-19 17:36:18 +00:00
uuid = { version = "0.8", features = ["v4"] }
fix: add workspace-hack
2022-01-18 23:10:42 +00:00
workspace-hack = { path = "../workspace-hack"}
feat: wire up the write buffer to the ingester process (#3533) This adds the scaffolding for the ingester server to consume data from Kafka. This ingests data in an in memory structure while creating records in the catalog for any partitions that don't yet exist. I've removed catalog_update.rs in ingester for now. That was mostly a placeholder and will be going in a combination of handler.rs and data.rs on my next PR which will have some primitive lifecycle wired up. There's one ugly bit here where the DML write is cloned because it's getting borrowed to output spans and metrics. I'll need to follow up with a refactor to make it so that the DML write's tables can be consumed without it gumming up the metrics stuff. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-03 11:47:28 +00:00
write_buffer = { path = "../write_buffer" }
tokio-util = { version = "0.6.9" }
trace = { path = "../trace" }
fix: merge record batches into one batch (#3535) * fix: merge record batches into one batch refactor: address review comments * chore: update test output
2022-01-25 23:29:16 +00:00
feat: Implement a snapshot method on DataBuffer (#3518) * feat: Implement a snapshot method on DataBuffer Fixes #3510. * test: Add a test snapshotting batches with different but compatible schemas * fix: Simplify min/max sequencer number collection The first batch should always have the min sequencer number. The last batch should always have the max sequencer number. The min should always be less than (or equal to, in case there's only one batch) the max.
2022-01-26 15:22:51 +00:00
[dev-dependencies]
mutable_batch_lp = { path = "../mutable_batch_lp" }
test_helpers = { path = "../test_helpers" }