Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
influxdb/ingester/Cargo.toml

61 lines
2.0 KiB
TOML
Raw Normal View History

feat: add ingester crate and a few basic data structures for its data lifecycle
2022-01-17 20:38:03 +00:00
[package]
name = "ingester"
feat: Use workspace package metadata to reduce differences and repetition
2022-09-26 14:43:00 +00:00
version.workspace = true
authors.workspace = true
edition.workspace = true
license.workspace = true
feat: add ingester crate and a few basic data structures for its data lifecycle
2022-01-17 20:38:03 +00:00
[dependencies]
feat: Use workspace dep inheritance for the arrow crate
2022-10-24 17:58:38 +00:00
arrow = { workspace = true, features = ["prettyprint"] }
chore: refactor ingester to use upstream arrow-flight (#6622) Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-19 15:16:13 +00:00
arrow-flight = { workspace = true }
feat: add but ignore a few compaction tests
2022-01-24 17:00:23 +00:00
arrow_util = { path = "../arrow_util" }
chore(deps): Bump async-trait from 0.1.63 to 0.1.64 (#6769) Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.63 to 0.1.64. - [Release notes](https://github.com/dtolnay/async-trait/releases) - [Commits](https://github.com/dtolnay/async-trait/compare/0.1.63...0.1.64) --- updated-dependencies: - dependency-name: async-trait dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-01-31 10:18:27 +00:00
async-trait = "0.1.64"
feat: backoff retries in ingester (#3841) * feat: add `backoff` crate * feat: backoff retries in ingester Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 17:58:16 +00:00
backoff = { path = "../backoff" }
chore(deps): Bump bytes from 1.2.1 to 1.3.0 (#6199) Bumps [bytes](https://github.com/tokio-rs/bytes) from 1.2.1 to 1.3.0. - [Release notes](https://github.com/tokio-rs/bytes/releases) - [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md) - [Commits](https://github.com/tokio-rs/bytes/commits) --- updated-dependencies: - dependency-name: bytes dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-22 08:23:24 +00:00
bytes = "1.3"
chore: Alphabetize some dependencies I can't handle not knowing where to look for a dependency or knowing where to add a new dependency.
2022-10-28 14:34:23 +00:00
chrono = { version = "0.4", default-features = false }
data_types = { path = "../data_types" }
feat: Use workspace dep inheritance for datafusion instead of shim crate
2022-10-24 17:47:45 +00:00
datafusion = { workspace = true }
test: include materialized delete predicates in NG query tests (#4371) * refactor: move `batch_filter` to `datafusion_util` * fix: outdated docstring * feat: allow passing record batches to `iox_tests` parquet files * test: include materialized delete predicates in NG query tests * docs: improve wording Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-21 13:00:13 +00:00
datafusion_util = { path = "../datafusion_util" }
chore: Alphabetize some dependencies I can't handle not knowing where to look for a dependency or knowing where to add a new dependency.
2022-10-28 14:34:23 +00:00
dml = { path = "../dml" }
chore: Update datafusion and arrow/arrow-flight/parquet to `26.0.0` (#6061) * chore: Update datafusion and arrow/arrow-flight/parquet to `26.0.0` * fix: Update query_functions * fix: update for TimestampNanosecondArray API changes * fix: update for TimestampNanosecondArray API changes * chore: Update flatbuffers and remove rustsec warning * chore: Update text * fix: update more test * fix: Lock ahash to exactly 0.8.0 * fix: Update datafusion pin * chore: Run cargo hakari tasks Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com> Co-authored-by: CircleCI[bot] <circleci@influxdata.com> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-07 11:01:58 +00:00
flatbuffers = "22.9.29"
feat: Write parquet to object storage in a similar way as parquet_file::Storage
2022-01-26 20:46:03 +00:00
futures = "0.3"
feat: Create a new proto definition for the new design's IoxMetadata
2022-01-27 19:44:20 +00:00
generated_types = { path = "../generated_types" }
refactor: use a workspace dependency for hashbrown
2022-11-11 17:12:30 +00:00
hashbrown = { workspace = true }
feat: Add scaffolding for ingester server * Adds a new ingester command to start an ingester server * Moves previous ingester server over to handler * Skeleton for gRPC and HTTP handlers
2022-01-21 23:02:19 +00:00
hyper = "0.14"
feat: commit in order to pull and merge new commit from main
2022-01-18 21:11:25 +00:00
iox_catalog = { path = "../iox_catalog" }
chore: Alphabetize some dependencies I can't handle not knowing where to look for a dependency or knowing where to add a new dependency.
2022-10-28 14:34:23 +00:00
iox_query = { path = "../iox_query" }
iox_time = { path = "../iox_time" }
feat: Add scaffolding for ingester server * Adds a new ingester command to start an ingester server * Moves previous ingester server over to handler * Skeleton for gRPC and HTTP handlers
2022-01-21 23:02:19 +00:00
metric = { path = "../metric" }
refactor: address review comments
2022-01-19 01:57:13 +00:00
mutable_batch = { path = "../mutable_batch"}
feat: prepare data to send to querier (#3825) * feat: changes needed to apply tombstones correctly on the life-cycle ingest bacthes * refactor: adjust the design after discussing with Paul * feat: apply the coming tombstone on all data but persiting one * chore: fmt * fix: build on buffer tombstone * test: delete & write tests for a parition and some cleanup * feat: No need add processed tombstones for newly created parquet file in the ingester becasue all deletes before that parquet file is created were applied * chore: cleanup * feat: intitial implementation for preparing data to send back to the Querier * feat: full implementation of prepare_data_to_querier * fix: apply filters for the batches * chore: Apply suggestions from code review Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com> * chore: cleanup * fix: typos in comments * fix: typos in comments * fix: typos in comments * test: create different scenarios and test them * chore: fix typos * test: add tests with deletes * chore: make pub pub(crate) * chore: Apply suggestions from code review Co-authored-by: Jake Goulding <jake.goulding@integer32.com> * refactor: address review comments * fix: keep batches in their arrival order * refactor: not assign unecessary values to enum * refactor: use bitflags enum * fix: use bitflags correctly * chore: Apply suggestions from code review Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * refactor: avoid using use at the end of the function * chore: merge main to branch * fix: fix downgrade versions * refactor: address review comments * chore: remove unnecessary comments * refactor: Make the whole test_utils module test-only and bring paths into module scope Co-authored-by: Paul Dix <paul@pauldix.net> Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com> Co-authored-by: Jake Goulding <jake.goulding@integer32.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
2022-03-01 01:00:45 +00:00
mutable_batch_lp = { path = "../mutable_batch_lp" }
chore(deps): Bump object_store from 0.5.1 to 0.5.2 (#6339) * chore(deps): Bump object_store from 0.5.1 to 0.5.2 Bumps [object_store](https://github.com/apache/arrow-rs) from 0.5.1 to 0.5.2. - [Release notes](https://github.com/apache/arrow-rs/releases) - [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md) - [Commits](https://github.com/apache/arrow-rs/compare/object_store_0.5.1...object_store_0.5.2) --- updated-dependencies: - dependency-name: object_store dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore: Run cargo hakari tasks Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: CircleCI[bot] <circleci@influxdata.com> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-06 07:53:54 +00:00
object_store = "0.5.2"
feat: Send IngesterQueryResponse data back as response of doGet Flight request (#3772) * fix: Adjust fields of IngesterQueryResponse * feat: Adjust IngestHandler query method to call prepare_data_to_querier * feat: Send ingest query result data back through Flight doGet * feat: Send delete predicates and max sequencer number in metadata * fix: greater_than_sequence_number should be of type SequenceNumber * fix: Remove DeletePredicates from IngesterQueryResponse Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-18 17:42:49 +00:00
observability_deps = { path = "../observability_deps" }
chore: Alphabetize some dependencies I can't handle not knowing where to look for a dependency or knowing where to add a new dependency.
2022-10-28 14:34:23 +00:00
once_cell = "1"
chore: upgrade rskafka and parking_lot (#3592)
2022-02-01 11:50:42 +00:00
parking_lot = "0.12"
refactor: Move IoxMetadata to parquet_file
2022-01-28 16:56:48 +00:00
parquet_file = { path = "../parquet_file" }
fix: supervise and shutdown ingester background tasks (#3769) * fix: supervise and shutdown ingester background tasks Closes #3761. Closes #3762. * docs: improve wording Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com> * test: join/shutdown handling for ingester Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2022-02-18 09:35:29 +00:00
pin-project = "1.0"
feat: initial implementation for ingester's compaction
2022-01-20 23:22:41 +00:00
predicate = { path = "../predicate" }
chore: Update datafusion / arrow/parquet/arrow-flight and prost/tonic ecosystem (#5360) * chore: Update datafusion and arrow * chore: Update Cargo.lock * chore: update to Decimal128 * chore: Update tonic/prost/pbjson/etc * chore: Run cargo hakari tasks * fix: doctest in generated types Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-08-09 17:30:44 +00:00
prost = "0.11"
chore: Alphabetize some dependencies I can't handle not knowing where to look for a dependency or knowing where to add a new dependency.
2022-10-28 14:34:23 +00:00
rand = "0.8.5"
feat: initial implementation for ingester's compaction
2022-01-20 23:22:41 +00:00
schema = { path = "../schema" }
feat: Add the catalog service to ingester, querier, and compactor So that `remote get` that uses the catalog service can work no matter what kind of server you contact.
2022-10-28 14:49:24 +00:00
service_grpc_catalog = { path = "../service_grpc_catalog"}
refactor: address review comments
2022-01-19 01:57:13 +00:00
snafu = "0.7"
feat: Add scaffolding for ingester server * Adds a new ingester command to start an ingester server * Moves previous ingester server over to handler * Skeleton for gRPC and HTTP handlers
2022-01-21 23:02:19 +00:00
thiserror = "1.0"
chore(deps): Bump tokio from 1.24.2 to 1.25.0 Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.24.2 to 1.25.0. - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](https://github.com/tokio-rs/tokio/commits/tokio-1.25.0) --- updated-dependencies: - dependency-name: tokio dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
2023-01-30 01:57:27 +00:00
tokio = { version = "1.25", features = ["macros", "parking_lot", "rt-multi-thread", "sync", "time"] }
chore: Alphabetize some dependencies I can't handle not knowing where to look for a dependency or knowing where to add a new dependency.
2022-10-28 14:34:23 +00:00
tokio-util = { version = "0.7.4" }
chore: Update datafusion / arrow/parquet/arrow-flight and prost/tonic ecosystem (#5360) * chore: Update datafusion and arrow * chore: Update Cargo.lock * chore: update to Decimal128 * chore: Update tonic/prost/pbjson/etc * chore: Run cargo hakari tasks * fix: doctest in generated types Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-08-09 17:30:44 +00:00
tonic = { version = "0.8" }
chore: Alphabetize some dependencies I can't handle not knowing where to look for a dependency or knowing where to add a new dependency.
2022-10-28 14:34:23 +00:00
trace = { path = "../trace" }
feat: track ingester jobs (#3836)
2022-02-23 15:33:47 +00:00
tracker = { path = "../tracker" }
chore(deps): Bump sqlx to 0.6.0 and uuid to 1 (#4894) Closes #4889. Closes #4890. Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-17 10:28:28 +00:00
uuid = { version = "1", features = ["v4"] }
fix: add workspace-hack
2022-01-18 23:10:42 +00:00
workspace-hack = { path = "../workspace-hack"}
feat: wire up the write buffer to the ingester process (#3533) This adds the scaffolding for the ingester server to consume data from Kafka. This ingests data in an in memory structure while creating records in the catalog for any partitions that don't yet exist. I've removed catalog_update.rs in ingester for now. That was mostly a placeholder and will be going in a combination of handler.rs and data.rs on my next PR which will have some primitive lifecycle wired up. There's one ugly bit here where the DML write is cloned because it's getting borrowed to output spans and metrics. I'll need to follow up with a refactor to make it so that the DML write's tables can be consumed without it gumming up the metrics stuff. Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-03 11:47:28 +00:00
write_buffer = { path = "../write_buffer" }
feat: Add `SequencerProgress` reporting to ingester (#4238) * feat: Add `SequencerProgress` reporting to ingester * refactor: Use KafkaPartition in write_summary * fix: Update docstrings * refactor: Change ingester to use KafkaPartition everywhere * refactor: add SequencerProgress::combine * refactor: return new SequencerProgress rather than updating * fix: distinguish between yes/no/unknown in WriteSummary * docs: Update data_types2/src/lib.rs Co-authored-by: Paul Dix <paul@pauldix.net> Co-authored-by: Paul Dix <paul@pauldix.net> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-06 15:13:21 +00:00
write_summary = { path = "../write_summary" }
feat(ingester): rpc write endpoint Adds a handler implementation of the gRPC WriteService to receive direct RPC writes from a router. This code is currently unused.
2022-11-16 18:11:39 +00:00
mutable_batch_pb = { version = "0.1.0", path = "../mutable_batch_pb" }
fix: merge record batches into one batch (#3535) * fix: merge record batches into one batch refactor: address review comments * chore: update test output
2022-01-25 23:29:16 +00:00
feat: Implement a snapshot method on DataBuffer (#3518) * feat: Implement a snapshot method on DataBuffer Fixes #3510. * test: Add a test snapshotting batches with different but compatible schemas * fix: Simplify min/max sequencer number collection The first batch should always have the min sequencer number. The last batch should always have the max sequencer number. The min should always be less than (or equal to, in case there's only one batch) the max.
2022-01-26 15:22:51 +00:00
[dev-dependencies]
refactor: ingester table & namespace count metrics Record the number of tables / namespaces an ingester process has observed.
2022-03-17 17:19:45 +00:00
assert_matches = "1.5.0"
refactor(ingester): use partition buffer FSM This commit makes use of the partition buffer state machine introduced in https://github.com/influxdata/influxdb_iox/pull/5943. This commit significantly changes the buffering, and querying, of data from a partition, swapping out the existing "DataBuffer" for the new state machine implementation (itself simplified due to temporary lack of incremental snapshot generation, see #5944). This commit simplifies the query path, removing multiple types that wrapped one-another to pass around various state necessary to perform a query, with various query functions needing different types or combinations of types. The query path now operates using a single type (named "QueryAdaptor") that provides a queryable interface over the set of RecordBatch returned from a partition. There is significantly increased testing of the PartitionData itself, covering data in various states and the ordering of returned RecordBatch (to ensure correct materialisation of updates). There are also invariants upheld by the type system / compiler to minimise the complexities of working with empty batches & states, and many asserts that ensure (mostly existing!) invariants are upheld.
2022-10-21 12:25:43 +00:00
lazy_static = "1.4.0"
chore(deps): Bump paste from 1.0.10 to 1.0.11 (#6430) Bumps [paste](https://github.com/dtolnay/paste) from 1.0.10 to 1.0.11. - [Release notes](https://github.com/dtolnay/paste/releases) - [Commits](https://github.com/dtolnay/paste/compare/1.0.10...1.0.11) --- updated-dependencies: - dependency-name: paste dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-19 10:31:05 +00:00
paste = "1.0.11"
refactor: ingest stream handler Refactors the stream_in_sequenced_entries() into a new impl in the SequencedStreamHandler type, decoupling the reading / decoding of ops from Kafka (and associated error handling) from the "what happens to those ops" concern to ease testing, encapsulate the specifics of "how to get an op" and improve flexibility. This is intended to provide robust error handling within what is reasonably possible (unexpected errors are always unexpected!) while retaining the existing metrics and functionality. I've also separated out code that exists in the current impl specifically to drive tests from the prod code path, instead driving those behaviours through mocks. As of this commit, the handler is not used - this commit simply adds the new impl.
2022-03-31 18:33:49 +00:00
test_helpers = { path = "../test_helpers", features = ["future_timeout"] }
chore(deps): Bump tokio-stream from 0.1.10 to 0.1.11 (#5838) Bumps [tokio-stream](https://github.com/tokio-rs/tokio) from 0.1.10 to 0.1.11. - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](https://github.com/tokio-rs/tokio/compare/tokio-stream-0.1.10...tokio-stream-0.1.11) --- updated-dependencies: - dependency-name: tokio-stream dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-12 12:37:24 +00:00
tokio-stream = {version = "0.1.11", default_features = false }