Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
influxdb/Cargo.toml

116 lines
2.5 KiB
TOML
Raw Normal View History

Initial commit with some notes and proto
2019-11-22 21:59:04 +00:00
[package]
name = "delorean"
version = "0.1.0"
authors = ["Paul Dix <paul@pauldix.net>"]
edition = "2018"
feat: Utility to take line protocol and make write requests
2020-04-15 20:00:54 +00:00
default-run = "delorean"
Initial commit with some notes and proto
2019-11-22 21:59:04 +00:00
refactor: Move the point utilities into a workspace crate
2020-04-23 15:26:37 +00:00
[workspace]
refactor: extract a crate with our custom assertions There's probably an existing crate that we should use directly, but I haven't found an exact match yet.
2020-05-01 16:52:00 +00:00
members = [
refactor: hoist tsm reader into own crate
2020-06-17 08:49:21 +00:00
"delorean_generated_types",
feat: Line Protocol Schema extractor (#108) * feat: schema inference from iterator of parsed lines * fix: Clean up error handing even more * fix: fmt * fix: make a sacrifice to the clippy gods
2020-06-03 22:29:57 +00:00
"delorean_ingest",
refactor: extract the line protocol parser to a separate crate This will facilitate reusing the parser for other tasks.
2020-05-22 20:46:02 +00:00
"delorean_line_parser",
feat: Object storage in S3 and GCS
2020-05-29 15:06:06 +00:00
"delorean_object_store",
feat: Add parquet writer, hook up conversion in dstool (#124) * feat: Add parquet writer, hook up conversion in dstool * fix: use bigger executor for test * fix: less cloning * fix: make unsupported messages less pejorative * fix: fmt * fix: Rename writer and do not require std::File, add example * fix: clippy and fmt * fix: remove unnecessary module in end to end tests * fix: remove strange use of tempfile * fix: Apply suggestions from code review Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com> * fix: Apply suggestions from code review Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com> * fix: cleanup use * fix: Use more specific error messages * fix: comment tweak * fix: touchup temp path creation * fix: clippy! Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-06-08 20:25:24 +00:00
"delorean_parquet",
feat: Add column packing code (#114) * feat: Add column packing code * fix: remove dependency on assert_approx_equal in favor of delorean_test_helpers * fix: Cleanups from pr comments * fix: Apply suggestions from code review Co-authored-by: Jake Goulding <jake.goulding@integer32.com> Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com> * fix: more cleanup per code review * fix: pr comments * fix: remove explict string creation from caller Co-authored-by: Jake Goulding <jake.goulding@integer32.com> Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-06-06 10:04:41 +00:00
"delorean_table",
refactor: Rename crate line_protocol_schema --> delorean_table_schema (#129) * refactor: Rename crate line_protocol_schema --> delorean_table_schema * fix: fmt
2020-06-09 15:56:16 +00:00
"delorean_table_schema",
refactor: extract a crate with our custom assertions There's probably an existing crate that we should use directly, but I haven't found an exact match yet.
2020-05-01 16:52:00 +00:00
"delorean_test_helpers",
refactor: hoist tsm reader into own crate
2020-06-17 08:49:21 +00:00
"delorean_tsm",
feat: Introduce a WAL tailored for delorean
2020-05-11 19:09:10 +00:00
"delorean_wal",
feat: Set up an InfluxData 2.0 client crate
2020-07-15 19:10:43 +00:00
"influxdb2_client",
refactor: extract a crate with our custom assertions There's probably an existing crate that we should use directly, but I haven't found an exact match yet.
2020-05-01 16:52:00 +00:00
]
Initial commit with some notes and proto
2019-11-22 21:59:04 +00:00
perf: Add a benchmark for the line parser
2020-03-27 20:39:22 +00:00
[profile.release]
debug = true
Initial commit with some notes and proto
2019-11-22 21:59:04 +00:00
[dependencies]
style: Re-enable the elided lifetimes lint and move generated types to their own crate (#119) * refactor: rename the module containing generated types The nested `delorean` was confusing anyway, and this will make more sense when we extract a new crate. * refactor: Move the generated types to their own crate This allows us to have more lax warnings in that crate alone, keeping the main crate more strict. * style: Re-enable elided lifetimes lint in the main crate
2020-06-05 20:22:27 +00:00
delorean_generated_types = { path = "delorean_generated_types" }
refactor: move all dstool code into delorean binary (#131) * refactor: move all dstool code into delorean binary * fix: Move code/mods to make it compile and run * fix: warn if db dir does not exist * refactor: Match argument subcommands w/ more idomatic rust * fix: Apply suggestions from code review Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com> fix: restore hyper logging fix: Apply suggestions from code review Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com> * fix: update expected code Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-06-10 20:04:46 +00:00
delorean_ingest = { path = "delorean_ingest" }
refactor: extract the line protocol parser to a separate crate This will facilitate reusing the parser for other tasks.
2020-05-22 20:46:02 +00:00
delorean_line_parser = { path = "delorean_line_parser" }
refactor: move all dstool code into delorean binary (#131) * refactor: move all dstool code into delorean binary * fix: Move code/mods to make it compile and run * fix: warn if db dir does not exist * refactor: Match argument subcommands w/ more idomatic rust * fix: Apply suggestions from code review Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com> fix: restore hyper logging fix: Apply suggestions from code review Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com> * fix: update expected code Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-06-10 20:04:46 +00:00
delorean_parquet = { path = "delorean_parquet" }
delorean_table = { path = "delorean_table" }
delorean_table_schema = { path = "delorean_table_schema" }
feat: Introduce a WAL tailored for delorean
2020-05-11 19:09:10 +00:00
delorean_wal = { path = "delorean_wal" }
feat: Object storage in S3 and GCS
2020-05-29 15:06:06 +00:00
delorean_object_store = { path = "delorean_object_store" }
refactor: hoist tsm reader into own crate
2020-06-17 08:49:21 +00:00
delorean_tsm = { path = "delorean_tsm" }
feat: Introduce a WAL tailored for delorean
2020-05-11 19:09:10 +00:00
build: Update bytes to latest version
2020-02-14 18:02:39 +00:00
bytes = "0.5.4"
feat(encoding): add RLE encoder/decoder
2019-12-09 18:04:01 +00:00
integer-encoding = "1.0.7"
Initial commit with some notes and proto
2019-11-22 21:59:04 +00:00
refactor: Switch to a hyper server
2020-02-13 18:07:29 +00:00
hyper = "0.13"
tokio = { version = "0.2", features = ["full"] }
Add server skeleton with Actix and RocksDB
2019-12-11 23:03:00 +00:00
test: traits for database and tests for http handler (#284) * test: traits for database and tests for http handler * refactor: Use generics and trait bounds instead of trait objects * refactor: Replace trait objects with an associated type * refactor: Extract an associated Error type on the Database traits * refactor: Remove some explicit conversions to_string that Snafu takes care of * docs: add comments * refactor: move traits into storage module Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@integer32.com>
2020-09-11 21:42:00 +00:00
async-trait = "0.1.40"
refactor: move all dstool code into delorean binary (#131) * refactor: move all dstool code into delorean binary * fix: Move code/mods to make it compile and run * fix: warn if db dir does not exist * refactor: Match argument subcommands w/ more idomatic rust * fix: Apply suggestions from code review Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com> fix: restore hyper logging fix: Apply suggestions from code review Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com> * fix: update expected code Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-06-10 20:04:46 +00:00
clap = "2.33.1"
build: Update dotenv to latest version
2020-02-14 16:32:08 +00:00
dotenv = "0.15.0"
feat: automatically create db dir and test dir
2020-01-06 11:29:11 +00:00
dirs = "2.0.2"
build: Update env_logger to latest version
2020-02-14 16:30:52 +00:00
env_logger = "0.7.1"
refactor: Switch to a hyper server
2020-02-13 18:07:29 +00:00
log = "0.4.8"
Add server skeleton with Actix and RocksDB
2019-12-11 23:03:00 +00:00
futures = "0.3.1"
serde_json = "1.0.44"
build: Upgrade assert_cmd to 1.0 This requires that we opt into the serde `derive` feature that is no longer implicitly added from upstream.
2020-04-05 20:32:32 +00:00
serde = { version = "1.0", features = ["derive"] }
Add basic read endpoint This commit adds a basic read endpoint to pull data out of the database. In order to provide the basic functionality a few things were added: * Time package with limited support for parsing Flux style durations * API endpoint at /api/v2/read with query paramters of org_id, bucket_name, predicate, start, and stop The start and stop query parameters only support relative durations. The predicate parameter supports what is possible in the parse_predicate method and in the RocksDB implementation (only == comparisons on tags and AND or OR)
2020-01-05 00:07:54 +00:00
csv = "1.1"
feat: upgrade to latest arrow / byteorder (#154)
2020-06-17 16:50:23 +00:00
byteorder = "1.3.4"
Add server skeleton with Actix and RocksDB
2019-12-11 23:03:00 +00:00
feat(storage): Implement bucket definitiions and persistence This updates to build system to use Prost to build the protobuf objects. It adds tests for creating, storing and loading bucket definitions. The tests use an actual on disk RocksDB implementation to ensure that its tested all the way to persistence.
2019-12-17 17:41:28 +00:00
num_cpus = "1.11.1"
build: Upgrade tonic to 0.2
2020-04-05 20:35:00 +00:00
tonic = "0.2.0"
build: Update prost to latest version
2020-02-14 16:39:59 +00:00
prost = "0.6.1"
feature: Add the storage gRPC proto definitions
2020-02-19 14:43:38 +00:00
prost-types = "0.6.1"
feat: enable rpc debug tracing, tweaked logging levels, respect RUST_FMT env var
2020-06-29 12:45:31 +00:00
tracing = "0.1"
tracing-futures="0.2.4"
Add server skeleton with Actix and RocksDB
2019-12-11 23:03:00 +00:00
feat: add Block Type This commit adds a new Block type, which is used to keep track of values associated with individual block, and then serialise them.
2020-04-06 14:19:12 +00:00
crc32fast = "1.2.0"
num = "0.2.1"
Add basic tag key/value index This commit brings in a Roaring Bitmap implementation to keep postings lists of tag key/value pairs to the set of series ids that have those pairs. The croaring implementation was used becasue the Treemap was required for u64 support for series ids and it was serializable (unlike the other pure Rust roaring implementation). This doesn't shard the postings lists based on size. It also doesn't implement the time/index levels. The predicate matching currently only works for a simple key = "value" match.
2019-12-24 18:44:30 +00:00
# using croaring here because we needed Treemap support for u64
# for series ids and we needed serialization, which the pure Rust
# crate didn't offer.
feat: Update croaring Jake dug into why the end-to-end tests fail with delorean running in the Docker image I built, and it appears to be a crash with an illegal instruction from CRoaring. We think it's this issue: https://github.com/saulius/croaring-rs/pull/62 which was merged and released, so let's try updating CRoaring.
2020-07-08 12:39:34 +00:00
croaring = "0.4.5"
refactor: Switch to a hyper server
2020-02-13 18:07:29 +00:00
http = "0.2.0"
serde_urlencoded = "0.6.1"
feat: Error when parsing lines with duplicate tags
2020-03-12 02:02:13 +00:00
snafu = "0.6.2"
either = "1.5.3"
feat: Integrate the WAL with delorean
2020-04-24 14:36:59 +00:00
flatbuffers = "0.6.1"
refactor: move all dstool code into delorean binary (#131) * refactor: move all dstool code into delorean binary * fix: Move code/mods to make it compile and run * fix: warn if db dir does not exist * refactor: Match argument subcommands w/ more idomatic rust * fix: Apply suggestions from code review Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com> fix: restore hyper logging fix: Apply suggestions from code review Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com> * fix: update expected code Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-06-10 20:04:46 +00:00
libflate = "1.0.0"
feat: Initial prototype of WriteBuffer and WAL (#271) This is the initial prototype of the WriteBuffer and WAL. This does the following: * accepts a slice of ParsedLine into the DB * writes those into an in memory structure with tags represented as u32 dictionaries and all field types supported * persists those writes into the WAL as Flatbuffer blobs (one WAL entry per slice of lines written, or WriteBatch) * has a method to return a table from the buffer as an Arrow RecordBatch * recovers the WAL after the database is closed and opened back up again * has a single test that covers the end-to-end from the DB side * It doesn't include partitioning yet. Although the write_lines method does actually try to do partitions on time. That'll get changed to be something more general defined by a per database configuration. * hooked up to the v2 HTTP write API * hooked up to a read API which will execute a SQL query against the data in the buffer This includes a refactor of the WAL: Refactors the WAL to remove async and threading so that it can be moved higher up. This simplifies the API while keeping just about the same amount of code in ParitionStore to handle the asynchronous writes. This also modifies the WAL to remove the SideFile implementation, which was causing significant performance problems and write amplification. The downside is that WAL writes are no longer guarranteed atomic. Further, this modifies the WAL to keep the active segement file handle open. Appends now don't have to list the directory contents and look for the latest file and open the file handle to do appends, which should also improve performance and reduce iops.
2020-09-08 18:12:16 +00:00
arrow = { git = "https://github.com/apache/arrow.git", rev = "62dfa114d6683172927fab40fa6c4ddabae8fef4"}
string-interner = "0.12.0"
chrono = "0.4"
sqlparser = "0.6.1"
datafusion = { git = "https://github.com/apache/arrow.git", rev = "62dfa114d6683172927fab40fa6c4ddabae8fef4" }
Add basic tag key/value index This commit brings in a Roaring Bitmap implementation to keep postings lists of tag key/value pairs to the set of series ids that have those pairs. The croaring implementation was used becasue the Treemap was required for u64 support for series ids and it was serializable (unlike the other pure Rust roaring implementation). This doesn't shard the postings lists based on size. It also doesn't implement the time/index levels. The predicate matching currently only works for a simple key = "value" match.
2019-12-24 18:44:30 +00:00
test: add float encoder/decoder bencmarks This commit adds benchmarks for the float encoder and decoder. The following scenarios are benchmarked: - sequential values; - random values; - real CPU values (from Telegraf output). Each scenario is benchmarked with a variety of block sizes.
2020-01-20 15:41:38 +00:00
[dev-dependencies]
build: Upgrade assert_cmd to 1.0 This requires that we opt into the serde `derive` feature that is no longer implicitly added from upstream.
2020-04-05 20:32:32 +00:00
assert_cmd = "1.0.0"
feat: support org and bucket ID in entries
2020-05-28 15:11:10 +00:00
criterion = "0.3"
delorean_test_helpers = { path = "delorean_test_helpers" }
feat: add API for series key information
2020-06-02 10:40:41 +00:00
hex = "0.4.2"
feat: Set up an InfluxData 2.0 client crate
2020-07-15 19:10:43 +00:00
influxdb2_client = { path = "influxdb2_client" }
feat: support org and bucket ID in entries
2020-05-28 15:11:10 +00:00
libflate = "1.0.0"
refactor: Move the point utilities into a workspace crate
2020-04-23 15:26:37 +00:00
rand = "0.7.2"
fix: Remove the blocking feature from reqwest This should have been removed in fdcef79 but wasn't.
2020-07-15 19:48:17 +00:00
reqwest = "0.10.1"
feat: support org and bucket ID in entries
2020-05-28 15:11:10 +00:00
predicates = "1.0.4"
tempfile = "3.1.0"
test: add float encoder/decoder bencmarks This commit adds benchmarks for the float encoder and decoder. The following scenarios are benchmarked: - sequential values; - random values; - real CPU values (from Telegraf output). Each scenario is benchmarked with a variety of block sizes.
2020-01-20 15:41:38 +00:00
[[bench]]
refactor: Move all encoder benchmarks to one file
2020-02-12 15:31:41 +00:00
name = "encoders"
test: add timestamp benchmarks
2020-02-07 13:26:50 +00:00
harness = false
perf: Add a benchmark for the line parser
2020-03-27 20:39:22 +00:00
[[bench]]
name = "line_parser"
harness = false
perf: Improve line protocol --> parquet conversion performance by ~20% (#177) * feat: benchmark for lp->parquet performance * feat: improve parser performance by storing contiguous EscapedStr * fix: remove all string copies during LP-Parquet conversion * refactor: Implement from_str as From<&str> only * refactor: implement Deref instead of as_str * refactor: Remove ends_with because Deref now makes it work * refactor: Eq can be derived * refactor: Remove unused From implementation * refactor: Replace single-character strings with chars as requested by clippy Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@integer32.com>
2020-06-23 09:42:19 +00:00
test: add benchmark for mapping fields
2020-07-02 16:39:05 +00:00
[[bench]]
name = "mapper"
harness = false
perf: Improve line protocol --> parquet conversion performance by ~20% (#177) * feat: benchmark for lp->parquet performance * feat: improve parser performance by storing contiguous EscapedStr * fix: remove all string copies during LP-Parquet conversion * refactor: Implement from_str as From<&str> only * refactor: implement Deref instead of as_str * refactor: Remove ends_with because Deref now makes it work * refactor: Eq can be derived * refactor: Remove unused From implementation * refactor: Replace single-character strings with chars as requested by clippy Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@integer32.com>
2020-06-23 09:42:19 +00:00
[[bench]]
name = "line_protocol_to_parquet"
harness = false
test: add benchmarks for packers
2020-07-30 11:54:51 +00:00
[[bench]]
name = "packers"
harness = false