Refactored Partition to be a concrete type enum with three different types of partitions: memory, s3, or remote server. Had to move to this rather than a trait because the methods on partition are async.
Refactored database to use the new partition structure and for now only use a memory based database. Persistence will be reintroduced later with the S3 partition implementation.
Refactored main and rpc to use the new database.
Update ReadMergeStream to use a single vec to store stream state.
Update ReadBatch to always assume values in the batch. Update memdb to not put batches with no values in the result set.
Needed because the PartialOrd/Ord trait implementations would then
disagree with the PartialEq/Eq trait implementations, which isn't really
what's desired.
Change the method name to make it clearer that this isn't using the Ord
trait's sort.
This adds the Partition trait along with helper functions and structs to combine the results from multiple partitions.
It also refactors MemDB to have methods that align with the partition trait so that it can be used later by database.rs.
There are obvious places where it is inefficient, but this should be enough to get started with refactoring database.rs to use this rather than rocksdb, series_store, and inverted_index.
As suggested by the redundant_clone clippy lint.
This is a little weird because different constructions of `points`
now need clones in different places, but rustc will tell us when
we need to clone and clippy will tell us when we don't.
As found by the identity_conversion clippy lint. Clippy was warning
about this on master, but warnings didn't fail the build until a commit
I'm adding in this PR.
As found by the float_cmp clippy lint.
There are crates that provide macros for this, but it's small enough
that I think a test helper function in-tree is fine.
The way these tests are written, any truncation will be made on both the
values used in the test and the values expected in the test, so this is
probably fine.
But we might not be testing with the exact values we think we're testing
with, so we should audit the values clippy warns about at some point.
For reference, the [2018 edition guide][guide] talks about some of the
big differences.
The two that are applied here are:
1. `extern crate` is basically not needed at all anymore; you can do
`use cratename` instead. This makes importing things more uniform
between your own crate and other crates.
1. Rust does a reasonable amount of [*lifetime elision*][elision] so
we don't have to type `'a` in as many places. However, one that
ended up tripping up people is when a generic lifetime was part of
a type. The compiler cared about this lifetime, but since it
wasn't visible, people would forget it's there, then try to use it
as if it wasn't constrained by the lifetime.
A good example is the `Chars` iterator. It references the original
`&str` and cannot live longer than the string. With the original
way this was being passed (`&mut Chars`) it was visually evident
that there was *some* lifetime, thanks to seeing the `&`, but it
wasn't obvious that there's *another* lifetime — the string.
With the addition of the *anonymous lifetime* (`'_`), it's now
encouraged to use that when a type has a lifetime parameter that
isn't relevant to prevent confusing mistakes that lead to compiler
errors.
There are probably a few more things enabled by the lint as well. I
forget the exact reason that these are not yet enabled by default,
though.
[guide]: https://doc.rust-lang.org/edition-guide/rust-2018/index.html
[elision]: https://doc.rust-lang.org/book/ch10-03-lifetime-syntax.html#lifetime-elision
This commit adds benchmarks for the float encoder and decoder. The
following scenarios are benchmarked:
- sequential values;
- random values;
- real CPU values (from Telegraf output).
Each scenario is benchmarked with a variety of block sizes.
This commit adds a test framework for the InvertedIndex and SeriesStore traits. Structs that implement these traits can call into their tests to ensure they work.
This commit also adds an in memory database that keeps time series data in a ring buffer per series and an inverted index in memory using a combination of HashMap and BTreeMap.
This commit pulls the database behavior into three traits: ConfigStore, InvertedIndex, and SeriesStore. The ConfigStore holds the Bucket definitions, the InvertedIndex holds the index of measurement names, tags, and fields, and the SeriesStore holds the raw time series data indexed only by bucket id, series id, and time.
This is the initital work to enable memory and S3 backed databases. It is also the preliminary work to break data out into shards.
There was a race condition when inserting new series into RocksDB that would cause series being inserted by two separate threads to cause an error. The thread that inserts is fine, but the one after would see that it inserted, but not read in the ID so it can later be used to write the points.
The earlier version of this line protocol parser incorrectly used a space as a delimiter between fields. This updates it to use a comma as it is in InfluxDB 1.x and 2.x.
Updates to read API in main.rs to return values for float series. I'm not terribly happy with the way I had to do this, but I was struggling a bit with the type system gymnastics. I assume I'll have to revisit this anyway when I add support for other storage backends.
Adds support for for f64 time series in RocksDB. Series data types are now stored in the index under the id to key mapping, which is now id to type and key.
This doesn't enforce the same data type for values being written into a series, which will come later. Also later will be adding support for float64 series in the read API.
Adds support for f64 to the line protocol parser. Also updates the return value of parse to return a Vec of mixed type points that can be later written into the database.
The PointType struct is only for use in this context. In the context of querying or working with time series for compaction, we'll want vectors of actual typed points of the same kind so we don't have to do inefficient enum matches.
This commit fixes all the linter warnings. However, there are a number of spots, particularly in the encoders where I added `#[allow(dead_code)]` to get past them. We should go through and fix those up at some point, I'll log an issue to track.
This commit adds a basic read endpoint to pull data out of the database. In order to provide the basic functionality a few things were added:
* Time package with limited support for parsing Flux style durations
* API endpoint at /api/v2/read with query paramters of org_id, bucket_name, predicate, start, and stop
The start and stop query parameters only support relative durations.
The predicate parameter supports what is possible in the parse_predicate method and in the RocksDB implementation (only == comparisons on tags and AND or OR)
This commit adds iterators for iterating over series and batches of points for a read range request. The exact signature/structure of the iterators are likely to change when this is generalized for other data types and other storage backends (S3 & memory).
This commit updates the write_points method to use the bucket id and series id in the key for a stored point value.
It also updates the Database methods to be immutable borrows moving any mutable concerns into interior structures so it can be easily called from many threads.
This commit brings in a Roaring Bitmap implementation to keep postings lists of tag key/value pairs to the set of series ids that have those pairs. The croaring implementation was used becasue the Treemap was required for u64 support for series ids and it was serializable (unlike the other pure Rust roaring implementation).
This doesn't shard the postings lists based on size. It also doesn't implement the time/index levels.
The predicate matching currently only works for a simple key = "value" match.
This commit is the beginning of the RocksDB based index for series and their tag metadata.
I started to stub out different index levels but stopped short of implementing them.
There are a number of spots where I'm unwrapping return values that we may want to revisit later. For now I want to have the program panic if those things pop up.
This updates to build system to use Prost to build the protobuf objects.
It adds tests for creating, storing and loading bucket definitions.
The tests use an actual on disk RocksDB implementation to ensure that its tested all the way to persistence.
Carol (Nichols || Goulding)
Edd Robinson
Carol (Nichols || Goulding)
Jake Goulding
Paul Dix