By default, the RocksDB C library is compiled using the number of
cores on the machine. In CircleCI, this is 36 cores. Unfortunately,
that appears to completely blow out the memory usage, causing the C
compiler invocations to be killed.
This commit reduces the *entire* parallelism of the build to avoid
that.
This configures a circleci pipeline that runs fmt, lint, test, and build operations.
I changed the fmt command to --check since --overwrite is no longer supported.
The pipeline will always run on nightly builds of rust.
This commit adds benchmarks for the float encoder and decoder. The
following scenarios are benchmarked:
- sequential values;
- random values;
- real CPU values (from Telegraf output).
Each scenario is benchmarked with a variety of block sizes.
This commit adds a test framework for the InvertedIndex and SeriesStore traits. Structs that implement these traits can call into their tests to ensure they work.
This commit also adds an in memory database that keeps time series data in a ring buffer per series and an inverted index in memory using a combination of HashMap and BTreeMap.
This commit pulls the database behavior into three traits: ConfigStore, InvertedIndex, and SeriesStore. The ConfigStore holds the Bucket definitions, the InvertedIndex holds the index of measurement names, tags, and fields, and the SeriesStore holds the raw time series data indexed only by bucket id, series id, and time.
This is the initital work to enable memory and S3 backed databases. It is also the preliminary work to break data out into shards.
There was a race condition when inserting new series into RocksDB that would cause series being inserted by two separate threads to cause an error. The thread that inserts is fine, but the one after would see that it inserted, but not read in the ID so it can later be used to write the points.
The earlier version of this line protocol parser incorrectly used a space as a delimiter between fields. This updates it to use a comma as it is in InfluxDB 1.x and 2.x.
Updates to read API in main.rs to return values for float series. I'm not terribly happy with the way I had to do this, but I was struggling a bit with the type system gymnastics. I assume I'll have to revisit this anyway when I add support for other storage backends.
Adds support for for f64 time series in RocksDB. Series data types are now stored in the index under the id to key mapping, which is now id to type and key.
This doesn't enforce the same data type for values being written into a series, which will come later. Also later will be adding support for float64 series in the read API.
Adds support for f64 to the line protocol parser. Also updates the return value of parse to return a Vec of mixed type points that can be later written into the database.
The PointType struct is only for use in this context. In the context of querying or working with time series for compaction, we'll want vectors of actual typed points of the same kind so we don't have to do inefficient enum matches.
This commit fixes all the linter warnings. However, there are a number of spots, particularly in the encoders where I added `#[allow(dead_code)]` to get past them. We should go through and fix those up at some point, I'll log an issue to track.
This commit adds a basic read endpoint to pull data out of the database. In order to provide the basic functionality a few things were added:
* Time package with limited support for parsing Flux style durations
* API endpoint at /api/v2/read with query paramters of org_id, bucket_name, predicate, start, and stop
The start and stop query parameters only support relative durations.
The predicate parameter supports what is possible in the parse_predicate method and in the RocksDB implementation (only == comparisons on tags and AND or OR)
This commit adds iterators for iterating over series and batches of points for a read range request. The exact signature/structure of the iterators are likely to change when this is generalized for other data types and other storage backends (S3 & memory).
This commit updates the write_points method to use the bucket id and series id in the key for a stored point value.
It also updates the Database methods to be immutable borrows moving any mutable concerns into interior structures so it can be easily called from many threads.
This commit brings in a Roaring Bitmap implementation to keep postings lists of tag key/value pairs to the set of series ids that have those pairs. The croaring implementation was used becasue the Treemap was required for u64 support for series ids and it was serializable (unlike the other pure Rust roaring implementation).
This doesn't shard the postings lists based on size. It also doesn't implement the time/index levels.
The predicate matching currently only works for a simple key = "value" match.
This commit is the beginning of the RocksDB based index for series and their tag metadata.
I started to stub out different index levels but stopped short of implementing them.
There are a number of spots where I'm unwrapping return values that we may want to revisit later. For now I want to have the program panic if those things pop up.
Jake Goulding
Edd Robinson
AJ
AJ Bond
Paul Dix
Paul Dix