7f9eaf51d5
Merge pull request #74 from influxdata/cn-generate-points
2020-04-24 08:08:32 -04:00
f1d5d50b92
Merge pull request #68 from influxdata/er-block-writer
...
feat: add Block Type
2020-04-23 22:48:38 +01:00
fa69101945
refactor: Move the point utilities into a workspace crate
2020-04-23 11:26:37 -04:00
2ecb3b0525
feat: Add some debug logging when receiving a write request
2020-04-23 11:11:56 -04:00
6791b9598c
feat: Utility to take line protocol and make write requests
2020-04-23 11:11:51 -04:00
0956b665d8
feat: Add a utility to generate points in line protocol to stdout
2020-04-23 11:11:46 -04:00
4ee8c1ac46
docs: clarify checksum usage
2020-04-23 10:39:17 +01:00
e5e3903771
fix: seek back to last offset
2020-04-23 10:39:17 +01:00
7d625648c9
refactor: remove redundant block members
2020-04-23 10:25:00 +01:00
8dc258b153
refactor: improve push
2020-04-22 13:45:41 +01:00
138c54b1a9
refactor: return option to reference
2020-04-22 12:18:51 +01:00
3a9f8aeeb8
docs: specify endianness
2020-04-21 17:31:59 +01:00
79506f3d3b
refactor: use u16 for summary counts
2020-04-21 17:25:07 +01:00
d31bcd2c0e
fix: Support negative timestamp values
...
Also add proper errors for out-of-range values.
2020-04-17 10:32:58 -04:00
4aae16066b
fix: Support negative integer and floating-point field values
...
Also add proper errors for out-of-range values.
2020-04-17 10:32:55 -04:00
40f8104e58
fix: Correct reference of RocksDB to MemDB; rest of comment still applies
2020-04-13 16:40:38 -04:00
fb8f736e54
refactor: Encode that Id cannot be 0 by using NonZeroU64
2020-04-13 16:40:38 -04:00
6dd7c0a3e8
fix: Treat gRPC bucket_id as bucket name because MemDB assigns bucket ID
2020-04-13 16:40:38 -04:00
aecb390fcd
feature: Add a create bucket HTTP api
...
For convenience when using cURL; easier to generate requests than using
gRPC.
2020-04-13 16:40:38 -04:00
429d0e1851
fix: Change name of the proto package to match proto used by Go
...
So that `storectl query` requests will work with delorean
2020-04-13 16:40:38 -04:00
a85f0edfc6
feature: Use the new ID type everywhere
2020-04-13 16:40:38 -04:00
7be1d3437e
feature: Add an ID type that matches the Go code
...
16 character hexadecimal strings to u64s. All zeroes are invalid.
2020-04-13 16:40:38 -04:00
27bb803ad3
fix: Return HTTP No Content on successful write
...
The `inch` tool expects status code 204 instead of 200, so actually send
no content instead of empty JSON.
2020-04-13 16:40:38 -04:00
4a0e2cddb9
test: Ensure the parser supports lines with a field but no time
2020-04-13 16:40:38 -04:00
9296eb4802
feature: If a line has no time, use the current time
2020-04-13 16:40:38 -04:00
0314bc6ba2
fix: Change the default gRPC port to match storectl
...
8082 appears to be the default here:
b304e751e1/cmd/storectl/query/query.go (L56)
2020-04-13 16:03:23 -04:00
d28d7440de
fix: Disallow elements that end with a backslash
2020-04-13 09:46:55 -04:00
8dfe58ca25
feat: Allow parsing line protocol without a provided timestamp
2020-04-13 09:46:02 -04:00
c29a2b725a
fix: Require at least one field in a field set
2020-04-13 09:41:05 -04:00
d7416da962
fix: Do not allow a raw newline in the middle of measurement / tag_set / field_key
2020-04-13 09:41:05 -04:00
261528aeab
test: Ensure that we allow newline literals as unknown escapes
2020-04-13 09:41:05 -04:00
aff4906a72
fix: Allow multiple whitespace characters between line protocol elements
2020-04-13 09:41:05 -04:00
9b9ddc8508
test: ensure that blank and commented lines are ignored
2020-04-13 09:41:04 -04:00
91bbfe454e
fix: Remove usage of str::lines as input can contain escaped newlines
2020-04-13 09:41:04 -04:00
5900111b01
style: Use Vec::new in closures instead of vec!
2020-04-10 15:01:05 -04:00
55ca9bc6b0
fix: DRY up repeated varint writing
2020-04-10 11:07:54 -04:00
3e268c00c5
fix: shorten summary add methods
2020-04-10 11:07:54 -04:00
48f5744f5a
fix: remove manual checks when building new block summaries
2020-04-10 11:07:54 -04:00
3fc9c0047d
fix: use stack-allocated array instead of a Vec
2020-04-10 10:16:29 -04:00
16be0b1922
fix: Use std instead of core
2020-04-08 17:03:05 -04:00
27520b9951
doc: correct typo
2020-04-08 16:41:04 -04:00
56d4d7e301
doc: use initial indentation in code fences
2020-04-08 16:39:17 -04:00
8a9f768f71
fix: Remove unneeded match and include basic error information
2020-04-08 16:32:00 -04:00
29279d393d
perf: Avoid string allocation when series is in the correct format
...
This increases the performance from 58.194 MiB/s to 126.68 MiB/s
2020-04-08 14:41:42 -04:00
93231c64e0
perf: Use a SmallVec for escaped strings and sets of tags and values
...
This increases the performance from 56.531 MiB/s to 58.194 MiB/s.
2020-04-08 14:41:42 -04:00
6fdd8b1935
refactor: Extract a method to generate the series base name
2020-04-08 14:41:17 -04:00
0c7a4a5f81
feat: Handle escaping in the tag key, tag value, and field key
2020-04-08 14:41:17 -04:00
8156e24255
feat: Handle escaping in the measurement name
2020-04-08 14:41:17 -04:00
1615aec46a
refactor: Use preceded combinator
2020-04-08 14:41:17 -04:00
8635076471
fix: Use write_all to ensure entire buffer is processed
...
write doesn't guarantee the whole buffer gets written and returns the
number of bytes it was able to write; whereas what's desired is
everything gets written and the number of bytes is the total of what
we're asking to write.
2020-04-08 11:18:17 -04:00
2879b6cad7
refactor: Remove some now-unneeded type annotations
2020-04-08 10:55:57 -04:00
051836ee4d
fix: Remove vec allocation where an inline slice will do
2020-04-08 10:33:54 -04:00
aa7ae9adaa
fix: Remove unnecessary casting
...
And replace necessary casting/asserts with conversions that panic if
they fail
2020-04-08 10:31:06 -04:00
a143708321
refactor: Remove unneeded `let _ =`
2020-04-08 10:02:32 -04:00
2a008dc6a7
refactor: Rewrite to not use unwrap
2020-04-08 09:48:38 -04:00
1a6d5c2163
fix: Return a slice of timestamp and values; let caller clone if needed
2020-04-08 09:36:28 -04:00
e4b6100f53
refactor: Use stdlib's `unzip`
2020-04-08 09:33:59 -04:00
367f96981b
fix: Remove unneeded clone
2020-04-08 09:12:40 -04:00
3d97829197
fix: Move dead code allow to be more specific
2020-04-08 09:07:15 -04:00
85e558844c
fix: Only use lifetimes when the type is a reference
2020-04-08 09:07:15 -04:00
47661f93f1
fix: Remove module declaration for module that no longer exists
2020-04-08 09:06:44 -04:00
4ee29db95c
docs: Small grammar/style fixes in doc comments I noticed while reading through
2020-04-08 08:45:31 -04:00
14369ae2ab
style: Wrap doc comments to 100 for ease of reading
2020-04-08 08:45:31 -04:00
9e20743b2c
feat: add Block Type
...
This commit adds a new Block type, which is used to keep track of values
associated with individual block, and then serialise them.
2020-04-08 13:37:48 +01:00
e60ae51c91
Merge pull request #60 from influxdata/cn-remove-dead-code
...
refactor: Remove code that's now unused
2020-04-06 17:00:56 -04:00
3ee7ba460d
Merge pull request #63 from influxdata/cn-read-group-take-3
...
feature: Implement ReadGroup gRPC
2020-04-06 17:00:33 -04:00
56a2b323be
feature: Implement ReadGroup
2020-04-06 10:23:43 -04:00
39dad5de31
fix: Always return long form of _f and _m from gRPC calls
2020-04-03 17:20:00 -04:00
a801f19b70
refactor: Extract a method to get sorted tags from ReadBatches
2020-04-03 17:18:05 -04:00
04fde87987
fix: Remove unused clone
2020-04-03 16:49:47 -04:00
79deaa3696
fix: index_pairs can't actually fail, so don't return Result
2020-04-03 16:06:39 -04:00
d0e339e590
refactor: Remove allowing of dead code in MemDB because it's used now
2020-04-03 15:34:37 -04:00
97843a5c90
refactor: Remove now unused SeriesStore
2020-04-03 15:30:52 -04:00
c0102f349b
refactor: Move ReadPoint's definition to the storage module
2020-04-03 15:30:11 -04:00
c163ee9134
refactor: Remove now unused ConfigStore
2020-04-03 15:22:55 -04:00
404e3cf2e5
refactor: Remove now unused InvertedIndex
2020-04-03 15:21:58 -04:00
9d1670b1bb
refactor: Extract a function for sending points
2020-04-03 15:15:37 -04:00
7356bc33ed
refactor: Rearrange construction of data
...
This was a temporary way of writing it to figure out the types, but I
find it easier to see the hierarchy.
2020-04-03 14:38:23 -04:00
52b2137ab0
fix: Don't create a bucket+org on write
2020-04-03 09:11:31 -04:00
db0942c861
feature: Implement the create bucket gRPC request
2020-04-03 09:11:08 -04:00
db950b4aee
fix: Return None instead of Err if org doesn't exist on bucket lookup
2020-04-02 16:45:21 -04:00
1334dafc0a
Revert "fix: Create org if it doesn't exist"
...
This reverts commit a7d33b803c
2020-04-02 16:03:05 -04:00
43a2af9a60
Revert "fix: Insert org if it's not found"
...
This reverts commit cd470908d7
2020-04-02 16:01:58 -04:00
cd470908d7
fix: Insert org if it's not found
2020-04-02 11:53:53 -04:00
32ed04c753
style: Cargo fmt
2020-04-02 11:53:07 -04:00
d097af781d
feature: Implement the get_buckets rpc to get bucket ids in the test
2020-04-02 11:52:39 -04:00
ee0afb5e02
fix: Rename field to not start with an underscore
2020-04-02 11:40:45 -04:00
a810113364
fix: Return ReadPoint.value instead of time twice
2020-04-02 11:18:46 -04:00
df67b9715a
Merge remote-tracking branch 'origin/master' into pd-partiton-store
2020-04-02 11:15:26 -04:00
67e9fdf5e7
refactor: Accept a path for the RocksDB location
2020-04-02 11:06:38 -04:00
40e7a11303
feature: Use a unique directory per RocksDB test run
2020-04-02 11:06:38 -04:00
a7d33b803c
fix: Create org if it doesn't exist
2020-04-02 11:01:54 -04:00
6549bebf2c
refactor: Make MemDB::new more idiomatic
2020-04-02 10:05:37 -04:00
d9cf5c952a
fix: Remove RocksDB code
2020-04-02 09:41:30 -04:00
38b39eb095
fix: Keep ParseError's description private; use Display instead
...
This gives us more flexibility to refactor ParseError in the future.
2020-04-02 09:41:30 -04:00
ac5ecb01dc
fix: Add some more logging for internal server errors
2020-04-02 09:02:52 -04:00
38ec343b33
Refactor partition_store, database, main, and rpc
...
Refactored Partition to be a concrete type enum with three different types of partitions: memory, s3, or remote server. Had to move to this rather than a trait because the methods on partition are async.
Refactored database to use the new partition structure and for now only use a memory based database. Persistence will be reintroduced later with the S3 partition implementation.
Refactored main and rpc to use the new database.
2020-04-01 16:46:07 -04:00
834d5b84a6
Update memdb and partitioned_store
...
Fix bug in memdb with pulling out a range of data.
Simplify code in partition store to address PR feedback.
2020-03-27 16:24:02 -04:00
fa88416dff
style: Cargo fmt
2020-03-27 15:53:37 -04:00
2eb09f29ef
refactor: Prefer match on the result of cmp to if/else if/else chains
2020-03-27 15:53:19 -04:00
1247544deb
refactor: remove unneeded explicit types
2020-03-27 14:28:38 -04:00
90dbac2df5
doc: Moving details about size estimates to the doc comment
2020-03-27 13:56:05 -04:00
2ab44b2603
doc: use doc comments instead of code comments
2020-03-27 13:51:16 -04:00
37e1526933
refactor: DRY up the Point -> ReadPoint conversion code
2020-03-27 13:48:46 -04:00
ace1060388
refactor: Take the advice of the clippy::explicit_iter_loop lint
2020-03-27 10:45:42 -04:00
533deedbdd
fix: Allow if-elses rather than matches
2020-03-27 10:45:42 -04:00
431f4c5633
refactor: Remove unneeded let binding
2020-03-27 09:30:18 -04:00
9411391c4c
fix: Remove unused imports
2020-03-27 09:24:02 -04:00
fed0a5e8b0
fix: Allow dead code temporarily while partition is being refactored
2020-03-27 09:23:51 -04:00
48954424e1
docs: Add some documentation and TODO comments
2020-03-27 09:14:09 -04:00
fa8f04ddb3
style: Cargo fmt
2020-03-27 09:13:48 -04:00
0ded436d71
Merge branch 'pd-partiton-store' of github.com:influxdata/delorean into pd-partiton-store
2020-03-26 16:26:23 -04:00
aa18015c29
Update partition_store and memdb
...
Update ReadMergeStream to use a single vec to store stream state.
Update ReadBatch to always assume values in the batch. Update memdb to not put batches with no values in the result set.
2020-03-26 16:23:58 -04:00
4375afd85c
Update partition_store
...
Updated StreamState to use a single vector to track the streams and the next value.
2020-03-26 13:28:32 -04:00
f223fe408b
fix: Remove redundant clear
2020-03-26 11:19:05 -04:00
97aedbd167
fix: Remove redundant empty checks
2020-03-26 11:15:10 -04:00
1b4b3fe7ad
refactor: Rename a variable to make its purpose clearer
...
I was reading this as "sortED" which is the opposite of what this means.
2020-03-25 10:03:08 -04:00
23e0b097ca
fix: Use sort_by_key on time rather than implementing traits
...
Needed because the PartialOrd/Ord trait implementations would then
disagree with the PartialEq/Eq trait implementations, which isn't really
what's desired.
Change the method name to make it clearer that this isn't using the Ord
trait's sort.
2020-03-25 10:00:44 -04:00
215c1138ea
Update memdb and partitioned_store
...
Run fmt and fix test failure for size check on memdb.
2020-03-22 18:08:04 -04:00
2663d9d756
Update memdb and partition_store
...
Clean up based on PR feedback.
2020-03-22 17:59:25 -04:00
8424d49c64
Add Partition Store
...
This adds the Partition trait along with helper functions and structs to combine the results from multiple partitions.
It also refactors MemDB to have methods that align with the partition trait so that it can be used later by database.rs.
There are obvious places where it is inefficient, but this should be enough to get started with refactoring database.rs to use this rather than rocksdb, series_store, and inverted_index.
2020-03-19 15:30:24 -04:00
4fd0c6f210
feat: Error when parsing lines with duplicate tags
2020-03-11 22:43:09 -04:00
167750669b
refactor: Change the line_parser to return a Result
2020-03-11 22:43:09 -04:00
566f0f4c5c
refactor: Add a line parser Result alias
2020-03-11 22:43:09 -04:00
40614bcf87
refactor: Extract function to convert the parsed line into a value
2020-03-11 22:43:09 -04:00
f5305fd468
feat: Ensure tags are sorted in the line parser
2020-03-11 22:43:07 -04:00
78a53aa391
refactor: Replace the hand-written parser with one built with nom
2020-03-06 10:00:29 -05:00
d5d00071e0
test: Ensure the line protocol parser includes the tag set in the series
2020-03-06 09:51:43 -05:00
db7bb94450
test: Split tests apart to provide better feedback
2020-03-06 09:51:43 -05:00
5972381ad2
Merge pull request #51 from influxdata/remove-failure
...
refactor: Remove unused failure crate
2020-03-02 08:34:10 -05:00
5d3f99da98
refactor: Remove unused failure crate
2020-02-28 16:54:28 -05:00
a1cbe4235f
style: Remove unused import
2020-02-28 09:41:51 -05:00
c2b49a7804
refactor: please clippy
2020-02-27 19:41:55 +00:00
38f23ac07a
refactor: merge master in
2020-02-27 14:27:23 +00:00
54a66c14c8
feature: Implement read filter request
2020-02-26 09:51:02 -05:00
7cac8cc10c
feature: Implement tag values request
2020-02-26 09:51:02 -05:00
2a4d2fe52d
feature: Implement tag keys request
2020-02-26 09:51:02 -05:00
1cd270fdea
feature: Add a trait for common parsing of gRPC inputs
2020-02-26 09:01:57 -05:00
a6cb0cd8a9
feature: Hook up Storage gRPC trait with requests unimplemented
2020-02-26 08:55:25 -05:00
1f59ead622
feature: Parse SeriesFilter keys to create Tags
2020-02-26 08:48:49 -05:00
5bd9015ca1
feature: Connect tag keys and tag value methods through Database
2020-02-26 08:47:11 -05:00
eaea52edc9
feature: Implement a mapping of bucket ID to bucket
2020-02-26 08:45:21 -05:00
12582af08a
feature: Add Send constraint to returned iterators
2020-02-26 08:43:43 -05:00
8b48503d10
refactor: Move RPC code to its own module
2020-02-26 08:38:45 -05:00
898e7458e3
refactor: Use TimestampRange from proto rather than Range
2020-02-26 08:38:38 -05:00
ff9085a096
refactor: No need to accept an explicit lock guard
2020-02-21 14:46:00 -05:00
efefe9eb9a
refactor: unify predicate evaluation between RocksDB and in-memory
2020-02-21 14:45:54 -05:00
2d95595681
refactor: simplify optionally finding the bucket
2020-02-21 11:44:50 -05:00
98c10de980
Merge pull request #45 from influxdata/raw-strings
...
refactor: use raw strings instead of escape sequences
2020-02-21 11:43:32 -05:00
c0b93d7b78
Merge pull request #43 from influxdata/unneeded-string
...
refactor: Remove unneeded String allocation
2020-02-21 11:43:21 -05:00
c85f4b45ed
refactor: use raw strings instead of escape sequences
2020-02-21 09:55:18 -05:00
07b65e32d5
refactor: Remove unneeded String allocation
2020-02-21 09:33:11 -05:00
7620adfc70
refactor: Prefer mutable slices over a mutable Vec
...
We never add anything to the collection, so don't allow for the
possibility.
2020-02-21 09:02:19 -05:00
998537da0f
refactor: Use the newer style of module file naming scheme
...
This cuts down on the number of mod.rs files open in your text editor at
the same time.
2020-02-19 08:43:06 -05:00
cb62590582
style: Warn on unnecessary use of iter and fix all cases
...
This is more idiomatic than calling `iter` explicitly.
2020-02-19 08:37:39 -05:00
c7a7dde51f
fix: Remove redundant clones in tests
...
As suggested by the redundant_clone clippy lint.
This is a little weird because different constructions of `points`
now need clones in different places, but rustc will tell us when
we need to clone and clippy will tell us when we don't.
2020-02-19 08:37:39 -05:00
196b364a00
fix: Assert on an expected error rather than failing if we get Ok
...
The clippy lints assertions_on_constants and single_match were pointing
at this spot, so it's as good a time as any to take care of the TODO!
2020-02-19 08:37:39 -05:00
cdca82fee5
fix: Remove an unnecessary conversion
...
As found by the identity_conversion clippy lint. Clippy was warning
about this on master, but warnings didn't fail the build until a commit
I'm adding in this PR.
2020-02-19 08:37:39 -05:00
41ddab1a54
fix: Change an arbitrary test input so that clippy doesn't think it looks like pi
...
I don't think we actually want to use a precise value for pi, but if we
put 3.14, the approx_constant clippy lint will suggest it.
2020-02-19 08:37:11 -05:00
eabf96a0d7
fix: Test that floats are approximately equal within machine epsilon
...
As found by the float_cmp clippy lint.
There are crates that provide macros for this, but it's small enough
that I think a test helper function in-tree is fine.
2020-02-19 08:37:11 -05:00
c6eda3a8b5
fix: Allow excessive precision for now
...
The way these tests are written, any truncation will be made on both the
values used in the test and the values expected in the test, so this is
probably fine.
But we might not be testing with the exact values we think we're testing
with, so we should audit the values clippy warns about at some point.
2020-02-19 08:37:11 -05:00
7cc871b305
style: Parenthesize potentially confusing expression
...
As suggested by the clippy precedence lint
2020-02-19 08:37:11 -05:00
dd7d6e838f
style: Allow unreadable literals in tests
2020-02-19 08:37:11 -05:00
3effd368d3
feature: Start gRPC server on a different port from the hyper server
2020-02-17 16:37:43 -05:00
9dc3699466
style: cargo fmt
2020-02-17 10:53:57 -05:00
155bfcbd4f
build: Update prost to latest version
2020-02-17 10:48:33 -05:00
23a9a800a6
Merge remote-tracking branch 'origin/master' into hyper
2020-02-17 08:54:19 -05:00
b6184cb778
Merge remote-tracking branch 'origin/master' into er-encoder-bench
2020-02-17 08:47:38 -05:00
bbbbf8ee07
fix: Remove unnecessary allocation
2020-02-17 08:10:53 -05:00
1ad21b3e90
refactor: apply clippy
2020-02-14 18:19:51 +00:00
6fc6fc3329
Merge remote-tracking branch 'origin/master' into er-encoder-bench
2020-02-14 12:47:33 -05:00
92baa3d7e8
refactor: apply clippy
2020-02-14 17:13:20 +00:00
b2cdd299f5
refactor: apply clippy
2020-02-14 17:13:05 +00:00
4dfd4d90ba
fix: Use BytesMut directly rather than through actix
2020-02-14 10:56:37 -05:00
f7b33d47de
fix: Adjust parameter type to avoid double allocation
2020-02-14 10:19:39 -05:00
12fbb23112
fix: Make both query parsing places return bad request on failure
2020-02-14 10:17:48 -05:00
dc7a2ec333
fix: Improve parameter type
2020-02-14 10:02:35 -05:00
a16c49537f
fix: Include limit in size exceeded error
2020-02-14 10:00:35 -05:00
8b1255be9d
refactor: Switch to a hyper server
2020-02-14 09:59:09 -05:00
062bbc5a34
Merge remote-tracking branch 'origin/master' into er-encoder-bench
2020-02-14 09:15:24 -05:00
615e0f6537
style: Apply rustfmt defaults to the entire project
2020-02-14 08:02:11 -05:00
77125bd8e5
improvement: Remove TODO comments that are now done
2020-02-13 10:47:01 -05:00
3577916307
Merge remote-tracking branch 'origin/master' into er-encoder-bench
2020-02-12 13:25:33 -05:00
5942dd5c8a
fix: Remove turbofish that are no longer needed
2020-02-12 09:46:29 -05:00
64223b70a9
refactor: Collapse the read_*_range functions
2020-02-12 09:43:42 -05:00
3399cea18a
refactor: Extract a trait to make read_*_range fns more similar
2020-02-12 09:43:42 -05:00
16c8834fbc
refactor: Collapse read_*_range functions into a generic function
2020-02-12 09:43:40 -05:00
2b642ffaac
refactor: Make read_*_bytes more similar by extracting a trait
2020-02-12 09:42:47 -05:00
af85249ea6
fix: Remove unneeded lifetime annotations
2020-02-12 09:42:47 -05:00
07bb075e93
refactor: Extract storing different types in SeriesData
2020-02-12 09:42:47 -05:00
0b515fe1f9
fix: Switch from Copy to Clone bounds
2020-02-12 09:42:46 -05:00
daa02069db
refactor: Remove unused function
2020-02-12 09:42:42 -05:00
867523c2d9
refactor: Extract the code for storing types' bytes in RocksDB
2020-02-12 09:36:53 -05:00
657059af9f
fix: Do not transmute unknown bytes to enums
...
Fixes #24
2020-02-11 20:47:29 -05:00
461ead862b
Merge pull request #25 from influxdata/reduce-vec-creation
...
perf: Reduce amount of Vecs created in the RocksDB code
2020-02-11 20:45:59 -05:00
26a6d1a272
Merge pull request #26 from influxdata/ok-or-else
...
refactor: Use Option::ok_or_else in RocksDB adapter code
2020-02-11 20:45:50 -05:00
d248c3e7f2
refactor: Use Option::ok_or_else in RocksDB adapter code
...
This helper reduces the boilerplate of creating errors for a missing
value.
2020-02-11 20:41:31 -05:00
b0b8925379
perf: Avoid creating a vector for a subslice
2020-02-11 20:40:21 -05:00
2f63ca7fdb
perf: Remove unneeded Vec clone
2020-02-11 20:40:16 -05:00
959f98f605
perf: Reduce unneeded Vec creation
...
- Integers can be directly converted to arrays of bytes
- We can extend vectors from other slices instead of `Vec`s
2020-02-11 20:40:12 -05:00
be3ed216c3
perf: Avoid taking `Vec` by reference
...
There's no benefit to accepting a reference to a `Vec` over a slice.
Further details available in https://stackoverflow.com/q/40006219/155423
2020-02-11 20:40:06 -05:00
b44f7d8869
perf: Avoid calculating the hashcode twice in the RocksDB adapter
...
Unfortunately, we can't use `Entry::or_insert_with` because we need to
use the key to construct the value.
2020-02-11 20:39:23 -05:00
b5879c8414
style: Enforce Rust 2018 idioms
...
For reference, the [2018 edition guide][guide] talks about some of the
big differences.
The two that are applied here are:
1. `extern crate` is basically not needed at all anymore; you can do
`use cratename` instead. This makes importing things more uniform
between your own crate and other crates.
1. Rust does a reasonable amount of [*lifetime elision*][elision] so
we don't have to type `'a` in as many places. However, one that
ended up tripping up people is when a generic lifetime was part of
a type. The compiler cared about this lifetime, but since it
wasn't visible, people would forget it's there, then try to use it
as if it wasn't constrained by the lifetime.
A good example is the `Chars` iterator. It references the original
`&str` and cannot live longer than the string. With the original
way this was being passed (`&mut Chars`) it was visually evident
that there was *some* lifetime, thanks to seeing the `&`, but it
wasn't obvious that there's *another* lifetime — the string.
With the addition of the *anonymous lifetime* (`'_`), it's now
encouraged to use that when a type has a lifetime parameter that
isn't relevant to prevent confusing mistakes that lead to compiler
errors.
There are probably a few more things enabled by the lint as well. I
forget the exact reason that these are not yet enabled by default,
though.
[guide]: https://doc.rust-lang.org/edition-guide/rust-2018/index.html
[elision]: https://doc.rust-lang.org/book/ch10-03-lifetime-syntax.html#lifetime-elision
2020-02-11 08:08:12 -05:00
7a9f69c921
refactor: address PR feedback
2020-02-07 22:28:11 +00:00
5327efd926
test: add timestamp benchmarks
2020-02-07 13:26:59 +00:00
4185307d78
test: add float encoder/decoder bencmarks
...
This commit adds benchmarks for the float encoder and decoder. The
following scenarios are benchmarked:
- sequential values;
- random values;
- real CPU values (from Telegraf output).
Each scenario is benchmarked with a variety of block sizes.
2020-01-21 15:01:35 +00:00
f1329a8a08
Remove unused code
2020-01-13 10:34:41 -05:00
bec6b3cf9c
Add MemDB and test framework
...
This commit adds a test framework for the InvertedIndex and SeriesStore traits. Structs that implement these traits can call into their tests to ensure they work.
This commit also adds an in memory database that keeps time series data in a ring buffer per series and an inverted index in memory using a combination of HashMap and BTreeMap.
2020-01-13 10:31:47 -05:00
80493ba517
Refactor database for arbitrary backends
...
This commit pulls the database behavior into three traits: ConfigStore, InvertedIndex, and SeriesStore. The ConfigStore holds the Bucket definitions, the InvertedIndex holds the index of measurement names, tags, and fields, and the SeriesStore holds the raw time series data indexed only by bucket id, series id, and time.
This is the initital work to enable memory and S3 backed databases. It is also the preliminary work to break data out into shards.
2020-01-13 10:31:47 -05:00
62a0f26066
refactor: src doesn't need to be mutable
2020-01-13 14:59:51 +00:00
c1492994e0
refactor: tidy up
2020-01-12 15:37:30 +00:00
2a1ddf5669
feat: add float encoder
2020-01-12 15:37:26 +00:00
80ff911259
feat: automatically create db dir and test dir
2020-01-06 15:55:22 +00:00
b06d1005a8
refactor: rustfmt
2020-01-06 13:49:39 +00:00
af9e1cae32
Fix race condition on new series in RocksDB
...
There was a race condition when inserting new series into RocksDB that would cause series being inserted by two separate threads to cause an error. The thread that inserts is fine, but the one after would see that it inserted, but not read in the ID so it can later be used to write the points.
2020-01-05 19:34:05 -05:00
04bd6b8a12
Update line protocol parser
...
The earlier version of this line protocol parser incorrectly used a space as a delimiter between fields. This updates it to use a comma as it is in InfluxDB 1.x and 2.x.
2020-01-05 19:32:35 -05:00
ce69c5a9ce
Update read API to return float series
...
Updates to read API in main.rs to return values for float series. I'm not terribly happy with the way I had to do this, but I was struggling a bit with the type system gymnastics. I assume I'll have to revisit this anyway when I add support for other storage backends.
2020-01-05 19:31:42 -05:00
b784563792
Add support for float64 to rocksdb
...
Adds support for for f64 time series in RocksDB. Series data types are now stored in the index under the id to key mapping, which is now id to type and key.
This doesn't enforce the same data type for values being written into a series, which will come later. Also later will be adding support for float64 series in the read API.
2020-01-05 18:38:45 -05:00
8754911b5a
Add support for float64 to line protocol parser
...
Adds support for f64 to the line protocol parser. Also updates the return value of parse to return a Vec of mixed type points that can be later written into the database.
The PointType struct is only for use in this context. In the context of querying or working with time series for compaction, we'll want vectors of actual typed points of the same kind so we don't have to do inefficient enum matches.
2020-01-05 16:28:12 -05:00
c7a862dca0
Fix compile warnings from the Rust linter
...
This commit fixes all the linter warnings. However, there are a number of spots, particularly in the encoders where I added `#[allow(dead_code)]` to get past them. We should go through and fix those up at some point, I'll log an issue to track.
2020-01-05 13:44:03 -05:00
1a851f8d0b
Add basic read endpoint
...
This commit adds a basic read endpoint to pull data out of the database. In order to provide the basic functionality a few things were added:
* Time package with limited support for parsing Flux style durations
* API endpoint at /api/v2/read with query paramters of org_id, bucket_name, predicate, start, and stop
The start and stop query parameters only support relative durations.
The predicate parameter supports what is possible in the parse_predicate method and in the RocksDB implementation (only == comparisons on tags and AND or OR)
2020-01-04 19:07:54 -05:00
fe9cb87c3d
Update rocks db series filter
...
Add the series key to the SeriesFilter struct that is returned when searching the index.
2020-01-03 17:35:18 -05:00
4265e7b11b
Update write API endpoint
...
Upates the actix-web and actix-rt versions to 2.0 and 1.0 respectively. Wires up the write endpoint to create buckets if they don't exist.
2020-01-03 17:35:18 -05:00
4892a87898
Implement read range on the database
...
This commit adds iterators for iterating over series and batches of points for a read range request. The exact signature/structure of the iterators are likely to change when this is generalized for other data types and other storage backends (S3 & memory).
2020-01-03 17:35:18 -05:00
c76ce39da5
Update rocksdb.rs write_points
...
This commit updates the write_points method to use the bucket id and series id in the key for a stored point value.
It also updates the Database methods to be immutable borrows moving any mutable concerns into interior structures so it can be easily called from many threads.
2020-01-03 17:35:18 -05:00
3c8f93a9a7
Implement OR in predicate
...
Adds support for logical OR operator in predicates against the index.
2019-12-24 15:25:43 -05:00
f3807456c9
Implement AND in predicate
...
Adds support for logical AND operator in predicates against the index.
2019-12-24 15:21:04 -05:00
6cd4c5b583
Add basic tag key/value index
...
This commit brings in a Roaring Bitmap implementation to keep postings lists of tag key/value pairs to the set of series ids that have those pairs. The croaring implementation was used becasue the Treemap was required for u64 support for series ids and it was serializable (unlike the other pure Rust roaring implementation).
This doesn't shard the postings lists based on size. It also doesn't implement the time/index levels.
The predicate matching currently only works for a simple key = "value" match.
2019-12-24 13:44:30 -05:00
f77b0a3842
Move storage module into directory
...
Moved the storage module into its own directory. Split into the rocksdb portion of the code and the predicate parsing.
2019-12-23 11:49:58 -05:00
71fe0aa71c
Update storage with predicate parsing
...
Adds a basic predicate parser to make testing the index easier
2019-12-23 11:36:12 -05:00
54ef130cea
Wire up get tag values in storage
...
This adds basic support for getting tag values in storage. Still needs to add predicate and time range support.
2019-12-20 13:46:41 -05:00
7effec0f48
Add shell of index functions and get tag keys
...
Added the shell of index functions to return series IDs that match a predicate, tag keys with a predicate, and tag values with a predicate.
2019-12-20 13:07:03 -05:00
7f2f4eaceb
Update mod.rs
...
Add parser for key/value pairs to be indexed. Measurement and field are represented as _m and _f respectively.
2019-12-20 13:05:59 -05:00
5d80d5e100
feat(storage): Add series to ID index
...
This commit is the beginning of the RocksDB based index for series and their tag metadata.
I started to stub out different index levels but stopped short of implementing them.
There are a number of spots where I'm unwrapping return values that we may want to revisit later. For now I want to have the program panic if those things pop up.
2019-12-19 15:58:00 -05:00
1a10243b46
Update integer.rs
...
Fix build error in test
2019-12-19 15:50:22 -05:00
617c2960a8
feat(storage): Implement bucket definitiions and persistence
...
This updates to build system to use Prost to build the protobuf objects.
It adds tests for creating, storing and loading bucket definitions.
The tests use an actual on disk RocksDB implementation to ensure that its tested all the way to persistence.
2019-12-17 17:01:41 -05:00
100d192538
Update main.rs
...
Fix build warnings for unused imports.
2019-12-17 17:01:41 -05:00
3856af8842
Update timestamp.rs
...
Update test to fix build error.
2019-12-17 17:01:41 -05:00
64efa35dea
refactor: tidy up comments
2019-12-17 12:10:27 +00:00
cf64234c2d
feat: integer encoder
2019-12-16 17:06:04 +00:00
36d3138a3d
refactor: timestamp encoder
2019-12-13 16:43:29 +00:00
d9b966579e
refactor: move timestamp-specific RLE
2019-12-13 11:12:56 +00:00
9cadb1bb52
Add server skeleton with Actix and RocksDB
2019-12-12 10:15:16 -05:00
0627ea0e5b
refactor: organise encoders
2019-12-12 13:47:14 +00:00
8662982233
feat(encoders): add timestamp block encoder
2019-12-11 19:21:56 +00:00
bff8704a0b
test: more simple8b tests
2019-12-11 19:21:38 +00:00
3bbc86a8a0
fix: fix RLE bug
2019-12-11 19:21:15 +00:00
e9db04292c
refactor: change simple8b API to use binary encoding
2019-12-10 20:05:41 +00:00
fb83e9c7fa
feat(encoding): add RLE encoder/decoder
2019-12-09 18:04:01 +00:00
4009e67bf3
refactor: change encoder/decoder API
2019-12-08 21:00:45 +00:00
55d711599e
tests: add tests for simple8b
2019-12-04 13:14:37 +00:00
824044eac1
refactor: move simple8b encoder/decode
2019-12-04 13:14:20 +00:00
46bbe4d317
feat(encoders): add simple8b encoder/decoder
2019-12-03 13:46:36 +00:00
df355da693
refactor: comment out imports
2019-12-03 13:46:09 +00:00
7a122b23cf
Add shell for line protocol parser and encoders
2019-11-22 17:06:34 -05:00
b9b5a815b7
Initial commit with some notes and proto
2019-11-22 16:59:04 -05:00
Carol (Nichols || Goulding)
Edd Robinson
Carol (Nichols || Goulding)
Jake Goulding
Paul Dix