97a5eb7e19
Merge pull request #197 from influxdata/alamb/log-requests
...
feat: Log gRPC calls using trace crate, allow custom log levels
2020-06-30 10:47:11 -04:00
283d6691c6
feat: enable rpc debug tracing, tweaked logging levels, respect RUST_FMT env var
2020-06-29 09:59:22 -04:00
ad1e3d04bb
feat: Add a local filesystem implementation of the object store
2020-06-29 08:48:48 -04:00
d15256e0e7
refactor: address PR feedback
2020-06-26 12:08:42 +01:00
9d889828c3
fix: ensure all rows are emitted for each column
2020-06-26 11:50:37 +01:00
68ce351a3a
refactor: remove direct parquet dependency from delorean_ingest
2020-06-23 16:58:31 -04:00
d7dbf061cb
feat: Implement String encoding/decoding
...
Fixes #148 .
2020-06-22 15:15:34 -04:00
106bd69b5a
feat: support converting from TSM->Parquet
2020-06-22 18:56:17 +01:00
85e0b4ec16
refactor: hoist tsm reader into own crate
2020-06-22 18:56:17 +01:00
ac7bb6bf68
refactor: make Packer generic
2020-06-22 11:24:29 +01:00
bfb0213ac3
feat: Update Rusoto to allow streaming data on uploads
2020-06-19 09:18:44 -04:00
8185c80c03
fix: fix logical merge conflict ( #169 )
2020-06-18 18:51:25 -04:00
ae37548980
feat: Add support for parsing string values in line protocol parser ( #155 )
...
* feat: add debug logging on parser error
* feat: Add support for parsing string values in line protocol parser
* fix: Fix comment
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-06-18 12:44:17 -04:00
cf248f2143
feat: upgrade to latest arrow / byteorder ( #154 )
2020-06-17 12:50:23 -04:00
d83c410a5c
feat: Update to the released version of cloud-storage
...
My submitted API improvements got merged in!
2020-06-10 17:23:52 -04:00
d3283b1096
feat: Object storage in S3 and GCS
2020-06-10 17:23:52 -04:00
faf3f534ac
refactor: move all dstool code into delorean binary ( #131 )
...
* refactor: move all dstool code into delorean binary
* fix: Move code/mods to make it compile and run
* fix: warn if db dir does not exist
* refactor: Match argument subcommands w/ more idomatic rust
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
fix: restore hyper logging
fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: update expected code
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-06-10 16:04:46 -04:00
0415b233ec
refactor: Instantiate the table writer on demand ( #128 )
...
* refactor: instantiate ParquetWriter on demand, prep for multi measurements
* fix: doc test
* fix: update names
2020-06-09 16:11:42 -04:00
986e12d62a
refactor: Rename crate line_protocol_schema --> delorean_table_schema ( #129 )
...
* refactor: Rename crate line_protocol_schema --> delorean_table_schema
* fix: fmt
2020-06-09 11:56:16 -04:00
f1a3058b24
feat: Add file / metadata inspection + dumping with dstool ( #112 )
...
* feat: Add file / metadata inspection + dumping
* fix: apply some PR review comments
* fix: apply suggestions from code review
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
* feat: Add tests, rearrange code into modules, add gzip aware interface
* fix: fix comment and test
* fix: test output and fmt
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2020-06-09 10:10:55 -04:00
8475b6d183
feat: Add parquet writer, hook up conversion in dstool ( #124 )
...
* feat: Add parquet writer, hook up conversion in dstool
* fix: use bigger executor for test
* fix: less cloning
* fix: make unsupported messages less pejorative
* fix: fmt
* fix: Rename writer and do not require std::File, add example
* fix: clippy and fmt
* fix: remove unnecessary module in end to end tests
* fix: remove strange use of tempfile
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: cleanup use
* fix: Use more specific error messages
* fix: comment tweak
* fix: touchup temp path creation
* fix: clippy!
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-06-08 16:25:24 -04:00
ca9f9d4cae
feat: Add column packing code ( #114 )
...
* feat: Add column packing code
* fix: remove dependency on assert_approx_equal in favor of delorean_test_helpers
* fix: Cleanups from pr comments
* fix: Apply suggestions from code review
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: more cleanup per code review
* fix: pr comments
* fix: remove explict string creation from caller
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-06-06 06:04:41 -04:00
2200def8ea
feat: Use rust nightly ( #123 )
2020-06-05 17:45:44 -04:00
68fb580b43
style: Re-enable the elided lifetimes lint and move generated types to their own crate ( #119 )
...
* refactor: rename the module containing generated types
The nested `delorean` was confusing anyway, and this will make more
sense when we extract a new crate.
* refactor: Move the generated types to their own crate
This allows us to have more lax warnings in that crate alone, keeping
the main crate more strict.
* style: Re-enable elided lifetimes lint in the main crate
2020-06-05 16:22:27 -04:00
887ffd5977
refactor: remove lifetime to make index re-usable
2020-06-04 14:36:43 +01:00
e3db077121
feat: add API for series key information
2020-06-04 14:36:43 +01:00
413738a264
feat: support org and bucket ID in entries
2020-06-04 14:36:43 +01:00
234b2f5752
feat: Line Protocol Schema extractor ( #108 )
...
* feat: schema inference from iterator of parsed lines
* fix: Clean up error handing even more
* fix: fmt
* fix: make a sacrifice to the clippy gods
2020-06-03 18:29:57 -04:00
5d2c5de39d
feat: Structs to represent line protocol schema ( #103 )
...
* feat: Structs to represent line protocol schema
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2020-06-03 08:39:35 -04:00
18b05ce9ef
fix: move test of dstool to its delorean_storage_tool package ( #107 )
2020-06-02 16:10:30 -04:00
1a2efdfd71
feat: Add dstool command line tool ( #102 )
...
* feat: Add dstool command line tool
* clippy
* Update delorean_storage_tool/src/main.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* Update delorean_storage_tool/src/main.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* Add in tests + PR comments
* fmt
* build first then run tests
* actually build before test
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-06-02 07:33:43 -04:00
924f20fd50
build: update semver-compatible versions
2020-05-29 13:40:44 -04:00
d4af54c3de
refactor: extract the line protocol parser to a separate crate
...
This will facilitate reusing the parser for other tasks.
2020-05-26 13:22:34 -04:00
f2823ccecd
feat: WAL file rollover based on size of file
2020-05-18 14:08:24 -04:00
e25a4e1e83
feat: Integrate the WAL with delorean
2020-05-11 15:38:47 -04:00
4dd7a8cea8
feat: Introduce a WAL tailored for delorean
2020-05-11 15:38:44 -04:00
bff4d2d6d9
refactor: Move temporary directory creation to test helpers
2020-05-11 15:26:00 -04:00
22136d5431
build: update semver-compatible versions
2020-05-11 15:25:54 -04:00
e369ada35a
refactor: extract a crate with our custom assertions
...
There's probably an existing crate that we should use directly, but I
haven't found an exact match yet.
2020-05-01 13:04:24 -04:00
7f9eaf51d5
Merge pull request #74 from influxdata/cn-generate-points
2020-04-24 08:08:32 -04:00
f1d5d50b92
Merge pull request #68 from influxdata/er-block-writer
...
feat: add Block Type
2020-04-23 22:48:38 +01:00
fa69101945
refactor: Move the point utilities into a workspace crate
2020-04-23 11:26:37 -04:00
6791b9598c
feat: Utility to take line protocol and make write requests
2020-04-23 11:11:51 -04:00
93231c64e0
perf: Use a SmallVec for escaped strings and sets of tags and values
...
This increases the performance from 56.531 MiB/s to 58.194 MiB/s.
2020-04-08 14:41:42 -04:00
9e20743b2c
feat: add Block Type
...
This commit adds a new Block type, which is used to keep track of values
associated with individual block, and then serialise them.
2020-04-08 13:37:48 +01:00
974a142cc8
build: update semver-compatible versions
2020-04-05 16:35:27 -04:00
8629072508
build: Upgrade tonic to 0.2
2020-04-05 16:35:00 -04:00
4a28abd4de
build: Upgrade assert_cmd to 1.0
...
This requires that we opt into the serde `derive` feature that is no
longer implicitly added from upstream.
2020-04-05 16:33:37 -04:00
48d5d16a1b
build: update semver-compatible dependency versions
2020-04-05 16:33:24 -04:00
df67b9715a
Merge remote-tracking branch 'origin/master' into pd-partiton-store
2020-04-02 11:15:26 -04:00
97d11633b8
feature: Use a unique directory per end-to-end test run
2020-04-02 11:06:36 -04:00
d9cf5c952a
fix: Remove RocksDB code
2020-04-02 09:41:30 -04:00
4fd0c6f210
feat: Error when parsing lines with duplicate tags
2020-03-11 22:43:09 -04:00
78a53aa391
refactor: Replace the hand-written parser with one built with nom
2020-03-06 10:00:29 -05:00
5d3f99da98
refactor: Remove unused failure crate
2020-02-28 16:54:28 -05:00
17051717e2
chore: remove dependency:
2020-02-28 12:55:28 +00:00
38f23ac07a
refactor: merge master in
2020-02-27 14:27:23 +00:00
c41652e45b
feature: Add the storage gRPC proto definitions
2020-02-24 08:26:28 -05:00
3438edd18b
feature: Switch from prost to tonic
2020-02-17 16:37:43 -05:00
68970f8ff3
build: Update bytes to latest version
2020-02-17 10:48:33 -05:00
155bfcbd4f
build: Update prost to latest version
2020-02-17 10:48:33 -05:00
eb3113b820
build: Update dotenv to latest version
2020-02-17 10:48:32 -05:00
31df104996
build: Update env_logger to latest version
2020-02-17 10:48:32 -05:00
e3bfa0f835
build: Update byteorder to latest version
2020-02-17 10:48:32 -05:00
04a7f716e2
build: Update dependencies to latest semver-compatible versions
2020-02-17 10:48:32 -05:00
8b1255be9d
refactor: Switch to a hyper server
2020-02-14 09:59:09 -05:00
062bbc5a34
Merge remote-tracking branch 'origin/master' into er-encoder-bench
2020-02-14 09:15:24 -05:00
9cce1e4882
test: Add an end-to-end test
...
This test:
- Runs the server in a thread
- Writes some data
- Reads some data
- Shuts down the server
2020-02-13 10:40:03 -05:00
4185307d78
test: add float encoder/decoder bencmarks
...
This commit adds benchmarks for the float encoder and decoder. The
following scenarios are benchmarked:
- sequential values;
- random values;
- real CPU values (from Telegraf output).
Each scenario is benchmarked with a variety of block sizes.
2020-01-21 15:01:35 +00:00
75d378974e
fix: cargo lock file
2020-01-07 16:05:46 +00:00
80ff911259
feat: automatically create db dir and test dir
2020-01-06 15:55:22 +00:00
1a851f8d0b
Add basic read endpoint
...
This commit adds a basic read endpoint to pull data out of the database. In order to provide the basic functionality a few things were added:
* Time package with limited support for parsing Flux style durations
* API endpoint at /api/v2/read with query paramters of org_id, bucket_name, predicate, start, and stop
The start and stop query parameters only support relative durations.
The predicate parameter supports what is possible in the parse_predicate method and in the RocksDB implementation (only == comparisons on tags and AND or OR)
2020-01-04 19:07:54 -05:00
4265e7b11b
Update write API endpoint
...
Upates the actix-web and actix-rt versions to 2.0 and 1.0 respectively. Wires up the write endpoint to create buckets if they don't exist.
2020-01-03 17:35:18 -05:00
6cd4c5b583
Add basic tag key/value index
...
This commit brings in a Roaring Bitmap implementation to keep postings lists of tag key/value pairs to the set of series ids that have those pairs. The croaring implementation was used becasue the Treemap was required for u64 support for series ids and it was serializable (unlike the other pure Rust roaring implementation).
This doesn't shard the postings lists based on size. It also doesn't implement the time/index levels.
The predicate matching currently only works for a simple key = "value" match.
2019-12-24 13:44:30 -05:00
617c2960a8
feat(storage): Implement bucket definitiions and persistence
...
This updates to build system to use Prost to build the protobuf objects.
It adds tests for creating, storing and loading bucket definitions.
The tests use an actual on disk RocksDB implementation to ensure that its tested all the way to persistence.
2019-12-17 17:01:41 -05:00
9cadb1bb52
Add server skeleton with Actix and RocksDB
2019-12-12 10:15:16 -05:00
fb83e9c7fa
feat(encoding): add RLE encoder/decoder
2019-12-09 18:04:01 +00:00
55d711599e
tests: add tests for simple8b
2019-12-04 13:14:37 +00:00
b9b5a815b7
Initial commit with some notes and proto
2019-11-22 16:59:04 -05:00
Andrew Lamb
alamb
Jake Goulding
Edd Robinson
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
Paul Dix