ef501871bb
feat: remove partition_store ( #387 )
2020-10-26 14:39:38 -04:00
880958d9c7
feat: switch end-to-end test to use write_buffer implementation rather than partitioned store ( #386 )
...
* feat: switch end-to-end test to use write_buffer implementation rather than partitioned store
* fix: Apply PR suggestions
2020-10-26 13:42:38 -04:00
6223cbdc99
feat: Add an authorization token to all client requests
...
Connects to influxdata/fusion#59 . Delorean currently ignores this
header.
Also add an example of using this to connect to an InfluxDB 2 instance;
I tested this out with a locally running Influx DB 2 and I was able to
write points!
2020-09-02 10:33:45 -04:00
5f49543fdf
feat: Support storage of points with String and Boolean fields
2020-08-14 10:57:20 -04:00
d59702ec79
feat: Make the create bucket HTTP API match the Influx 2.0 API
...
The `/api/v2/create_bucket` API was delorean-specific for testing
purposes. This change makes it match the [Influx 2.0 API][influx] and
adds a method to the client for creating buckets.
The client will always send an empty array of `retentionRules` because
that is a required parameter for the Influx API. Delorean always ignores
`retentionRules`. The `description` and `rp` parameters are optional and
are never sent.
[influx]: https://v2.docs.influxdata.com/v2.0/api/#operation/PostBuckets
I believe the gRPC create bucket is also delorean-specific and perhaps
not needed, but I'm leaving it in for now with a note.
2020-08-12 10:08:32 -04:00
e15efb71cf
test: Have the end-to-end test use DataPoint from the influx client
...
This is revealing a bug in the client
2020-07-27 14:03:06 -04:00
876796a59d
refactor: Remove duplication by having write call write_line_protocol
2020-07-22 17:03:29 -04:00
879b981897
refactor: Rename write methods to clarify their uses
2020-07-22 17:03:29 -04:00
143c350ecb
Merge pull request #250 from influxdata/alamb/feat-multi-col-stats
...
feat: Update stats command to handle directories of files
2020-07-20 16:48:31 -04:00
bfd0f30152
test: Avoid an allocation and have a consistent API
...
This is temporary until we have a better point-building method anyway.
2020-07-17 17:14:20 -04:00
ca1bd79902
feat: Update stats command to handle directories of files
2020-07-17 16:47:11 -04:00
668aefae9b
feat: Implement a rudimentary write API in the influx client
2020-07-17 10:28:19 -04:00
b1ce3958e5
fix: Define a fully-featured error for the Id type
2020-07-16 20:30:54 -04:00
5afc3a1e3b
fix: Ignore anything in the WAL dir that doesn't parse as an Org ID instead of crashing
2020-07-16 11:28:57 -04:00
6ee72e5733
Merge pull request #239 from influxdata/alamb/less-chatty
...
fix: Make default output of delorean less chatty
2020-07-10 06:59:54 -04:00
d1e334f3fe
fix: Make default output of delorean less chatty
2020-07-08 18:02:48 -04:00
50ef521e6c
feat: add support for converting multiple TSM files
...
This commit extends the ingest crate to support converting multiple TSM
files to a single Parquet file by merging identical measurements across
the TSM files.
This does not yet support merging blocks that overlap.
2020-07-08 22:57:15 +01:00
0deb608207
fix: Use correct route to ping
2020-07-03 15:35:04 -04:00
5f045fd9da
fix: more cleanup
2020-07-03 07:01:22 -04:00
5fad7db0ac
feat: Add better error handling and logging for http_routes
2020-07-01 16:08:49 -04:00
b5a91fae36
Merge pull request #207 from influxdata/alamb/faster-test-harness
...
fix: speed up end-to-end test harness so it polls rather than waits; Add PING route
2020-07-01 12:46:05 -04:00
6021ddd222
fix: more PR feedback
2020-07-01 12:41:59 -04:00
1f61fd820b
fix: style cleanup
2020-07-01 12:31:42 -04:00
d16aec7e80
fix: Update tests/end-to-end.rs
...
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2020-07-01 12:29:17 -04:00
d7ddcfa92f
fix: fixup errors introduced during PR comments
2020-07-01 08:44:24 -04:00
f608bf4f0f
fix: Update tests/end-to-end.rs
...
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2020-07-01 08:41:21 -04:00
6b71fcacde
fix: Apply suggestions from code review
...
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2020-07-01 08:39:56 -04:00
8be278aadd
fix: speed up end-to-end test harness so it polls rather than waits
2020-06-30 12:21:25 -04:00
dc8d655486
feat: Implement CapabilitiesResponse
2020-06-30 11:17:45 -04:00
c6f2508abe
fix: Use clap's possible_values method
...
This gets us built-in help text and error messages, and does less work
before failing because of an unsupported value.
Before this change, the help text was:
```
OPTIONS:
--compression-level <compression_level>
Compression level: max or compatibility (default). [default: compatibility]
```
After this change, the help text is:
```
OPTIONS:
--compression-level <compression_level>
How much to compress the output data. 'max' compresses the most; 'compatibility' compresses in a manner more
likely to be readable by other tools. [default: compatibility] [possible values: max, compatibility]
```
Before this change, if you supplied an unsupported value, the error was:
```
[2020-06-29T14:47:42Z INFO delorean::commands::convert] convert starting
[2020-06-29T14:47:42Z INFO delorean::commands::convert] Preparing to convert 591 bytes from tests/fixtures/lineproto/temperature.lp
Conversion failed: Error creating a parquet table writer Unknown compression level 'foo'. Valid options 'max' or 'compatibility'
```
After this change, the error is:
```
error: 'foo' isn't a valid value for '--compression-level <compression_level>'
[possible values: compatibility, max]
```
2020-06-29 10:53:01 -04:00
c35b0e835c
fix: clippy
2020-06-29 10:25:43 -04:00
e8a0233b09
feat: add configurable compression level
2020-06-29 10:25:43 -04:00
511d2b6436
refactor: move benchmark test data to tests/fixtures
2020-06-26 17:41:37 -04:00
672afa1195
refactor: Merge unknown and missing extension errors
2020-06-26 16:07:35 -04:00
aca63b68b2
refactor: Use more unique errors for the commands
2020-06-26 16:01:59 -04:00
bca4d00716
refactor: use SNAFU more idiomatically in command modules
2020-06-26 13:26:51 -04:00
99268f5260
test: add coverage for converting tsm file
2020-06-26 11:50:37 +01:00
322a491b9d
perf: Improve line protocol --> parquet conversion performance by ~20% ( #177 )
...
* feat: benchmark for lp->parquet performance
* feat: improve parser performance by storing contiguous EscapedStr
* fix: remove all string copies during LP-Parquet conversion
* refactor: Implement from_str as From<&str> only
* refactor: implement Deref instead of as_str
* refactor: Remove ends_with because Deref now makes it work
* refactor: Eq can be derived
* refactor: Remove unused From implementation
* refactor: Replace single-character strings with chars as requested by clippy
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@integer32.com>
2020-06-23 05:42:19 -04:00
e507183fbd
refactor: cleanup + clippy
2020-06-22 18:56:17 +01:00
506a7f19d5
feat: add stats command for computing storage statistics ( #160 )
...
* feat: add stats command for computing storage statistics
* fix: Make function name less redundant
* fix: Apply suggestions from code review
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
* fix: remove changes to Cargo.lock
* fix: Cleanup to use `impl Into<String>` rather than String::from
* fix: fmt
* fix: less mut builder
* fix: fmt
* fix: clippy/fmt
* fix: fixup test
* fix: remove dstool reference
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2020-06-22 07:33:52 -04:00
78c2fb99ac
refactor: Remove all references to 'dstool'
...
Since it got merged into the main `delorean` binary.
2020-06-19 11:08:59 -04:00
a106e55fa6
feat: Add parquet metadata dumping ( #159 )
...
* feat: Add parquet metadata dumping
* fix: Update delorean_parquet/src/error.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-06-18 18:34:49 -04:00
91de50a3a7
refactor: Refactor convert command code to have a place for TSM ( #164 )
2020-06-18 09:57:54 -04:00
abb3338483
test: add an end to end test for writing multiple parquet files ( #145 )
...
* test: add an end to end test for writing multiple parquet files
* fix: whitespace ocd
2020-06-15 07:12:16 -04:00
d9278263a7
feat: write multiple measurements to multiple parquet files ( #138 )
...
* feat: write to a directory of parquet files
* feat: change LineProtocolConverter to push style, move sampling there
* feat: full push mode, write to multiple measurements
* fix: clarify comments on finalize
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: clippy/fmt
* fix: remove whitespace
* fix: Apply suggestions from code review
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
* fix: fmt
* fix: make it compile again
* fix: fixup comments
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
* fix: remove unecessary debug implementation
* fix: cleaner comment
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
* fix: clearer iterator name
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
* fix: Apply suggestions from code review
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
* fix: clean
* fix: make it compile
* fix: type fix
* fix: whitespace
* fix: more review comments
* fix: more review comments
* fix: code review comments + fmt
* fix: clippy
* fix: Use EscapedStr directly for performance
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2020-06-12 17:19:35 -04:00
faf3f534ac
refactor: move all dstool code into delorean binary ( #131 )
...
* refactor: move all dstool code into delorean binary
* fix: Move code/mods to make it compile and run
* fix: warn if db dir does not exist
* refactor: Match argument subcommands w/ more idomatic rust
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
fix: restore hyper logging
fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: update expected code
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-06-10 16:04:46 -04:00
81810c2faa
fix: rename measurement.txt to measurement.lp for consistency ( #122 )
...
* fix: rename measurement.txt to measurement.lp for consistency
* fix: rename the file
2020-06-05 17:28:06 -04:00
e0c38d0976
chore: Add test to check for tsm reading errors, update doc example ( #117 )
...
* chore: Add a test that decodes the entire tsm index
* fix: update test and change example to not use hard coded len
* fix: comment cleanup
* fix: clippy
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
* fix: fmt/clippy after code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2020-06-05 16:22:56 -04:00
68fb580b43
style: Re-enable the elided lifetimes lint and move generated types to their own crate ( #119 )
...
* refactor: rename the module containing generated types
The nested `delorean` was confusing anyway, and this will make more
sense when we extract a new crate.
* refactor: Move the generated types to their own crate
This allows us to have more lax warnings in that crate alone, keeping
the main crate more strict.
* style: Re-enable elided lifetimes lint in the main crate
2020-06-05 16:22:27 -04:00
413738a264
feat: support org and bucket ID in entries
2020-06-04 14:36:43 +01:00
18b05ce9ef
fix: move test of dstool to its delorean_storage_tool package ( #107 )
2020-06-02 16:10:30 -04:00
fbc74348c6
fix: consistent capitalization and data location ( #105 )
2020-06-02 08:00:13 -04:00
1a2efdfd71
feat: Add dstool command line tool ( #102 )
...
* feat: Add dstool command line tool
* clippy
* Update delorean_storage_tool/src/main.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* Update delorean_storage_tool/src/main.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* Add in tests + PR comments
* fmt
* build first then run tests
* actually build before test
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-06-02 07:33:43 -04:00
0a02afe016
test: Wait for test server to stop before restarting ( #92 )
2020-05-26 17:34:54 -04:00
e25a4e1e83
feat: Integrate the WAL with delorean
2020-05-11 15:38:47 -04:00
bff4d2d6d9
refactor: Move temporary directory creation to test helpers
2020-05-11 15:26:00 -04:00
7d568704cd
feat: Implement MeasurementFields gRPC request
2020-05-04 15:52:01 -04:00
e615607f4a
feat: Implement MeasurementTagValues gRPC request
2020-05-04 15:52:01 -04:00
1ab9230654
feat: Implement measurement tag keys request
2020-05-04 15:52:00 -04:00
e6e911ce3c
feat: Implement the Measurement Names gRPC request
2020-05-04 15:52:00 -04:00
e0eb895a14
test: Add more measurements to the e2e test data
2020-05-04 15:52:00 -04:00
a6682467a7
fix: Use approximate floating point comparisons for slices as well
2020-05-01 13:05:00 -04:00
6dd7c0a3e8
fix: Treat gRPC bucket_id as bucket name because MemDB assigns bucket ID
2020-04-13 16:40:38 -04:00
429d0e1851
fix: Change name of the proto package to match proto used by Go
...
So that `storectl query` requests will work with delorean
2020-04-13 16:40:38 -04:00
a85f0edfc6
feature: Use the new ID type everywhere
2020-04-13 16:40:38 -04:00
0314bc6ba2
fix: Change the default gRPC port to match storectl
...
8082 appears to be the default here:
b304e751e1/cmd/storectl/query/query.go (L56)
2020-04-13 16:03:23 -04:00
56a2b323be
feature: Implement ReadGroup
2020-04-06 10:23:43 -04:00
747585e0b5
test: Make more interesting test data
2020-04-06 09:08:34 -04:00
39dad5de31
fix: Always return long form of _f and _m from gRPC calls
2020-04-03 17:20:00 -04:00
a801f19b70
refactor: Extract a method to get sorted tags from ReadBatches
2020-04-03 17:18:05 -04:00
7356bc33ed
refactor: Rearrange construction of data
...
This was a temporary way of writing it to figure out the types, but I
find it easier to see the hierarchy.
2020-04-03 14:38:23 -04:00
7bdea751d2
test: Explicitly create the bucket in the end-to-end test
2020-04-03 09:18:15 -04:00
38e660818e
style: Cargo fmt
2020-04-02 16:39:57 -04:00
c83805854d
Fix end-to-end test for new rpc.rs behavior
2020-04-02 15:47:43 -04:00
d097af781d
feature: Implement the get_buckets rpc to get bucket ids in the test
2020-04-02 11:52:39 -04:00
f77d28367d
feature: Respect TEST_DELOREAN_DB_DIR in the end-to-end test
2020-04-02 11:06:38 -04:00
97d11633b8
feature: Use a unique directory per end-to-end test run
2020-04-02 11:06:36 -04:00
d66cc552a6
refactor: Use an Error/Result type alias
2020-04-02 11:04:20 -04:00
54a66c14c8
feature: Implement read filter request
2020-02-26 09:51:02 -05:00
7cac8cc10c
feature: Implement tag values request
2020-02-26 09:51:02 -05:00
2a4d2fe52d
feature: Implement tag keys request
2020-02-26 09:51:02 -05:00
c85f4b45ed
refactor: use raw strings instead of escape sequences
2020-02-21 09:55:18 -05:00
cc0d830fec
test: Exercise gRPC in the end-to-end test
2020-02-17 16:37:43 -05:00
5384d0d3b8
refactor: Switch org_id to primarily be u32 in tests
...
This will make it easier to have one source of truth for the org id
used in the gRPC and regular requests.
2020-02-17 16:37:43 -05:00
fdcef79ae3
improvement: Make end-to-end test async
...
This doesn't matter very much right now, since we're waiting for each
call to be done before asserting, but tonic client methods are async.
2020-02-17 16:37:43 -05:00
9cce1e4882
test: Add an end-to-end test
...
This test:
- Runs the server in a thread
- Writes some data
- Reads some data
- Shuts down the server
2020-02-13 10:40:03 -05:00
Andrew Lamb
Carol (Nichols || Goulding)
alamb
Edd Robinson
Jake Goulding
Carol (Nichols || Goulding)
Paul Dix