Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
1,110 Commits
266 Branches
455 Tags
360 MiB
Commit Graph

438 Commits (a3b88d55067c3b022e242ebc641cb563f04c4e4e)

Author SHA1 Message Date
Carol (Nichols || Goulding) 5f49543fdf feat: Support storage of points with String and Boolean fields 2020-08-14 10:57:20 -04:00
Carol (Nichols || Goulding) d59702ec79 feat: Make the create bucket HTTP API match the Influx 2.0 API 
The `/api/v2/create_bucket` API was delorean-specific for testing
purposes. This change makes it match the [Influx 2.0 API][influx] and
adds a method to the client for creating buckets.

The client will always send an empty array of `retentionRules` because
that is a required parameter for the Influx API. Delorean always ignores
`retentionRules`. The `description` and `rp` parameters are optional and
are never sent.

[influx]: https://v2.docs.influxdata.com/v2.0/api/#operation/PostBuckets

I believe the gRPC create bucket is also delorean-specific and perhaps
not needed, but I'm leaving it in for now with a note.
 2020-08-12 10:08:32 -04:00
Carol (Nichols || Goulding) 34c0977820 test: Add line parser tests inspired by line protocol generation 2020-07-22 17:03:29 -04:00
Carol (Nichols || Goulding) 9302af7c56 refactor: Improve InputPath to InputReader conversion 
By extracting the conversion code into a method named input_reader.
 2020-07-20 20:32:09 -04:00
Andrew Lamb 143c350ecb
Merge pull request #250 from influxdata/alamb/feat-multi-col-stats 
feat: Update stats command to handle directories of files
 2020-07-20 16:48:31 -04:00
alamb b8dce655b3 fix: Fix use of predicate 2020-07-20 16:29:44 -04:00
Andrew Lamb 48c37e193e
fix: Apply suggestions from code review 
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
 2020-07-20 16:22:46 -04:00
Carol (Nichols || Goulding) b3f6d4ab69 refactor: Make conversion from paths to InputReaders clearer 2020-07-20 10:49:05 -04:00
alamb ca1bd79902 feat: Update stats command to handle directories of files 2020-07-17 16:47:11 -04:00
Carol (Nichols || Goulding) 9a743e7c36
fix: Switch logging level from debug to info 
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
 2020-07-16 20:32:55 -04:00
Carol (Nichols || Goulding) b1ce3958e5 fix: Define a fully-featured error for the Id type 2020-07-16 20:30:54 -04:00
Carol (Nichols || Goulding) 5afc3a1e3b fix: Ignore anything in the WAL dir that doesn't parse as an Org ID instead of crashing 2020-07-16 11:28:57 -04:00
Edd Robinson 627805c567 Merge branch 'er/feat/merge-blocks' of github.com:influxdata/delorean into er/feat/merge-blocks 2020-07-13 10:41:03 +01:00
Edd Robinson b62810676d feat: add support for merging blocks 2020-07-13 10:39:36 +01:00
Edd Robinson ad099fc722
refactor: PR feedback 
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
 2020-07-13 10:38:47 +01:00
Edd Robinson e47277b967 feat: add support for merging blocks 2020-07-10 14:02:38 +01:00
alamb 6d78647602 Merge remote-tracking branch 'origin/master' into alamb/fix-partition-errors 2020-07-10 08:49:09 -04:00
Andrew Lamb 6ee72e5733
Merge pull request #239 from influxdata/alamb/less-chatty 
fix: Make default output of delorean less chatty
 2020-07-10 06:59:54 -04:00
alamb d1e334f3fe fix: Make default output of delorean less chatty 2020-07-08 18:02:48 -04:00
Edd Robinson bd5d39f60c refactor: address PR feedback 2020-07-08 22:57:15 +01:00
Edd Robinson d19094d023 refactor: PR feedback 
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
 2020-07-08 22:57:15 +01:00
Edd Robinson 4e455ff595 refactor: PR feedback 
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
 2020-07-08 22:57:15 +01:00
Edd Robinson a3a8b8273b refactor: PR feedback 
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
 2020-07-08 22:57:15 +01:00
Edd Robinson a8f77c49e2 refactor: PR feedback 
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
 2020-07-08 22:57:15 +01:00
Edd Robinson c5d6509ad5 feat: extend multi file conversion to CLI 2020-07-08 22:57:15 +01:00
alamb bb2518b7e9 fix: Remove StorageError 2020-07-08 17:32:16 -04:00
alamb febfbe4d45 fix: update error handling in memdb.rs and partitioned_store.rs to use standard errors 2020-07-08 17:32:16 -04:00
alamb 49555aad05 Merge remote-tracking branch 'origin/master' into alamb/fix-block-errors 2020-07-08 17:26:03 -04:00
Andrew Lamb 4a9fc5b785
fix: Apply suggestions from code review 
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
 2020-07-08 17:25:44 -04:00
alamb e24e520176 fix: update error handling in block.rs 2020-07-08 11:37:06 -04:00
alamb 003a08331b fix: remove special case error message 2020-07-08 11:16:19 -04:00
alamb c840317428 fix: update error handling in database.rs 2020-07-08 11:11:02 -04:00
alamb 0ad6cb61b1 refactor: make delorean main async 2020-07-06 16:32:37 -04:00
alamb 94a39d8700 fix: use fail in error handling 2020-07-04 07:00:28 -04:00
Jake Goulding cab7cbb001 fix: Correct typo 2020-07-03 15:21:57 -04:00
alamb 4de9783409 fix: fix logical merge conflict 2020-07-03 08:15:31 -04:00
alamb 0c69fb0311 fix: clippy 2020-07-03 07:58:16 -04:00
alamb 8bd0a7b332 Merge remote-tracking branch 'origin/master' into alamb/server-errors-204 2020-07-03 07:03:34 -04:00
alamb 43b7614491 fix: simplification 2020-07-03 06:59:32 -04:00
alamb d3fe1c2751 fix: return bucket_name rather than id, and improve legibility 2020-07-03 06:50:17 -04:00
alamb 7297b41ee3 fix: return bucket_name rather than id, and improve legibility 2020-07-03 06:40:21 -04:00
alamb 598b246335 fix: implement chunk of PR review comments 2020-07-03 06:33:07 -04:00
Andrew Lamb 6884f70a87
Merge pull request #212 from influxdata/alamb/storage-errors1 
fix: Improve storage predicate error handling
 2020-07-02 17:31:52 -04:00
alamb 5fad7db0ac feat: Add better error handling and logging for http_routes 2020-07-01 16:08:49 -04:00
alamb f92673a393 refactor: Break up server into modules 2020-07-01 12:46:44 -04:00
Andrew Lamb b5a91fae36
Merge pull request #207 from influxdata/alamb/faster-test-harness 
fix: speed up end-to-end test harness so it polls rather than waits; Add PING route
 2020-07-01 12:46:05 -04:00
Andrew Lamb 54ebf380de
Merge pull request #208 from influxdata/alamb/more-logging 
feat: improve error messages and logging
 2020-07-01 08:49:50 -04:00
Andrew Lamb d61d4ae79a
fix: Apply suggestions from code review 
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
 2020-07-01 08:46:52 -04:00
alamb 7c045b6be4 fix: Improve storage predicate error handling 2020-07-01 08:27:25 -04:00
alamb a509766e04 fix: remove whitespace for import 2020-06-30 16:23:53 -04:00
Andrew Lamb eb8aba75ec
fix: Update src/id.rs from PR review 
Co-authored-by: Edd Robinson <me@edd.io>
 2020-06-30 16:23:35 -04:00
alamb 9c11ae9869 feat: improve error messages and logging 2020-06-30 12:46:54 -04:00
alamb 8be278aadd fix: speed up end-to-end test harness so it polls rather than waits 2020-06-30 12:21:25 -04:00
alamb dc8d655486 feat: Implement CapabilitiesResponse 2020-06-30 11:17:45 -04:00
Andrew Lamb 97a5eb7e19
Merge pull request #197 from influxdata/alamb/log-requests 
feat: Log gRPC calls using trace crate, allow custom log levels
 2020-06-30 10:47:11 -04:00
Andrew Lamb 07e61cdf06
fix: fix comments 
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
 2020-06-29 16:35:07 -04:00
Carol (Nichols || Goulding) 2bf9037422 refactor: Use built-in clap functionality that uses FromStr to enum 
Then we can pass around an enum value and not worry about doing the
conversion anywhere else.
 2020-06-29 10:54:48 -04:00
Carol (Nichols || Goulding) c6f2508abe fix: Use clap's possible_values method 
This gets us built-in help text and error messages, and does less work
before failing because of an unsupported value.

Before this change, the help text was:

```
OPTIONS:
        --compression-level <compression_level>
            Compression level: max or compatibility (default). [default: compatibility]
```

After this change, the help text is:

```
OPTIONS:
        --compression-level <compression_level>
            How much to compress the output data. 'max' compresses the most; 'compatibility' compresses in a manner more
            likely to be readable by other tools. [default: compatibility]  [possible values: max, compatibility]
```

Before this change, if you supplied an unsupported value, the error was:

```
[2020-06-29T14:47:42Z INFO  delorean::commands::convert] convert starting
[2020-06-29T14:47:42Z INFO  delorean::commands::convert] Preparing to convert 591 bytes from tests/fixtures/lineproto/temperature.lp
Conversion failed: Error creating a parquet table writer Unknown compression level 'foo'. Valid options 'max' or 'compatibility'
```

After this change, the error is:

```
error: 'foo' isn't a valid value for '--compression-level <compression_level>'
	[possible values: compatibility, max]
```
 2020-06-29 10:53:01 -04:00
alamb 5fe022366b fix: Use std::str::FromStr and add -c 2020-06-29 10:25:43 -04:00
alamb e8a0233b09 feat: add configurable compression level 2020-06-29 10:25:43 -04:00
alamb 283d6691c6 feat: enable rpc debug tracing, tweaked logging levels, respect RUST_FMT env var 2020-06-29 09:59:22 -04:00
Jake Goulding 99549953d6
Merge pull request #193 from influxdata/idiomatic-snafu 2020-06-26 16:35:23 -04:00
Jake Goulding 672afa1195 refactor: Merge unknown and missing extension errors 2020-06-26 16:07:35 -04:00
Jake Goulding aca63b68b2 refactor: Use more unique errors for the commands 2020-06-26 16:01:59 -04:00
Carol (Nichols || Goulding) 7f838efd80
Merge pull request #187 from influxdata/cn-bool-enc 2020-06-26 15:45:42 -04:00
Jake Goulding bca4d00716 refactor: use SNAFU more idiomatically in command modules 2020-06-26 13:26:51 -04:00
Carol (Nichols || Goulding) 9ef07cac14 feat: Implement boolean encoding 
Fixes #149.
 2020-06-26 10:36:27 -04:00
Carol (Nichols || Goulding) 4df99f1a7c style: Enable the clippy warning to use `Self` when recommended 
Fixes #158.
 2020-06-25 07:38:58 -04:00
Carol (Nichols || Goulding) afcd1efd1e style: Unify lints everywhere 
Then fix the failures, mostly by adding derives and then removing some
unneeded (cheap) clones.

Document places where we purposefully don't use the same lints.

Not unifying missing_docs.

👀 https://github.com/rust-lang/cargo/issues/5034
 2020-06-25 07:28:42 -04:00
Carol (Nichols || Goulding) 423ee71f5e refactor: Remove duplicated lint rules 
These get inherited from crate root files, so the lint rules in
src/main.rs apply in this file already.
 2020-06-24 16:56:16 -04:00
Andrew Lamb ab22384009
Merge pull request #186 from influxdata/alamb/refactor-parquet-deps 
refactor: clean up parquet library deps and remove use of InputReaderAdapter (related to parquet dependencies)
 2020-06-24 09:42:44 -04:00
Carol (Nichols || Goulding) 6fb107af68
Merge pull request #178 from influxdata/cn-u64-enc 2020-06-24 08:48:57 -04:00
alamb 2c4a9dba53 fix: cleanup comment + code order 2020-06-23 17:21:20 -04:00
alamb b22423621b refactor: remove InputReaderAdapter 2020-06-23 17:15:02 -04:00
Andrew Lamb 86a425e5ef
feat: Add support for parsing bool values in line protocol parser (#156) 
* feat: Implement boolean support for the line protcol parser

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: fmt+clippy

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
 2020-06-22 16:58:38 -04:00
Carol (Nichols || Goulding) 294163bed0 feat: Implement unsigned encoding 2020-06-22 16:52:24 -04:00
Carol (Nichols || Goulding) 1e341a7321 fix: Encode and decode string data as bytes 
String data isn't guaranteed to be UTF-8
 2020-06-22 15:32:14 -04:00
Carol (Nichols || Goulding) d7dbf061cb feat: Implement String encoding/decoding 
Fixes #148.
 2020-06-22 15:15:34 -04:00
Edd Robinson 106bd69b5a feat: support converting from TSM->Parquet 2020-06-22 18:56:17 +01:00
Edd Robinson 621f2f91f0 refactor: hoist tsm mapper to delorean_tsm 2020-06-22 18:56:17 +01:00
Edd Robinson f046dbeea0 refactor: organise code in delorean_tsm crate 2020-06-22 18:56:17 +01:00
Edd Robinson 0ca6fdfa5f refactor: StorageError -> TSMError 2020-06-22 18:56:17 +01:00
Edd Robinson 85e0b4ec16 refactor: hoist tsm reader into own crate 2020-06-22 18:56:17 +01:00
Edd Robinson fd9f2ea5b8 refactor: split out index reading and block decoding 
This commit splits out the functionality required to read a TSM file's
index, and decode the blocks within the file.
 2020-06-22 18:56:17 +01:00
Edd Robinson 6339083b87 feat: implement mapping between blocks and table 
This commit implements the ability to map from multiple columns into a
single tablular view, where columns are aligned by their timestamp
components.
 2020-06-22 18:56:17 +01:00
Edd Robinson 5418b34fcc feat(tsm): map TSM data model to table model 
This commit adds a new type `TSMMeasurementMapper` that will iterate
through a `TSMReader`'s index and collect together all series and blocks
by measurement. These units are called `MeasurementTable`s.
 2020-06-22 18:56:17 +01:00
Andrew Lamb 506a7f19d5
feat: add stats command for computing storage statistics (#160) 
* feat: add stats command for computing storage statistics

* fix: Make function name less redundant

* fix: Apply suggestions from code review

Co-authored-by: Jake Goulding <jake.goulding@integer32.com>

* fix: remove changes to Cargo.lock

* fix: Cleanup to use `impl Into<String>` rather than String::from

* fix: fmt

* fix: less mut builder

* fix: fmt

* fix: clippy/fmt

* fix: fixup test

* fix: remove dstool reference

Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
 2020-06-22 07:33:52 -04:00
Carol (Nichols || Goulding) 78c2fb99ac refactor: Remove all references to 'dstool' 
Since it got merged into the main `delorean` binary.
 2020-06-19 11:08:59 -04:00
Andrew Lamb 8185c80c03
fix: fix logical merge conflict (#169) 2020-06-18 18:51:25 -04:00
Andrew Lamb a106e55fa6
feat: Add parquet metadata dumping (#159) 
* feat: Add parquet metadata dumping

* fix: Update delorean_parquet/src/error.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
 2020-06-18 18:34:49 -04:00
Andrew Lamb ae37548980
feat: Add support for parsing string values in line protocol parser (#155) 
* feat: add debug logging on parser error

* feat: Add support for parsing string values in line protocol parser

* fix: Fix comment

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
 2020-06-18 12:44:17 -04:00
Andrew Lamb 91de50a3a7
refactor: Refactor convert command code to have a place for TSM (#164) 2020-06-18 09:57:54 -04:00
Andrew Lamb 8f51b8a5c1
fix: Avoid hard coded length in doc example (#146) 2020-06-16 16:38:29 -04:00
Andrew Lamb d9278263a7
feat: write multiple measurements to multiple parquet files (#138) 
* feat: write to a directory of parquet files

* feat: change LineProtocolConverter to push style, move sampling there

* feat: full push mode, write to multiple measurements

* fix: clarify comments on finalize

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: clippy/fmt

* fix: remove whitespace

* fix: Apply suggestions from code review

Co-authored-by: Jake Goulding <jake.goulding@integer32.com>

* fix: fmt

* fix: make it compile again

* fix: fixup comments

Co-authored-by: Jake Goulding <jake.goulding@integer32.com>

* fix: remove unecessary debug implementation

* fix: cleaner comment

Co-authored-by: Jake Goulding <jake.goulding@integer32.com>

* fix: clearer iterator name

Co-authored-by: Jake Goulding <jake.goulding@integer32.com>

* fix: Apply suggestions from code review

Co-authored-by: Jake Goulding <jake.goulding@integer32.com>

* fix: clean

* fix: make it compile

* fix: type fix

* fix: whitespace

* fix: more review comments

* fix: more review comments

* fix: code review comments + fmt

* fix: clippy

* fix: Use EscapedStr directly for performance

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
 2020-06-12 17:19:35 -04:00
Edd Robinson 5ff6652cfc refactor: simplify TSMReader API 
This commit simplifies the TSMReader API to reduce the amount of mutable
state, and simplify how it's used as an iterator.
 2020-06-10 22:42:24 +01:00
Andrew Lamb faf3f534ac
refactor: move all dstool code into delorean binary (#131) 
* refactor: move all dstool code into delorean binary

* fix: Move code/mods to make it compile and run

* fix: warn if db dir does not exist

* refactor: Match argument subcommands w/ more idomatic  rust

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

fix: restore hyper logging

fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: update expected code

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
 2020-06-10 16:04:46 -04:00
Andrew Lamb 1bc9517b5d
refactor: Move delorean server code into its own module (#130) 2020-06-09 12:28:56 -04:00
Andrew Lamb f1a3058b24
feat: Add file / metadata inspection + dumping with dstool (#112) 
* feat: Add file / metadata inspection + dumping

* fix: apply some PR review comments

* fix: apply suggestions from code review

Co-authored-by: Jake Goulding <jake.goulding@integer32.com>

* feat: Add tests, rearrange code into modules, add gzip aware interface

* fix: fix comment and test

* fix: test output and fmt

Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
 2020-06-09 10:10:55 -04:00
Jake Goulding df39eca043 style: Apply standard lints across all crates 2020-06-05 17:02:54 -04:00
Andrew Lamb e0c38d0976
chore: Add test to check for tsm reading errors, update doc example (#117) 
* chore: Add a test that decodes the entire tsm index

* fix: update test and change example to not use hard coded len

* fix: comment cleanup

* fix: clippy

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>

* fix: fmt/clippy after code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
 2020-06-05 16:22:56 -04:00