Commit Graph

3993 Commits (a27d8fd85907c72363273b374bb0544c6d9cce86)

Author SHA1 Message Date
Andrew Lamb e0c38d0976
chore: Add test to check for tsm reading errors, update doc example (#117)
* chore: Add a test that decodes the entire tsm index

* fix: update test and change example to not use hard coded len

* fix: comment cleanup

* fix: clippy

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>

* fix: fmt/clippy after code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2020-06-05 16:22:56 -04:00
Jake Goulding 68fb580b43
style: Re-enable the elided lifetimes lint and move generated types to their own crate (#119)
* refactor: rename the module containing generated types

The nested `delorean` was confusing anyway, and this will make more
sense when we extract a new crate.

* refactor: Move the generated types to their own crate

This allows us to have more lax warnings in that crate alone, keeping
the main crate more strict.

* style: Re-enable elided lifetimes lint in the main crate
2020-06-05 16:22:27 -04:00
Edd Robinson 82d554a288
Merge pull request #111 from influxdata/er-tsm-reader
feat: add InfluxDB 2.x TSM file reader
2020-06-04 16:04:39 +01:00
Edd Robinson 4201f7ebbd refactor: address PR feedback 2020-06-04 15:47:27 +01:00
Andrew Lamb e43ab6dc31
fix(dstool): extract schema from a sample of input rather than the whole thing (#113)
* fix: extract schema from references

* fix: use a slice reference rather than iterator

* fix: fmt and clippy
2020-06-04 10:25:36 -04:00
Edd Robinson 138ff7329d refactor: please the clippy gods 2020-06-04 14:36:43 +01:00
Edd Robinson aeeff5cfb7 docs: add some TSMReader documentation 2020-06-04 14:36:43 +01:00
Edd Robinson 887ffd5977 refactor: remove lifetime to make index re-usable 2020-06-04 14:36:43 +01:00
Edd Robinson 481ce5f136 refactor: use constants for block type 2020-06-04 14:36:43 +01:00
Edd Robinson 49c58f007e chore: ignore tsm files 2020-06-04 14:36:43 +01:00
Edd Robinson 76442b752a refactor: clippy 2020-06-04 14:36:43 +01:00
Edd Robinson e3db077121 feat: add API for series key information 2020-06-04 14:36:43 +01:00
Edd Robinson b94d4ddd94 feat: add support for decoding blocks 2020-06-04 14:36:43 +01:00
Edd Robinson 413738a264 feat: support org and bucket ID in entries 2020-06-04 14:36:43 +01:00
Edd Robinson 97e77a681b feat: add basic TSM reader
This commit adds a basic TSM file reader, which will provide an iterator
over the TSM index.
2020-06-04 14:36:43 +01:00
Edd Robinson 2914535c72 refactor: move Encoder trait up 2020-06-04 14:36:43 +01:00
Andrew Lamb 234b2f5752
feat: Line Protocol Schema extractor (#108)
* feat: schema inference from iterator of parsed lines

* fix: Clean up error handing even more

* fix: fmt

* fix: make a sacrifice to the clippy gods
2020-06-03 18:29:57 -04:00
Carol (Nichols || Goulding) f00f408f34
Merge pull request #110 from influxdata/cn-cleanup 2020-06-03 08:55:57 -04:00
Carol (Nichols || Goulding) 2e4008eb2e docs: Small documentation readability fixes
Make a URL into a link, make some punctuation consistent, add some code
formatting.
2020-06-03 08:50:29 -04:00
Carol (Nichols || Goulding) eff7fda966 docs: Change doc comment to apply to the containing item 2020-06-03 08:41:28 -04:00
Andrew Lamb 5d2c5de39d
feat: Structs to represent line protocol schema (#103)
* feat: Structs to represent line protocol schema

Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2020-06-03 08:39:35 -04:00
Andrew Lamb 54125e35c8
chore: touch up docs + test for multi-part EscapedStr (#109)
* chore: Doc + Test for multi-part EscapedStr

* fix: Add additional test case, for completeness
2020-06-03 08:09:12 -04:00
Andrew Lamb 18b05ce9ef
fix: move test of dstool to its delorean_storage_tool package (#107) 2020-06-02 16:10:30 -04:00
AJ f13c9658e2
Merge pull request #97 from influxdata/chore/build-image
chore(build): setup nightly ci image build
2020-06-02 10:21:49 -04:00
AJ Bond 86949a6fc1
fix(circleci): remove nightly install tools 2020-06-02 10:14:10 -04:00
AJ Bond da523ff290
fix(docker): include missing dependencies
This includes libssl-dev and pkg-config since I disabled install recommends
2020-06-02 10:00:22 -04:00
AJ Bond 888c876429
chore(docker): updated based on review
Updated the nightly image based on review comments.
Switched base image and removed a few tools that caused the final image to be bloated
2020-06-02 10:00:22 -04:00
AJ Bond 6ad580127c
chore(Docker): change base image 2020-06-02 10:00:21 -04:00
AJ Bond 5aafbf1977
chore(build): setup nightly ci image build
We can reduce the number of install steps by using a custom container. This container will build nightly and publish to Quay
2020-06-02 10:00:17 -04:00
Andrew Lamb fbc74348c6
fix: consistent capitalization and data location (#105) 2020-06-02 08:00:13 -04:00
Edd Robinson fded360a4e
Merge pull request #104 from influxdata/er/spr
chore: ensure commit messages are semantic
2020-06-02 12:41:28 +01:00
Edd Robinson 2df6328384 chore: ensure commit messages are semantic 2020-06-02 12:34:30 +01:00
Andrew Lamb 1a2efdfd71
feat: Add dstool command line tool (#102)
* feat: Add dstool command line tool

* clippy

* Update delorean_storage_tool/src/main.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* Update delorean_storage_tool/src/main.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* Add in tests + PR comments

* fmt

* build first then run tests

* actually build before test

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-06-02 07:33:43 -04:00
Edd Robinson 11017e297f
Merge pull request #101 from influxdata/er/float-influxdb
refactor: add influxdb support to decoder
2020-06-02 12:28:48 +01:00
Edd Robinson 42bd4bf492 refactor: add influxdb support to decoder
InfluxDB's float encoder uses a different NaN value to terminate a
block. In order to support decoding float blocks that are inside of TSM
files created by InfluxDB this commit adds the option to explictly
decode those via another decode method.

In the future this API may get simplified if we decide we only need the
float encoder/decoder for encoding and decoding blocks solely
originating from InfluxDB.
2020-06-02 12:19:48 +01:00
Edd Robinson 84b3e4c463
Merge pull request #100 from influxdata/er-fix-rle
fix(encoders): RLE compatibility with influxdb
2020-06-01 16:08:22 +01:00
Edd Robinson 979e8ee2c2 fix(encoders): RLE compatibility with influxdb
The RLE encoder was implemented in a correct way, but not in a way that
was byte for byte compatible with the InfluxDB RLE encoder/decoder. This
commit changes the implementation to make this encoder compatible with
the InfluxDB implementation.
2020-05-31 10:02:06 +01:00
Jake Goulding e468b3c044
Merge pull request #99 from influxdata/updates
build: update semver-compatible versions
2020-05-29 17:36:17 -04:00
Jake Goulding 924f20fd50 build: update semver-compatible versions 2020-05-29 13:40:44 -04:00
Andrew Lamb 3e0c7e25ff
test: Add comments and some more tests to parsers (#96)
* test: Add comments and some more tests to parsers

* Update delorean_line_parser/src/lib.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* Update src/line_parser.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* Update src/line_parser.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* Update delorean_line_parser/src/lib.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* Remove end to end test as it is now covered by docs

* Update src/line_parser.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* one final tweak

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-05-28 05:32:40 -04:00
Carol (Nichols || Goulding) 54dfe826d7
Merge pull request #89 from influxdata/cn-wal-delete 2020-05-27 09:55:26 -04:00
Carol (Nichols || Goulding) 936e7c2a6f docs: Change some answered questions to statements 2020-05-27 09:49:05 -04:00
Carol (Nichols || Goulding) d84fdb13ec feat: Delete entries from the WAL up to some entry number 2020-05-27 09:46:02 -04:00
Carol (Nichols || Goulding) 4ac063088d
Merge pull request #91 from influxdata/cn-wal-metadata 2020-05-27 09:22:27 -04:00
Jake Goulding ac9958db30
Merge pull request #94 from influxdata/jpg/extract-parser 2020-05-27 08:36:55 -04:00
Carol (Nichols || Goulding) 0a02afe016
test: Wait for test server to stop before restarting (#92) 2020-05-26 17:34:54 -04:00
Jake Goulding d4af54c3de refactor: extract the line protocol parser to a separate crate
This will facilitate reusing the parser for other tasks.
2020-05-26 13:22:34 -04:00
Jake Goulding d394e2177f test: Rewrite line_parser tests at a lower level
This will ease extracting them to a separate crate
2020-05-25 20:57:48 -04:00
Jake Goulding e573ca0061 test: Combine duplicate tests 2020-05-25 20:57:39 -04:00
Andrew Lamb 4a0b2cd671
doc: Initial file encoding thoughts (#88) 2020-05-22 17:28:24 -04:00