Commit Graph

602 Commits (8be278aadd23b43896a823caffa2ecd17ce31afc)

Author SHA1 Message Date
Andrew Lamb 986e12d62a
refactor: Rename crate line_protocol_schema --> delorean_table_schema (#129)
* refactor: Rename crate line_protocol_schema --> delorean_table_schema

* fix: fmt
2020-06-09 11:56:16 -04:00
Andrew Lamb f1a3058b24
feat: Add file / metadata inspection + dumping with dstool (#112)
* feat: Add file / metadata inspection + dumping

* fix: apply some PR review comments

* fix: apply suggestions from code review

Co-authored-by: Jake Goulding <jake.goulding@integer32.com>

* feat: Add tests, rearrange code into modules, add gzip aware interface

* fix: fix comment and test

* fix: test output and fmt

Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2020-06-09 10:10:55 -04:00
Jake Goulding 9844cafc5d
refactor: Use SNAFU contexts more idiomatically (#127)
* refactor: Use SNAFU contexts more idiomatically

* fix: restore source error message

Co-authored-by: alamb <andrew@nerdnetworks.org>
2020-06-09 08:25:24 -04:00
Andrew Lamb 8475b6d183
feat: Add parquet writer, hook up conversion in dstool (#124)
* feat: Add parquet writer, hook up conversion in dstool

* fix: use bigger executor for test

* fix: less cloning

* fix: make unsupported messages less pejorative

* fix: fmt

* fix: Rename writer and do not require std::File, add example

* fix: clippy and fmt

* fix: remove unnecessary module in end to end tests

* fix: remove strange use of tempfile

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: cleanup use

* fix: Use more specific error messages

* fix: comment tweak

* fix: touchup temp path creation

* fix: clippy!

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-06-08 16:25:24 -04:00
Andrew Lamb ca9f9d4cae
feat: Add column packing code (#114)
* feat: Add column packing code

* fix: remove dependency on assert_approx_equal in favor of delorean_test_helpers

* fix: Cleanups from pr comments

* fix: Apply suggestions from code review

Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: more cleanup per code review

* fix: pr comments

* fix: remove explict string creation from caller

Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-06-06 06:04:41 -04:00
Jake Goulding 59c2e7b135
Merge pull request #121 from influxdata/common-lints
style: Apply standard lints across all crates
2020-06-05 22:27:02 -04:00
Andrew Lamb 2200def8ea
feat: Use rust nightly (#123) 2020-06-05 17:45:44 -04:00
Andrew Lamb 81810c2faa
fix: rename measurement.txt to measurement.lp for consistency (#122)
* fix: rename measurement.txt to measurement.lp for consistency

* fix: rename the file
2020-06-05 17:28:06 -04:00
Jake Goulding df39eca043 style: Apply standard lints across all crates 2020-06-05 17:02:54 -04:00
Andrew Lamb e0c38d0976
chore: Add test to check for tsm reading errors, update doc example (#117)
* chore: Add a test that decodes the entire tsm index

* fix: update test and change example to not use hard coded len

* fix: comment cleanup

* fix: clippy

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>

* fix: fmt/clippy after code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2020-06-05 16:22:56 -04:00
Jake Goulding 68fb580b43
style: Re-enable the elided lifetimes lint and move generated types to their own crate (#119)
* refactor: rename the module containing generated types

The nested `delorean` was confusing anyway, and this will make more
sense when we extract a new crate.

* refactor: Move the generated types to their own crate

This allows us to have more lax warnings in that crate alone, keeping
the main crate more strict.

* style: Re-enable elided lifetimes lint in the main crate
2020-06-05 16:22:27 -04:00
Edd Robinson 82d554a288
Merge pull request #111 from influxdata/er-tsm-reader
feat: add InfluxDB 2.x TSM file reader
2020-06-04 16:04:39 +01:00
Edd Robinson 4201f7ebbd refactor: address PR feedback 2020-06-04 15:47:27 +01:00
Andrew Lamb e43ab6dc31
fix(dstool): extract schema from a sample of input rather than the whole thing (#113)
* fix: extract schema from references

* fix: use a slice reference rather than iterator

* fix: fmt and clippy
2020-06-04 10:25:36 -04:00
Edd Robinson 138ff7329d refactor: please the clippy gods 2020-06-04 14:36:43 +01:00
Edd Robinson aeeff5cfb7 docs: add some TSMReader documentation 2020-06-04 14:36:43 +01:00
Edd Robinson 887ffd5977 refactor: remove lifetime to make index re-usable 2020-06-04 14:36:43 +01:00
Edd Robinson 481ce5f136 refactor: use constants for block type 2020-06-04 14:36:43 +01:00
Edd Robinson 49c58f007e chore: ignore tsm files 2020-06-04 14:36:43 +01:00
Edd Robinson 76442b752a refactor: clippy 2020-06-04 14:36:43 +01:00
Edd Robinson e3db077121 feat: add API for series key information 2020-06-04 14:36:43 +01:00
Edd Robinson b94d4ddd94 feat: add support for decoding blocks 2020-06-04 14:36:43 +01:00
Edd Robinson 413738a264 feat: support org and bucket ID in entries 2020-06-04 14:36:43 +01:00
Edd Robinson 97e77a681b feat: add basic TSM reader
This commit adds a basic TSM file reader, which will provide an iterator
over the TSM index.
2020-06-04 14:36:43 +01:00
Edd Robinson 2914535c72 refactor: move Encoder trait up 2020-06-04 14:36:43 +01:00
Andrew Lamb 234b2f5752
feat: Line Protocol Schema extractor (#108)
* feat: schema inference from iterator of parsed lines

* fix: Clean up error handing even more

* fix: fmt

* fix: make a sacrifice to the clippy gods
2020-06-03 18:29:57 -04:00
Carol (Nichols || Goulding) f00f408f34
Merge pull request #110 from influxdata/cn-cleanup 2020-06-03 08:55:57 -04:00
Carol (Nichols || Goulding) 2e4008eb2e docs: Small documentation readability fixes
Make a URL into a link, make some punctuation consistent, add some code
formatting.
2020-06-03 08:50:29 -04:00
Carol (Nichols || Goulding) eff7fda966 docs: Change doc comment to apply to the containing item 2020-06-03 08:41:28 -04:00
Andrew Lamb 5d2c5de39d
feat: Structs to represent line protocol schema (#103)
* feat: Structs to represent line protocol schema

Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2020-06-03 08:39:35 -04:00
Andrew Lamb 54125e35c8
chore: touch up docs + test for multi-part EscapedStr (#109)
* chore: Doc + Test for multi-part EscapedStr

* fix: Add additional test case, for completeness
2020-06-03 08:09:12 -04:00
Andrew Lamb 18b05ce9ef
fix: move test of dstool to its delorean_storage_tool package (#107) 2020-06-02 16:10:30 -04:00
AJ f13c9658e2
Merge pull request #97 from influxdata/chore/build-image
chore(build): setup nightly ci image build
2020-06-02 10:21:49 -04:00
AJ Bond 86949a6fc1
fix(circleci): remove nightly install tools 2020-06-02 10:14:10 -04:00
AJ Bond da523ff290
fix(docker): include missing dependencies
This includes libssl-dev and pkg-config since I disabled install recommends
2020-06-02 10:00:22 -04:00
AJ Bond 888c876429
chore(docker): updated based on review
Updated the nightly image based on review comments.
Switched base image and removed a few tools that caused the final image to be bloated
2020-06-02 10:00:22 -04:00
AJ Bond 6ad580127c
chore(Docker): change base image 2020-06-02 10:00:21 -04:00
AJ Bond 5aafbf1977
chore(build): setup nightly ci image build
We can reduce the number of install steps by using a custom container. This container will build nightly and publish to Quay
2020-06-02 10:00:17 -04:00
Andrew Lamb fbc74348c6
fix: consistent capitalization and data location (#105) 2020-06-02 08:00:13 -04:00
Edd Robinson fded360a4e
Merge pull request #104 from influxdata/er/spr
chore: ensure commit messages are semantic
2020-06-02 12:41:28 +01:00
Edd Robinson 2df6328384 chore: ensure commit messages are semantic 2020-06-02 12:34:30 +01:00
Andrew Lamb 1a2efdfd71
feat: Add dstool command line tool (#102)
* feat: Add dstool command line tool

* clippy

* Update delorean_storage_tool/src/main.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* Update delorean_storage_tool/src/main.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* Add in tests + PR comments

* fmt

* build first then run tests

* actually build before test

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-06-02 07:33:43 -04:00
Edd Robinson 11017e297f
Merge pull request #101 from influxdata/er/float-influxdb
refactor: add influxdb support to decoder
2020-06-02 12:28:48 +01:00
Edd Robinson 42bd4bf492 refactor: add influxdb support to decoder
InfluxDB's float encoder uses a different NaN value to terminate a
block. In order to support decoding float blocks that are inside of TSM
files created by InfluxDB this commit adds the option to explictly
decode those via another decode method.

In the future this API may get simplified if we decide we only need the
float encoder/decoder for encoding and decoding blocks solely
originating from InfluxDB.
2020-06-02 12:19:48 +01:00
Edd Robinson 84b3e4c463
Merge pull request #100 from influxdata/er-fix-rle
fix(encoders): RLE compatibility with influxdb
2020-06-01 16:08:22 +01:00
Edd Robinson 979e8ee2c2 fix(encoders): RLE compatibility with influxdb
The RLE encoder was implemented in a correct way, but not in a way that
was byte for byte compatible with the InfluxDB RLE encoder/decoder. This
commit changes the implementation to make this encoder compatible with
the InfluxDB implementation.
2020-05-31 10:02:06 +01:00
Jake Goulding e468b3c044
Merge pull request #99 from influxdata/updates
build: update semver-compatible versions
2020-05-29 17:36:17 -04:00
Jake Goulding 924f20fd50 build: update semver-compatible versions 2020-05-29 13:40:44 -04:00
Andrew Lamb 3e0c7e25ff
test: Add comments and some more tests to parsers (#96)
* test: Add comments and some more tests to parsers

* Update delorean_line_parser/src/lib.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* Update src/line_parser.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* Update src/line_parser.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* Update delorean_line_parser/src/lib.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* Remove end to end test as it is now covered by docs

* Update src/line_parser.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* one final tweak

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-05-28 05:32:40 -04:00
Carol (Nichols || Goulding) 54dfe826d7
Merge pull request #89 from influxdata/cn-wal-delete 2020-05-27 09:55:26 -04:00