53692a63d7
refactor: DRY up Packers accessors
2020-06-22 11:37:29 +01:00
128a3c3fc0
refactor: delegate on
2020-06-22 11:33:02 +01:00
49b5322487
feat: add resize_exact to packers
2020-06-22 11:25:17 +01:00
28057ca8ce
refactor: update delorean_table/src/packers.rs
...
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2020-06-22 11:25:17 +01:00
cba797e5ee
refactor: update delorean_table/src/packers.rs
...
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2020-06-22 11:25:17 +01:00
b4f44b61e4
refactor: Update delorean_table/src/packers.rs
...
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2020-06-22 11:25:09 +01:00
c26ac10b3b
refactor: update delorean_ingest/src/lib.rs
...
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2020-06-22 11:24:29 +01:00
f011748345
refactor: update delorean_parquet/tests/read_write.rs
...
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2020-06-22 11:24:29 +01:00
11132b9b30
docs: Update delorean_parquet/src/writer.rs
...
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2020-06-22 11:24:29 +01:00
146000d55b
refactor: update delorean_ingest/src/lib.rs
...
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2020-06-22 11:24:29 +01:00
cd435d9b51
refactor: update delorean_ingest/src/lib.rs
...
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2020-06-22 11:24:29 +01:00
ac7bb6bf68
refactor: make Packer generic
2020-06-22 11:24:29 +01:00
ac153fa3d2
Merge pull request #172 from influxdata/cn-dstool-cleanup
...
refactor: Remove all references to 'dstool'
2020-06-19 11:21:03 -04:00
99b5f95cfe
Merge pull request #171 from influxdata/doc-links
...
docs: Use correct Markdown link syntax
2020-06-19 11:17:28 -04:00
78c2fb99ac
refactor: Remove all references to 'dstool'
...
Since it got merged into the main `delorean` binary.
2020-06-19 11:08:59 -04:00
0e9653cd23
docs: Use correct Markdown link syntax
2020-06-19 10:39:58 -04:00
c778af92f5
Merge pull request #170 from influxdata/reduce-rebuilds
...
build: Prevent spurious rebuilds due to the object-store
2020-06-19 10:36:51 -04:00
6a97995a19
Merge pull request #144 from influxdata/rusoto-stream
2020-06-19 09:25:30 -04:00
bfb0213ac3
feat: Update Rusoto to allow streaming data on uploads
2020-06-19 09:18:44 -04:00
d5c39d931c
build: Prevent spurious rebuilds due to the object-store
...
The file `.env` wasn't precisely specified, which caused Cargo to
re-run the build script each time. Thankfully, `dotenv` returns the
file path that it picked to load, so we can use it directly.
2020-06-19 08:37:09 -04:00
8185c80c03
fix: fix logical merge conflict ( #169 )
2020-06-18 18:51:25 -04:00
a106e55fa6
feat: Add parquet metadata dumping ( #159 )
...
* feat: Add parquet metadata dumping
* fix: Update delorean_parquet/src/error.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-06-18 18:34:49 -04:00
ae37548980
feat: Add support for parsing string values in line protocol parser ( #155 )
...
* feat: add debug logging on parser error
* feat: Add support for parsing string values in line protocol parser
* fix: Fix comment
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-06-18 12:44:17 -04:00
3fac49d1ba
fix: encode timestamp values properly in parquet files ( #166 )
2020-06-18 12:24:55 -04:00
91de50a3a7
refactor: Refactor convert command code to have a place for TSM ( #164 )
2020-06-18 09:57:54 -04:00
2be21dab57
fix: Name benchmark group consistently ( #161 )
2020-06-17 20:01:17 -04:00
94f1968deb
feat: Improve line protocol parser error recovery, avoid infinite loop ( #152 )
...
* feat: Improve line protocol parser error recovery, avoid infinite loop
feat: port splitLines logic to rust line protocol parser
fix: consume trailing optional whitespace after timestamp
test: Add tests for same
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-06-17 17:44:04 -04:00
cf248f2143
feat: upgrade to latest arrow / byteorder ( #154 )
2020-06-17 12:50:23 -04:00
8f51b8a5c1
fix: Avoid hard coded length in doc example ( #146 )
2020-06-16 16:38:29 -04:00
7190b07b83
fix: Add additional encoding thoughts to doc ( #151 )
2020-06-15 10:33:02 -04:00
abb3338483
test: add an end to end test for writing multiple parquet files ( #145 )
...
* test: add an end to end test for writing multiple parquet files
* fix: whitespace ocd
2020-06-15 07:12:16 -04:00
d9278263a7
feat: write multiple measurements to multiple parquet files ( #138 )
...
* feat: write to a directory of parquet files
* feat: change LineProtocolConverter to push style, move sampling there
* feat: full push mode, write to multiple measurements
* fix: clarify comments on finalize
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: clippy/fmt
* fix: remove whitespace
* fix: Apply suggestions from code review
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
* fix: fmt
* fix: make it compile again
* fix: fixup comments
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
* fix: remove unecessary debug implementation
* fix: cleaner comment
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
* fix: clearer iterator name
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
* fix: Apply suggestions from code review
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
* fix: clean
* fix: make it compile
* fix: type fix
* fix: whitespace
* fix: more review comments
* fix: more review comments
* fix: code review comments + fmt
* fix: clippy
* fix: Use EscapedStr directly for performance
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2020-06-12 17:19:35 -04:00
c2c3e2f4c1
Merge pull request #143 from influxdata/more-lints
...
chore: Apply common lints to recently-created crates
2020-06-12 11:20:15 -04:00
1e2cc310d4
refactor: cleanup trim_leading in the parser, add tests ( #142 )
2020-06-12 11:04:40 -04:00
2c1180b27a
test: add some additional parser tests ( #137 )
2020-06-12 11:02:57 -04:00
b00f3ee977
chore: Apply common lints to recently-created crates
2020-06-12 09:26:18 -04:00
82da779203
Merge pull request #135 from influxdata/er/tsm-reader-refactor
...
refactor: simplify TSMReader API
2020-06-11 19:44:53 +01:00
f69ddf9a73
Merge pull request #134 from influxdata/cn-object-store
2020-06-11 14:14:11 -04:00
d507713503
refactor: Switch from HashMap to BTreeMap
2020-06-11 13:44:04 -04:00
5ff6652cfc
refactor: simplify TSMReader API
...
This commit simplifies the TSMReader API to reduce the amount of mutable
state, and simplify how it's used as an iterator.
2020-06-10 22:42:24 +01:00
e3b26c9961
test: Only run GCS and S3 tests if the env vars are set
...
I don't really like this because the tests will silently not be compiled
if you haven't set the environment variables, so you'd only notice you
weren't running the tests if you looked for those tests' output lines
and saw they weren't there.
Ideally, I'd like to print a warning, but this isn't possible because:
- Anything printed in tests doesn't show up by default
- Cargo's build scripts can't tell whether you're building as a
dependency or building for that crate's tests, so the warning would show
up even if you just depended on delorean_object_store
(https://github.com/rust-lang/cargo/issues/2549 )
2020-06-10 17:26:28 -04:00
ea1471c503
feat: Add an in-memory object store
2020-06-10 17:23:52 -04:00
4ad805863f
feat: Make GCS list return a stream, by wrapping its still-sync API
2020-06-10 17:23:52 -04:00
d83c410a5c
feat: Update to the released version of cloud-storage
...
My submitted API improvements got merged in!
2020-06-10 17:23:52 -04:00
8c878cdfd3
docs: Update capabilities
2020-06-10 17:23:52 -04:00
fb5d68654d
feat: Change AWS list to stream back batches of object names
2020-06-10 17:23:52 -04:00
d3283b1096
feat: Object storage in S3 and GCS
2020-06-10 17:23:52 -04:00
faf3f534ac
refactor: move all dstool code into delorean binary ( #131 )
...
* refactor: move all dstool code into delorean binary
* fix: Move code/mods to make it compile and run
* fix: warn if db dir does not exist
* refactor: Match argument subcommands w/ more idomatic rust
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
fix: restore hyper logging
fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: update expected code
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-06-10 16:04:46 -04:00
0415b233ec
refactor: Instantiate the table writer on demand ( #128 )
...
* refactor: instantiate ParquetWriter on demand, prep for multi measurements
* fix: doc test
* fix: update names
2020-06-09 16:11:42 -04:00
1bc9517b5d
refactor: Move delorean server code into its own module ( #130 )
2020-06-09 12:28:56 -04:00
Edd Robinson
Carol (Nichols || Goulding)
Jake Goulding
Carol (Nichols || Goulding)
Andrew Lamb