cbc11717cc
fix: Only do one string interner lookup when we want to insert
...
Previously, we needed to add an entry to the WAL when we added a new
string in the dictionary; now that the WAL entries are self-describing,
we only need to do one lookup here.
2020-09-24 16:20:09 -04:00
77f58efca7
chore: update Arrow/Parquet/DataFusion versions, consolidate references into new crate ( #309 )
...
* chore: consolidate all arrow/parquet/datafusion dependencies
* chore: update datafusion version
2020-09-24 08:46:54 -04:00
fb1e20c402
Merge pull request #306 from influxdata/cn/self-describing
...
feat: Switch to self-describing WAL entries
2020-09-23 21:29:49 -04:00
0b64840282
fix: Update schema for wal flatbuffers to be self describing
...
As recommended by Paul here: https://github.com/influxdata/delorean/issues/277#issuecomment-693670676
2020-09-23 21:22:02 -04:00
498478c066
refactor: rename delorean_storage_interface to delorean_storage ( #308 )
2020-09-22 17:18:53 -04:00
d0f2902c8d
feat: implement tag_keys and measurement_tag_keys ( #307 )
...
* feat: implement tag_keys and measurement_tag_keys
* fix: fix timestamp bound evaluation
2020-09-22 16:42:45 -04:00
8f624ac897
Merge pull request #304 from influxdata/benchmark-wal-restore
2020-09-21 09:22:37 -04:00
cf4cb72b1c
feat: Add a WAL restoration benchmark with multiple entries and partitions
2020-09-18 16:45:03 -04:00
648d42568d
feat: Add a benchmark for restoring the WAL
2020-09-18 16:45:01 -04:00
5858aa2d08
refactor: Changes to write buffer unrelated to self-describing change ( #303 )
...
* refactor: Extract column count to be a method on ParsedLine
* refactor: Simplify parsing nanoseconds as datetime
* refactor: Extract a constructor for WalEntryBuilder
* refactor: Extract a method on WalEntryBuilder to get bytes
* refactor: Avoid an iteration and a vec allocation
* refactor: Organize and alphabetize dependencies and imports
* fix: Propagate warning config to new crates
2020-09-18 16:29:19 -04:00
c86beb834a
Merge pull request #302 from influxdata/cn/partial-restore-test
2020-09-18 11:54:04 -04:00
841ee6e808
test: Improve display of test failures
2020-09-18 11:15:29 -04:00
10f5c116cc
test: Write a failing test for only restoring part of the WAL
...
The expected output for `cpu` probably isn't right; but this doesn't run
successfully yet.
2020-09-18 11:15:29 -04:00
eb7d5469b7
feat: Keep track of partitions to record them in a wal batch
...
So that each wal batch describes the partitions it uses.
Only restore a partition if it hasn't already been restored.
2020-09-18 11:15:29 -04:00
b774e3b08c
Merge pull request #301 from influxdata/cn/extract-restore-wal-function
2020-09-18 10:14:08 -04:00
c66726600b
refactor: Make restoring to partitions entirely separate from database
...
By removing the database name argument (which was only used for errors)
and making a separate error type for restoration errors.
This makes it easier to benchmark the restore_partitions_from_wal
function independently. This also has a nice side effect of splitting
errors that can happen during restoration from errors that can happen at
other times.
Switched some fields that were named `_id` to not have a suffix to set
up easier switching of those from u32 to String in a future commit.
2020-09-18 09:38:03 -04:00
73455044bd
feat: Add more structured information to SchemaMismatch errors
2020-09-18 09:23:37 -04:00
a1f62e3b35
refactor: Extract function to restore partitions from wal entries
2020-09-18 08:58:11 -04:00
5a27466295
refactor: Reorder a few lines to make the next extraction clearer
2020-09-18 08:55:49 -04:00
474dfad9fc
refactor: Extract a struct to hold statistics about restoring from WAL
2020-09-18 08:55:42 -04:00
363d3abb4d
Merge pull request #300 from influxdata/alamb/split-modules4
...
refactor: move partitioned_store into its own crate (bonus!)
2020-09-18 08:51:40 -04:00
2418ee5ab0
refactor: move partitioned_store into its own module
2020-09-18 08:12:19 -04:00
642b1b4370
refactor: move write_buffer to delorean_write_buffer crate ( #299 )
2020-09-18 08:11:48 -04:00
d2c24ef7af
refactor: pull storage interface into delorean_storage_interface ( #298 )
2020-09-18 07:58:19 -04:00
5fe3bfd53c
refactor: extract WalDetails into delorean_wal_writer crate ( #297 )
2020-09-18 07:47:37 -04:00
9cb63f47ff
feat: Implement timestamp range in measurement_names gRPC call + tests ( #296 )
2020-09-18 06:40:03 -04:00
d0350b9aac
refactor: move test specific database code to its own module ( #295 )
2020-09-17 12:31:29 -04:00
881c2961f6
feat: [WriteBuffer][gRPC] support for measurement_names ( #293 )
...
* refactor: move GrpcInputs to its own module so I can reuse it
* feat: Basic gRPC support for listing measurements, tests for same
* fix: move Fixture definition, rename client
* fix: remove confusing doc comment
2020-09-16 17:45:28 -04:00
2d3a25b465
Merge pull request #288 from influxdata/cn/snappy-compression
...
feat: Compress WAL entries with Snappy
2020-09-16 08:35:17 -04:00
565e2909ea
docs: Add some additional context to write_buffer_routes.rs ( #292 )
2020-09-16 07:56:10 -04:00
358a548a0a
feat: implement skeleton Storage gRPC interface ( #290 )
2020-09-15 15:30:57 -04:00
2d55076c48
feat: Basic gRPC server tests ( #289 )
...
* refactor: move TestDatabaseStore to test_fixtures
* feat: Basic gRPC server tests
2020-09-15 15:11:02 -04:00
56d184f80c
chore: remove outdated comment from protobuf definition ( #291 )
2020-09-15 15:08:54 -04:00
11e16c78d9
feat: initial WriteBuffer gRPC implementation ( #287 )
...
* feat: initial WriteBuffer gRPC implementation
* fix: remove spurious comments found in code review
2020-09-14 15:35:41 -04:00
f9fe6cdd31
test: add test for gzip content-encoding ( #286 )
2020-09-14 15:23:51 -04:00
596c987956
feat: Compress WAL entries with Snappy
...
Fixes #276 .
2020-09-14 09:42:54 -04:00
90eba702ca
refactor: Remove org/bucket from WriteBufferDatabase interface ( #285 )
2020-09-13 06:09:19 -04:00
c4a738dfbe
Revert "refactor: Remove org/bucket from WriteBufferDatabase interface"
...
This reverts commit f7ad7afc4f
2020-09-12 07:06:33 -04:00
f7ad7afc4f
refactor: Remove org/bucket from WriteBufferDatabase interface
2020-09-12 07:05:54 -04:00
82d5f485c3
test: traits for database and tests for http handler ( #284 )
...
* test: traits for database and tests for http handler
* refactor: Use generics and trait bounds instead of trait objects
* refactor: Replace trait objects with an associated type
* refactor: Extract an associated Error type on the Database traits
* refactor: Remove some explicit conversions to_string that Snafu takes care of
* docs: add comments
* refactor: move traits into storage module
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@integer32.com>
2020-09-11 17:42:00 -04:00
44bb230b2d
Merge pull request #283 from influxdata/alamb/line-proto-display
...
feat: implement ParsedLine --> LineProtocol
2020-09-11 17:20:22 -04:00
9b9ff484bb
fix: implement escaping
2020-09-11 17:14:35 -04:00
4cf03b6243
fix: remove unecessary test for invalid conversion
2020-09-11 16:23:40 -04:00
d4ae7d3466
fix: postfix integer with "i"
2020-09-11 16:22:57 -04:00
babd54b321
docs: adjust comment
2020-09-11 16:20:08 -04:00
b7a061d2a8
feat: implement ParsedLine --> LineProtocol
2020-09-11 08:06:08 -04:00
37a982d3b0
Merge pull request #282 from influxdata/alamb/gzip
...
feat: support gzip content-encoding for api/v2/write endpoint
2020-09-10 12:04:13 -04:00
e55899f73b
feat: support gzip content-encoding for api/v2/write endpoing
2020-09-10 11:49:50 -04:00
ef8f0273a8
Merge pull request #281 from influxdata/alamb/docs
...
docs: add comments, adjust logging levels, proper error reporting for line protcol parse errors
2020-09-10 11:47:50 -04:00
3275fc8846
docs: add comments, adjust logging levels, proper error reporting for line protcol parse errors
2020-09-10 11:10:09 -04:00
Carol (Nichols || Goulding)
Andrew Lamb
Carol (Nichols || Goulding)
Jake Goulding
alamb