Commit Graph

1095 Commits (8e9dd227534f57e607c7d6b89fad055bde6e02fe)

Author SHA1 Message Date
Edd Robinson 3622a92c8b feat: wire in rb column metrics 2021-05-11 13:00:52 +01:00
Marco Neumann 795f5bfcb7 refactor: make `StatValues::{min,max}` optional + handle NaNs
This will allow us to:

- handle all-NULL columns correctly
- be in-line with Parquet (where min/max are optional)
- handle NaNs at least somewhat sane (they do not "poison" stats
  anymore)
2021-05-10 17:12:25 +02:00
Andrew Lamb f037c1281a
feat: Calculate all system tables "on demand" (#1452)
* feat: compute system.columns table on demand

* feat: compute system.chunk_columns on demand

* feat: compute system.operations on demand

* fix: fixup schemas

* fix: Log errors

* fix: clippy

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-10 14:43:55 +00:00
Marko Mikulicic 9f5350a6c5
fix: Load only databases for which a config exists
Closes #1450
2021-05-10 13:14:22 +02:00
Nga Tran c6b933eb63 chore: merge main to branch 2021-05-07 18:40:17 -04:00
Nga Tran 971500681f refactor: address Andrew's and Carol's comment 2021-05-07 17:33:19 -04:00
Nga Tran ba015ee4df refactor: clean up and add comments 2021-05-07 09:31:41 -04:00
Edd Robinson eae3fec571 feat: wire up regex UDF as predicate filter expr 2021-05-07 13:44:51 +01:00
Andrew Lamb b5ea71f45f
feat: Expose the storage usage for each column in system.chunk_columns (#1441)
* feat: Expose the storage usage for each column in system.chunk_columns

* fix: fixup logical conflicts

* refactor: move coalsce logic into the read buffer

* fix: Update system_tables to not use coalese

* fix: Improve comments

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-05-07 12:36:49 +00:00
Raphael Taylor-Davies 9320f59de0
feat: add shard sink indirection (#1447)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-07 11:04:51 +00:00
Andrew Lamb d7253c72c0
feat: Only calculate system.chunks table "on demand" (#1446)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-07 10:26:56 +00:00
Carol (Nichols || Goulding) febc1538ff
chore: Update Rust version (#1445)
* chore: Update Rust version

* refactor: Make struct constructor field orderings consistent

Sometimes I changed the struct definition, sometimes changed the struct
construction instance, depending on consistency with code around each
(other similar structs, function argument orders, etc)

More info: https://rust-lang.github.io/rust-clippy/master/index.html#inconsistent_struct_constructor

* refactor: Use flatten where appropriate

One instance is a false positive with a clippy bug.

More info:

- https://rust-lang.github.io/rust-clippy/master/index.html#filter_map_identity
- https://rust-lang.github.io/rust-clippy/master/index.html#manual_flatten

* refactor: Use Option map instead of match

More info: https://rust-lang.github.io/rust-clippy/master/index.html#manual_map

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 22:07:10 +00:00
Nga Tran 55bf848bd2 feat: Now we can query directly from files in object store 2021-05-06 18:02:17 -04:00
Raphael Taylor-Davies 7f6b11266d
feat: instrument catalog locks (#1355) (#1439)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 17:09:52 +00:00
Raphael Taylor-Davies 44de42906f
refactor: use Arc<str> instead of Arc<String> (#1442) 2021-05-06 17:05:08 +00:00
Raphael Taylor-Davies 49c0b8b90c
feat: pull-based metrics (#1355) (#1414)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 15:54:30 +00:00
Raphael Taylor-Davies 216903a949
refactor: move protobuf conversion logic to generated_types (#1437)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 15:49:27 +00:00
Andrew Lamb 884baf7329
feat: add column_type and influxdb_column_type, remove row_count from system.columns (#1415)
* feat: add column_type and influxdb_column_type, remove row_count from system.columns

* fix: update tests

* fix: more test update

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: fmt

* fix: copy/paste type conversion to avoid cross dependency between data_types and internal_types

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-05-06 12:59:30 +00:00
Marko Mikulicic 578dc0db25
feat: Add more logs to shed light on the curious incident with missing metrics in the nighttime 2021-05-06 14:42:48 +02:00
Raphael Taylor-Davies 10f89a3e8d
refactor: split entry out into separate crate (#1428)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 11:36:23 +00:00
Nga Tran a5c92fae8a chore: merge main to branch 2021-05-05 13:48:42 -04:00
Raphael Taylor-Davies 411cf134e9
refactor: explode arrow_deps (#1425)
* refactor: explode arrow_deps

* chore: workaround doctest bug
2021-05-05 16:59:12 +00:00
kodiakhq[bot] 4395ede244
Merge branch 'main' into debug-chunk-metrics 2021-05-05 15:43:32 +00:00
Marko Mikulicic 2b0d7cfb91
feat: Add debug to update_chunk_state metrics 2021-05-05 17:37:57 +02:00
Nga Tran fcb37a0b1d feat: more testing scenarios for quering parquet files 2021-05-05 10:57:02 -04:00
Carol (Nichols || Goulding) 4a64e22e64 refactor: Use trait object and deref instead of cloning Arc in tests 2021-05-05 10:55:12 -04:00
Carol (Nichols || Goulding) e32fa43a53 docs: Add note about implication of write buffer errors 2021-05-05 10:55:12 -04:00
Carol (Nichols || Goulding) 7d5c988fba feat: Actually route SequencedEntry to the Write Buffer, if present
Connects to #1157.

Rearrange some code and comments to be consistent with the design. Make
some more places not care whether they're getting an owned or borrowed
SequencedEntry.
2021-05-05 10:55:11 -04:00
Carol (Nichols || Goulding) 54c5f984d5 fix: Use stdlib's path manipulation rather than format
The syntax highlighting in my editor broke because of the unmatched
double quote, which got me to look a bit closer at this test. These
tests would have failed on Windows.
2021-05-05 10:55:11 -04:00
Carol (Nichols || Goulding) 231abd221f refactor: Extract a TestDbBuilder 2021-05-05 10:55:11 -04:00
Carol (Nichols || Goulding) 62dfb47825 refactor: Reorganize test imports 2021-05-05 10:55:11 -04:00
Marco Neumann 9e61b470e7 feat: change MemoryStream to accept multiple record batches 2021-05-05 13:29:16 +02:00
Marco Neumann 34754ebcdb refactor: move MemoryStream to arrow_deps 2021-05-05 13:29:16 +02:00
Edd Robinson 733d502350 refactor: fix tests 2021-05-04 18:38:42 +01:00
Edd Robinson 9aa144e0f4 feat: add per-stage current chunk storage 2021-05-04 17:43:53 +01:00
Andrew Lamb 3b7c5ac350
fix(storage rpc): do not send back tags with empty values (#1403) 2021-05-04 10:35:24 +00:00
Marko Mikulicic b579ef8646
feat: Add jemalloc stats 2021-05-03 12:10:48 +02:00
kodiakhq[bot] 3c5595d046
Merge branch 'main' into ntran/unload_chunks 2021-04-30 22:02:38 +00:00
Paul Dix 979f5f9347 refactor: write buffer to use sequenced entry and new segment
This refactors the write buffer to use the sequenced entry structure and the new segment definition. It removes the old replicated write and write_buffer.fbs.

Finally, it updates the SequencedEntry wrapper type around the Flatbuffer structure to be a trait so that SequencedEntry can be initialized from a borrowed Flatbuffer or an owned Vec<u8>.

How writes go into segments in the buffer and any kind of validation will likely have to be updated based on what kinds of guarantees we want to make in the buffer. However, that should probably come after we've rethought the design a bit around the new layout of chunks in the Parquet persistence.
2021-04-30 17:00:23 -04:00
Raphael Taylor-Davies a967ebfabd
refactor: rename closing to closed (#1396)
* refactor: rename closing to closed

* refactor: further renames
2021-04-30 20:59:45 +00:00
Nga Tran 34a3388a49 feat: unload chunks from read buffer but keep them in object store 2021-04-30 16:12:02 -04:00
Raphael Taylor-Davies c2f7e7efea
feat: warn on dropping from open partition (#1395)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-30 18:56:49 +00:00
Edd Robinson ebe671e59e refactor: include default labels 2021-04-30 14:03:09 +01:00
Andrew Lamb 40b9b09cdc
refactor: rename assert_table_eq to assert_batches_eq (#1368) 2021-04-30 10:51:08 +00:00
Nga Tran c9b33c6b7d chore: Merge branch 'main' into ntran/test_query_parquets 2021-04-29 14:22:34 -04:00
Nga Tran f89d56af8a chore: merge mian to branch 2021-04-29 14:14:04 -04:00
Andrew Lamb 2d643d4407
fix: set nullability correctly for system tables (#1387) 2021-04-29 18:09:27 +00:00
Nga Tran e87973babe refactor: address review comments 2021-04-29 13:15:43 -04:00
Edd Robinson ea476c86be feat: add chunk size histogram 2021-04-29 17:10:57 +01:00
Andrew Lamb 5ef9be4abd
refactor: use RecordBatch::try_from_iter to make system tables (#1374)
* refactor: use RecordBatch::try_from_iter to make system tables

* fix: clippy
2021-04-29 15:32:35 +00:00
Andrew Lamb 0788892413
feat: add row_count to system.chunks and Chunk managment API (#1373)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-29 13:53:25 +00:00
Marko Mikulicic 3672a20570
feat: Add ingest_entries_bytes_total counter 2021-04-29 14:03:05 +02:00
Raphael Taylor-Davies 262bf446ab
feat: change background worker to use backoff instead of poll (#1339) (#1347)
* feat: change background worker to use backoff instead (#1339)

* chore: fix comment

* chore: fix tests

* chore: review comments

* chore: review feedback
2021-04-29 11:10:51 +00:00
Andrew Lamb b1924d10b4
fix: Only show current database's system.operations table (#1310)
* test: add test for operations.system_tables

* fix: only show operations for current database

* fix: update test

* fix: improve test

* refactor: filter in Schema provider rather than in job tracker

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-28 21:06:19 +00:00
Nga Tran 402d9c748c chore: cargo fmt 2021-04-28 16:52:52 -04:00
Nga Tran 0ea8861a65 chore: fix merge that causes test failures 2021-04-28 16:38:29 -04:00
Nga Tran 2a2760bd18 feat: complete tests where data in both RUB and OS 2021-04-28 16:14:07 -04:00
Nga Tran 140d96dbea feat: tests ffor loading data to object store and make sure twe still query read buffer 2021-04-28 15:59:17 -04:00
Carol (Nichols || Goulding) 9aefcd216f fix: Validate that ClockValue is never 0 2021-04-28 13:54:55 -04:00
kodiakhq[bot] dcc732b154
Merge branch 'main' into cn/server-id-fix 2021-04-28 17:19:29 +00:00
Carol (Nichols || Goulding) 2f4d7189ff fix: Validate ServerId when creating structs from flatbuffers
When we get the flatbuffers, we won't have the server ID in addition to
the flatbuffers-- it's in the flatbuffers. But we want to validate the
`ServerId` once when the `SequencedEntry` is created so that its
`server_id` method can assume it has a valid `ServerId`.
2021-04-28 13:06:12 -04:00
Edd Robinson c4259d5336 refactor: update benchmarks 2021-04-28 16:11:57 +00:00
Edd Robinson a9ef604ef6 perf: avoid using channels for query execution
Pre-sized channels get full when the results to send over them are larger than the capacities. This causes significant runtime overhead and slows down query performance.

This commit removes the intermediate channels. The potential downside to this approach is there may be more buffering which could increase memory usage during query and also block a thread for longer periods of time.
2021-04-28 16:11:57 +00:00
Edd Robinson d6c0f40630 test: add read_group benchmark 2021-04-28 16:11:57 +00:00
Edd Robinson 5dfe312867 test: add benchmark for influxrpc read_filter 2021-04-28 16:11:57 +00:00
Marko Mikulicic b6054f9df9
feat: Make background task period configurable 2021-04-28 15:47:31 +02:00
Raphael Taylor-Davies 9b57214344
feat: move closing chunks as soon as possible (#1329)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-28 07:50:15 +00:00
Raphael Taylor-Davies 6bdc153361
feat: sort RUB (read buffer) chunks (#1308)
* feat: sort chunks before upserting to read buffer (#1216)

* chore: review feedback

* chore: fix merge conflict

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-27 16:50:48 +00:00
Raphael Taylor-Davies 7ca1da3fcd
feat: pushdown table and partition key predicates to catalog (#736) (#1327)
* feat: catalog predicate pushdown (#736)

* chore: fix lints

* chore: review comments

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-27 15:31:47 +00:00
Marco Neumann 91bccdfca3 ci: pass `--document-private-items` to `cargo doc` 2021-04-27 15:42:07 +02:00
kodiakhq[bot] e78a81ee59
Merge branch 'main' into crepererum/ci_doc 2021-04-27 12:03:04 +00:00
kodiakhq[bot] c3cb86fc1a
Merge branch 'main' into crepererum/issue1296 2021-04-27 11:58:10 +00:00
Marco Neumann eddc9319ff docs: deny broken intradoc links 2021-04-27 13:22:28 +02:00
Raphael Taylor-Davies 20117de078
feat: string dictionary encoding (#1220) (#1262)
* feat: string dictionary encoding (#1220)

* chore: review comments

* chore: fix lint

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-27 09:36:58 +00:00
Marco Neumann c8594ce743 refactor: chunks shall always have data (= 1 table)
Chunks now always have "data" (aka exactly 1 table including
schema/columns). Open chunks can only be created with data. Rollovers do
NOT create open chunks anymore (this is now only done for incoming
data).

Closes #1296.
2021-04-27 10:28:57 +02:00
Andrew Lamb 35bc09ef7c
refactor: use `FromIter` rather than array builders to create system table arrays (#1246)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-26 19:47:11 +00:00
Carol (Nichols || Goulding) 272cdb85ce fix: Use the ServerId type everywhere, for writing, querying, anything 2021-04-26 18:44:32 +00:00
Carol (Nichols || Goulding) b8face3335 refactor: Organize use statements 2021-04-26 18:44:32 +00:00
Jake Goulding 67f5ad841d refactor: Introduce ServerId and CurrentServerId types 2021-04-26 18:44:32 +00:00
Raphael Taylor-Davies 0a835436ac
feat: use bitmasks within MUB (#1274) (#1289)
* feat: use bitmasks within MUB (#1274)

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-26 18:00:16 +00:00
Andrew Lamb e33af0c084 fix: remove wait_for_complete 2021-04-26 16:59:36 +00:00
Andrew Lamb fcf5c304a6 fix: fixup merge conflicts 2021-04-26 16:59:36 +00:00
Andrew Lamb 348f38c2c9 feat: add status column 2021-04-26 16:59:36 +00:00
Andrew Lamb 9c74074095 feat: List background operations as system.operations table 2021-04-26 16:59:36 +00:00
Edd Robinson faec98eab9 refactor: remove time column from row group 2021-04-26 09:51:06 +00:00
Edd Robinson dd2bee3343 fix: don't over count new chunks 2021-04-23 20:25:37 +01:00
Edd Robinson 552409ea70 feat: add metrics for chunk lifecycle 2021-04-23 17:13:43 +00:00
Edd Robinson 0e3b7bff49 refactor: plumb metrics registry to catalog 2021-04-23 17:13:43 +00:00
Edd Robinson eacfe9af6a refactor: address PR feedback 2021-04-23 15:58:48 +00:00
Edd Robinson d3218802ab refactor: more API metric coverage 2021-04-23 15:58:48 +00:00
Edd Robinson 97b2369140 refactor: swap existing metrics for THE NEW WAY 2021-04-23 15:58:48 +00:00
Marco Neumann f12f1eeef7 refactor: make chunks per-table
This changes the hierarchy from

```
database -> partition -> chunk -> table
```

to

```
database -> partition -> table -> chunk
```

Only the high-level APIs are changed for now. The chunk states (like
MutableBuffer and ReadBuffer) still multiplex tables, although they will
always only get a single table assigned (or no table if no data was
presented yet).

Closes #1256.
2021-04-23 09:18:55 +02:00
Raphael Taylor-Davies 74c25f541d
feat: fast MUB dictionary arrow conversion (#1273)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-22 20:31:44 +00:00
Marko Mikulicic 83d6550316 feat: Implement write_entry_downstream 2021-04-21 20:50:46 +00:00
Carol (Nichols || Goulding) 88ca1a5245 fix: Rename wal.fbs to write_buffer.fbs 2021-04-21 17:43:03 +00:00
Carol (Nichols || Goulding) 80995afb70 fix: Change WAL to Write Buffer in comments and documentation 2021-04-21 17:43:03 +00:00
Carol (Nichols || Goulding) 5f024401a4 refactor: Rename data_types::wal module to data_types::write_buffer 2021-04-21 17:43:03 +00:00
Carol (Nichols || Goulding) 46f9360c58 fix: Rename WAL_DIR to WRITE_BUFFER_DIR and redefine it as wb, not wal 2021-04-21 17:43:03 +00:00
Carol (Nichols || Goulding) 8baeaab8ca refactor: Rename Db.wal_buffer to write_buffer 2021-04-21 17:43:03 +00:00
Carol (Nichols || Goulding) 4bfcb68536 refactor: Remove a WalError that doesn't appear to be used 2021-04-21 17:43:03 +00:00
Carol (Nichols || Goulding) 0677c544bb refactor: Rename WalBufferRollover to WriteBufferRollover 2021-04-21 17:43:03 +00:00
Carol (Nichols || Goulding) ee45db402b refactor: Rename WalBufferConfig to WriteBufferConfig 2021-04-21 17:43:03 +00:00
Raphael Taylor-Davies 1cf3ab3978
feat: enable disabling MUB snapshot caching (#1203) (#1264)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-21 08:21:31 +00:00
Edd Robinson 508aa8be8c refactor: satisfy new clippy lints 2021-04-20 17:30:50 +00:00
Marco Neumann c5d99ba9af feat: wire up hard buffer limit
Fixes #1255.
2021-04-20 13:25:31 +02:00
Edd Robinson d05585f736 refactor: log chunk migration to rb at info 2021-04-19 16:08:49 +00:00
Carol (Nichols || Goulding) 82c1d94ce1 refactor: Use Option.map where possible 2021-04-19 08:48:11 -04:00
Marco Neumann fd0da7e74a chore: upgrade arrow and Rust
See https://github.com/apache/arrow/pull/10082 for upstream PR.
2021-04-19 14:00:04 +02:00
Nga Tran 4c23ca8888 feat: full implementation of parquet's read_filter for review 2021-04-16 16:03:24 -04:00
Andrew Lamb e226b5a820
feat: Use TimestampNanosecondArray for timestamps in IOx (#1230)
* refactor: Create Arrow arrays using iterators

* feat: use Timestamp64(TimeUnit::Nanosecond) for timestamps

* feat: add support for timestamp array

* fix: update more tests

* fix: remove unecessary code

Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-16 15:55:33 +00:00
Andrew Lamb f092294da3
fix: Use MAX (window end) for timestamps in read group (#1228)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-16 10:51:38 +00:00
Marko Mikulicic 878b1b318e feat: Initial scaffolding for routing layer
Part of #916

Adding first-class concept of ShardId in shard config, fixes #1156

NEXT:

- [ ] implement sharder
- [ ] implement `write_entry_downstream`
- [ ] add tests
2021-04-15 09:02:47 +00:00
kodiakhq[bot] 4001ec73c9
Merge branch 'main' into ntran/query_parquet_2 2021-04-14 20:33:03 +00:00
Nga Tran 4e2d59d9a5 feat: saimplement a few more functions as part of supporting query dfrom parquet files 2021-04-14 16:06:47 -04:00
Edd Robinson 025f760479 refactor: change sync::RwLock to parking_lot 2021-04-14 19:18:03 +00:00
Andrew Lamb 59ca090aef
feat: Use single db-wide executor for running queries (#1198)
* refactor: plumb executor into all Db instances

* refactor: Route all query executions through worker pool

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-14 16:46:02 +00:00
Edd Robinson 5bb34e9a97 refactor: use read_buffer column range for time range 2021-04-14 16:10:24 +00:00
Edd Robinson 04f594bf94 perf: use column_values in read buffer 2021-04-14 15:13:35 +00:00
Andrew Lamb f5f768d750
feat: Add a dedicated threadpool for running queries (#1191)
* feat: use a dedicated tokio threadpool for running queries

* feat: plumb number of executor threads through to command line

thread through command line

* fix: Logical merge conflict

* fix: another logical conflict

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-14 10:48:09 +00:00
Marco Neumann f430013141 refactor: rename `Db::load_chunk_to_object_store` to `Db::write_chunk_to_object_store
For #1083.
2021-04-14 12:37:43 +02:00
Marco Neumann 1c6690ae3b fix: fix constant use in `server::db::tests::read_from_read_buffer` 2021-04-14 12:37:43 +02:00
Marco Neumann 26ce520cf1 feat: hook RB->OS into lifecycle
Fixes #1083.
2021-04-14 12:37:43 +02:00
Marco Neumann 7b0b975433 feat: add Db::load_chunk_to_object_store_in_background
For #1083.
2021-04-14 12:37:43 +02:00
Marco Neumann 84b2ce4571 docs: extend docs of `Db::load_chunk_to_read_buffer_in_background` 2021-04-14 12:37:43 +02:00
Edd Robinson 9834c845db test: add influxrpc tag_values benches
The initial benchmarks look like this on my i9 MBP:

```
Data in one open chunk and one closed chunk of mutable buffer/tag0/no_pred           1.00     91.0±2.55ms        ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag0/with_pred         1.00     11.5±0.72ms        ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag1/no_pred           1.00    120.3±5.10ms        ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag1/with_pred         1.00     11.2±0.22ms        ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag2/no_pred           1.00    203.2±8.45ms        ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag2/with_pred         1.00     11.2±0.21ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag0/no_pred      1.00    100.3±3.73ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag0/with_pred    1.00     31.2±1.80ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag1/no_pred      1.00    126.7±2.29ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag1/with_pred    1.00     33.0±1.70ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag2/no_pred      1.00    212.0±6.86ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag2/with_pred    1.00     18.1±0.99ms        ? ?/sec
Data in single open chunk of mutable buffer/tag0/no_pred                             1.00     98.7±6.08ms        ? ?/sec
Data in single open chunk of mutable buffer/tag0/with_pred                           1.00     11.2±0.37ms        ? ?/sec
Data in single open chunk of mutable buffer/tag1/no_pred                             1.00    118.9±3.97ms        ? ?/sec
Data in single open chunk of mutable buffer/tag1/with_pred                           1.00     11.7±0.64ms        ? ?/sec
Data in single open chunk of mutable buffer/tag2/no_pred                             1.00    202.1±8.49ms        ? ?/sec
Data in single open chunk of mutable buffer/tag2/with_pred                           1.00     11.1±0.27ms        ? ?/sec
Data in two read buffer chunks/tag0/no_pred                                          1.00    109.2±5.20ms        ? ?/sec
Data in two read buffer chunks/tag0/with_pred                                        1.00     44.2±1.83ms        ? ?/sec
Data in two read buffer chunks/tag1/no_pred                                          1.00    132.9±3.79ms        ? ?/sec
Data in two read buffer chunks/tag1/with_pred                                        1.00     41.7±2.43ms        ? ?/sec
Data in two read buffer chunks/tag2/no_pred                                          1.00    222.4±7.00ms        ? ?/sec
Data in two read buffer chunks/tag2/with_pred                                        1.00     27.9±0.92ms        ? ?/sec
```
2021-04-14 09:36:39 +00:00
Edd Robinson 4db3a4b3b5 test: enable writer to split large batches 2021-04-14 09:36:39 +00:00
Edd Robinson 3cfbfe8ebf refactor: expose query_tests to benches crate 2021-04-14 09:36:39 +00:00
kodiakhq[bot] 8e0ee48018
Merge branch 'main' into ntran/query_local_parquet 2021-04-13 22:38:56 +00:00
Marko Mikulicic c9cbc74857 feat: Use a DID newtype instead of u32 for dictionary ids
Rationale
---------

We use `u32` throughout the codebase to reference for interned dictionary strings.
We also use `u32` for other reasons and it would be nice to get some help from the compiler
to avoid mixing them up
2021-04-13 22:38:41 +00:00
Nga Tran 05bf28ce85 feat: Add 2 main functions table_schema and table_names for Parquet Chunk ato pay a foundation for querying it 2021-04-13 18:23:55 -04:00
Andrew Lamb 150ed4e1d9
refactor: Remove async from `InfluxRPCPlanner` (#1200)
* refactor: Remove async from InfluxRPCPlanner

* fix: make it compile

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-13 22:17:19 +00:00
Marko Mikulicic 59c35e5a4f feat: Implement WriteBuffer gRPC 2021-04-13 21:43:54 +00:00
Raphael Taylor-Davies 61cd745ab6
refactor: remove mutable buffer predicate logic (#1186)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-13 19:09:36 +00:00
Nga Tran 4a6d6bd7ad feat: initial work for querying data from parquet file in object store 2021-04-13 13:57:46 -04:00
Paul Dix bd13c09bad refactor: make sharder optional when generating entry 2021-04-13 12:52:14 +00:00
Paul Dix 4d22982b5f fix: fix broken tests from column refactoring 2021-04-13 12:52:14 +00:00
Paul Dix 7e28f8ef66 feat: Implement Entry writing to Db
This removes the old ReplicatedWrite structure and implements the writing of an Entry to the Db. I also call out in `server/lib.rs` and in the `Db` where sharding and replication might happen.

I've also added helpers in various places to write line protocol to chunks, tables, and databases. That enabled removing a good amount of code from the test helpers crate.
2021-04-13 12:52:14 +00:00
Raphael Taylor-Davies 1997324344
feat: mutable buffer snapshotting (#1179)
* feat: mutable buffer snapshotting

* chore: review feedback
2021-04-13 12:14:54 +00:00
Nga Tran 494480bc57 fix: check filename to have deterministic output 2021-04-12 15:37:08 -04:00
Nga Tran 7f77a01e61 chore: merged main to branch and resolved conflicts 2021-04-12 12:09:04 -04:00
Nga Tran e86a02efa7 refactor: Address comments and add more tests 2021-04-12 11:46:31 -04:00
Nga Tran 453aeaf1a0 feat: Add tests for writing RB chunks to Object Store 2021-04-09 17:39:23 -04:00
Raphael Taylor-Davies 078c0f3fda
refactor: lift chunk and table summaries out of DBChunk (#1162)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-09 12:00:47 +00:00
Marko Mikulicic e76980928b feat: Implement Update API 2021-04-08 22:25:36 +00:00
Edd Robinson 738134004e feat: implement memory tracking on read buffer 2021-04-08 18:20:37 +00:00
Nga Tran bd3816a6c9 chore: merge main to branch 2021-04-08 10:59:33 -04:00
Edd Robinson d429cf9aeb refactor: tighten up Read Buffer API 2021-04-08 10:24:19 +00:00
Edd Robinson dae9f12593 refactor: remove deprecated API 2021-04-08 10:24:19 +00:00
Edd Robinson 232d6ec439 refactor: appease clippy gods 2021-04-08 10:19:11 +00:00
Edd Robinson bd21cf5d24 refactor: remove read buffer db from db 2021-04-08 10:19:11 +00:00
Edd Robinson 17853266ce refactor: add helper method for all names 2021-04-08 10:19:11 +00:00
Nga Tran c32fe3decd chore: clean up 2021-04-07 21:43:45 -04:00
Nga Tran f501a74aea refactor: Address review comments 2021-04-07 21:28:03 -04:00
Nga Tran be6e1e48e4 feat: add writer_id and object_store in Db 2021-04-07 18:36:07 -04:00
kodiakhq[bot] 319e66706e
Merge branch 'main' into cn/no-more-test-results 2021-04-07 16:45:32 +00:00
Carol (Nichols || Goulding) 82588d5c72 fix: Don't return Result from test functions 2021-04-07 12:40:00 -04:00
Raphael Taylor-Davies c2355aca6d
feat: add basic memory tracking (#1125)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-07 15:38:24 +00:00
Edd Robinson 3769ad3d21 test: update test 2021-04-07 10:46:08 +00:00
Raphael Taylor-Davies d4c090cb84
refactor: extract OnceNonZeroU32 (#1134) 2021-04-06 21:38:04 +00:00
Andrew Lamb 864f9bcd35
fix: avoid spamming the logs with errors that are not errors (#1129)
* fix: avoid spamming the logs with errors that are not errors

* fix: make it compile

* fix: change to warning, add database name

* fix: include buffer_size and soft_limit as well

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-06 20:21:56 +00:00
Raphael Taylor-Davies ef2dc952e1
feat: write once writer id (#1131)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-06 17:51:57 +00:00
Raphael Taylor-Davies 5cd1d6691d
refactor: use DatabaseName in DatabaseRules (#1127) 2021-04-06 13:26:30 +00:00
Andrew Lamb 7cc9f06e74
chore: Update arrow / datafusion deps again (#1126)
* chore: Update arrow dependencies

* test: add test for SHOW COLUMNS

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-06 12:41:41 +00:00
Raphael Taylor-Davies 9a2e636d8c
refactor: move tracking utilities into separate crate (#1124) 2021-04-06 11:43:11 +00:00
Nga Tran 6e01fbc382 feat: ause TableSummary as metadata for parquet chunk's tables and read buffer's read_filter ot get data 2021-04-05 15:37:34 -04:00
Nga Tran 59bd66a33a refactor: Merge remote-tracking branch 'origin' into ntran/write_parquet_3 2021-04-05 09:40:08 -04:00
Jacob Marble 80d55d0829 chore: rename tracing_deps to observability_deps
OpenTelemetry makes this necessary.
2021-04-02 13:14:30 -07:00
Nga Tran 4bdf8963e6 feat: continue buidling foundation for writing RB chunks to parquet files 2021-04-02 16:06:25 -04:00
kodiakhq[bot] 5a04c85da5
Merge branch 'main' into ntran/parquet_write_2 2021-04-02 14:43:46 +00:00
Carol (Nichols || Goulding) 0b880d3534 chore: Group all tracing-related crates under one crate for easier upgrade management 2021-04-02 09:54:39 -04:00
kodiakhq[bot] eba799e4e7
Merge branch 'main' into ntran/parquet_write_2 2021-04-01 21:54:19 +00:00
Andrew Lamb 839fe514a2 fix: do not make rows for empty partitions or empty tables 2021-04-01 21:22:04 +00:00
Andrew Lamb 18ca5c5292 refactor: Implement `as_str()` directly rather than using a crate 2021-04-01 21:22:04 +00:00
Andrew Lamb e52c6a8292 test: Add / update query tests 2021-04-01 21:22:04 +00:00
Raphael Taylor-Davies d0e901f503 chore: fix row_estimate to match comment 2021-04-01 21:22:04 +00:00
Raphael Taylor-Davies c0abeba792 feat: system tables 2021-04-01 21:22:04 +00:00
Nga Tran f66c67692c feat: add 2 more chunk states for writing chunks to object store 2021-04-01 17:04:08 -04:00
Nga Tran 49267114d3 chore: merge main into branch and resolve conflicts 2021-04-01 13:22:49 -04:00
Nga Tran 1463c6645f feat: Add ChunkState::ObjectStore and rename ParquetChunk to Chunk 2021-04-01 11:53:03 -04:00
Raphael Taylor-Davies 3a150594ab
refactor: push chunk and partition summary into catalog (#1103)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-01 13:34:21 +00:00
Andrew Lamb a59a6edbb8
feat: add SHOW TABLES and `select * from information_schema.columns` (#1102)
* chore: update dependencies

* test: add tests for new information schema
2021-04-01 12:50:37 +00:00
Raphael Taylor-Davies 0d599eeaf4
refactor: remove Db::writeable (#1101)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-01 11:45:40 +00:00
Raphael Taylor-Davies b0e21e5f9e
feat: automated chunk lifecycle (#1091)
* feat: automated chunk lifecycle

* chore: use >= for lifecycle timestamp comparisons

* chore: review fixes
2021-04-01 11:05:58 +00:00
Marko Mikulicic 8f69a12ec6 refactor: Add rwlock around db rules 2021-04-01 10:43:46 +00:00
Nga Tran de25dc1b53 refactor: remove stuff moved to different crate 2021-03-31 20:02:34 -04:00
Nga Tran 19a453a483 feat: finally have some framework with clear todos for writing a chunk into parquet files 2021-03-31 16:21:53 -04:00
Andrew Lamb b61875e0b2
feat: Add TableSummary calculation to ReadBuffer (#1092) 2021-03-31 16:26:37 +00:00
Andrew Lamb 569f90d937
feat: Add ability to get PartitionSummary statistics from a Db (#1090)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-31 14:18:53 +00:00
Raphael Taylor-Davies 13612047c0
feat: chunk sorting (#1077) 2021-03-31 12:30:59 +00:00
Nga Tran cd409b471f feat: continue the implementation 2021-03-30 21:31:51 -04:00
Nga Tran bccac45c75 chore: Merge remote-tracking branch 'origin' into ntran/parquet_write 2021-03-30 18:31:32 -04:00
Nga Tran 0bcd52d5c9 feat: Add more changes 2021-03-30 18:31:09 -04:00
Andrew Lamb 7154dfd5f6
feat: Add timestamps to ChunkSummary (#1079)
* refactor: Move timestamps from mutable_buffer::Chunk to catalog::Chunk

* feat: Add timestamps to ChunkSummary

* feat: Add timestamp conversion logic to protobuf types

* test: Add tests

* fix: Update data_types test

* fix: handle negative nanos during conversion

* fix: clippy

* fix: more clippy

* fix: even more clippy

* fix: even more clippy
2021-03-30 19:03:23 +00:00
Raphael Taylor-Davies d8d5b1568a
feat: automated chunk closing (#1074)
* feat: automated chunk closing

* chore: remove unused error variant

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-30 15:31:09 +00:00
Nga Tran 1a0e698571 feat: initial of write chunks to parquet files 2021-03-30 09:56:53 -04:00
Andrew Lamb f0b411cd43 feat: enable information_schema 2021-03-30 09:01:43 -04:00
Andrew Lamb 0b7dea09d5
refactor: Remove `MutableBufferDb` and Partition (#1068)
* refactor: Move test for missing tags as null to db

* refactor: move last_write_at and create_time to catalog partition

* refactor: remove mutable buffer partition and port tests

* fix: port over partition sorting logic

* fix: remove stale comment

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-30 12:51:37 +00:00
Raphael Taylor-Davies 18f3124721
refactor: remove DbCatalog newtype wrapper (#1067)
* refactor: remove DbCatalog newtype wrapper

* chore: review comments

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-30 07:52:16 +00:00
Andrew Lamb 6a48001d13
refactor: Manage storage directly in the Catalog (#1057)
* refactor: Manage mutable buffer chunks directly

* fix: do not use mutable_buffer for listing table names
2021-03-29 17:55:07 +00:00
Raphael Taylor-Davies 1a4c793270
feat: lifecycle rules (#1045) 2021-03-29 15:32:36 +00:00
Andrew Lamb eb0122655d
refactor: Remove async from PartitionChunk (#1062)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-29 13:00:36 +00:00
Andrew Lamb 02ae743e8e
refactor: Remove async from Database (#1063) 2021-03-29 12:48:12 +00:00
Raphael Taylor-Davies fb130ea99d
feat: use CatalogProvider and SchemaProvider (#1058)
* feat: use CatalogProvider and SchemaProvider

* refactor: review comments
2021-03-29 11:08:46 +00:00
Andrew Lamb 0ca9ad7285
refactor: Remove async from `PartitionChunk::table_schema` (#1060) 2021-03-27 18:08:12 +00:00
Andrew Lamb 790e5d3348
refactor: Add fine grained (object level) catalog locking (#1053)
* refactor: inline catalog crate to server

* refactor: Add fine grained (object level) catalog locking

* fix: Move mod definition and use to top of file

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-26 13:48:24 +00:00
Raphael Taylor-Davies 1b3fa7c5e8
fix: increase timing tolerance for tracker timing test (#1051) (#1052)
* fix: increase timing tolerance for tracker timing test (#1051)

* fix: fmt

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2021-03-25 15:09:09 +00:00
Raphael Taylor-Davies cda7d13110
refactor: unify serialization of database rules (#973) (#1047)
* refactor: unify serialization of database rules (#973)

* chore: fix test

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-25 14:13:14 +00:00
Andrew Lamb 4cbf7c6ae5
refactor: Make `Db::mutable_buffer_chunks` and `Db::read_buffer_chunks` private (#1044)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-24 20:57:15 +00:00
Andrew Lamb 44d67db733
feat: Rework Db to use Catalog for chunk state (#1043)
* feat: Rework Db to use Catalog for chunk state

* docs: Update server/src/db.rs

* fix: fmt

* fix: fmt

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-24 17:57:11 +00:00
Raphael Taylor-Davies 360df39d4c
feat: per-Db background tasks (#1032)
* feat: per-Db background tasks (#1007)

* chore: review comments

* fix: ensure tracker registry is drained on shutdown

* chore: formatting

* chore: further review fixes

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-24 09:43:52 +00:00
Raphael Taylor-Davies d4eef65f2a
feat: share job registry with Db struct (#1038) 2021-03-23 12:34:55 +00:00
Raphael Taylor-Davies 1cbfea7096
refactor: don't implement serialize for Db (#1037) 2021-03-23 11:36:46 +00:00
Nga Tran 9e9387352e
Merge branch 'main' into ntran/unsigned_tests 2021-03-22 16:26:08 -04:00
Nga Tran 6a2e5d69f1 chore: e2e test for unsigned types 2021-03-22 15:50:42 -04:00
Carol (Nichols || Goulding) c47a252c15 feat: Check in generated WAL flatbuffers code 2021-03-22 09:38:59 -04:00
Carol (Nichols || Goulding) 52e415ae1b fix: Verify flatbuffers when restoring from a file, and not after that 2021-03-22 09:38:59 -04:00
Carol (Nichols || Goulding) 24af745b23 chore: Upgrade to flatbuffers 0.8 2021-03-22 09:38:58 -04:00
Andrew Lamb 6e1795fda0
refactor: Move some types (not yet exposed to clients) into internal_types (#1015)
* refactor: Move some types (not yet exposed to clients) into internal_types

* docs: Add README.md explaining the rationale

* refactor: remove some stragglers

* fix: fix benches

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: add clippy lints

* fix: fmt

* docs: Apply suggestions from code review

fix typos

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-19 16:27:57 +00:00
Raphael Taylor-Davies 7e6c6d67b4
feat: graceful shutdown (#827) (#1018)
* feat: graceful shutdown (#827)

* chore: additional docs

* chore: further docs

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-19 15:36:49 +00:00
Raphael Taylor-Davies dd94a33bc7
feat: retain limited tracker history (#1005) 2021-03-17 16:32:34 +00:00
Andrew Lamb 3a53923684
feat: Management API + CLI command to close a chunk and move to read buffer (#1002)
* feat: Management API + CLI command to close a  chunk and move to read buffer

* refactor: Less copy-pasta

* fix: track only once, use `let _` instead of `.ok()`

* docs: Apply suggestions from code review

fix comments ( 🤦‍♀️  for copy/pasta)

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* docs: Update server/src/lib.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* refactor: Use DatabaseName rather than impl Into<String>

* fix: Fixup logical merge conflicts

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-17 15:25:27 +00:00
Raphael Taylor-Davies 1fee56274b
fix: prevent observing inconsistent tracker states (#994)
* fix: prevent observing inconsistent tracker states

* feat: further tracker docs

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-17 14:28:54 +00:00
Raphael Taylor-Davies 4b98d19d70
refactor: remove interior mutability from TrackerRegistry (#1004)
* refactor: remove interior mutability from TrackerRegistry

* fix: don't hold mutex across await point

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-17 12:58:27 +00:00
Andrew Lamb 2f77090ecf fix: clippy/fmt 2021-03-16 18:16:48 -04:00
Andrew Lamb ef9c3f3d8f
feat: Management API + CLI command to list chunks in a partition (#995)
* feat: Management API + CLI command to list chunks in a partition

For ease of use

* refactor: remove unecessary Result
2021-03-16 20:10:55 +00:00
Andrew Lamb 4781317647
feat: Management API + CLI command to create a new chunk (rollover partition) (#990)
* feat: Management API + CLI command to create a new chunk (rollover partition)

* fix: Update tests/end_to_end_cases/management_api.rs

fix typo

* fix: logical merge conflict

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-15 16:41:18 +00:00
Raphael Taylor-Davies 65f7a1ac5b
fix: use consistent crate versions (#989)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-15 15:42:19 +00:00
Raphael Taylor-Davies 7e25c4e896
feat: add fanout task tracking (#956)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-12 15:01:27 +00:00
Andrew Lamb 6ac7e2c1a7
feat: Add management API and CLI to list chunks (#968)
* feat: Add management API and CLI to list chunks

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: add comment to protobuf

* fix: fix comment

* fix: fmt, fixup merge errors

* fix: fascinating type dance with prost generated types

* fix: clippy

* fix: move command to influxdb_iox database chunk list

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-12 13:56:14 +00:00
Raphael Taylor-Davies 0ff527285c
refactor: remove unnecessary async from DatabaseStore trait (#965)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-11 11:33:53 +00:00
Marko Mikulicic 1b40ca7ab5 feat: Implement CRUD for remote addresses 2021-03-10 13:39:59 +00:00
Paul Dix 706115178b
chore: remove cruft from config/database rules (#935)
The replication, query, and subscription concepts here are going to be signficiantly different. Thought it would be best to just remove this cruft for now to avoid confusion.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-09 13:48:59 +00:00
Andrew Lamb a6965769b4
refactor: Remove impl of `query::PartititionChunk` in mutable_buffer: This PR (#923) 2021-03-05 10:51:08 +00:00
Andrew Lamb b1e5cfedf7
refactor: Move chunk predicate creation from query::Predicate into server crate (#922)
* refactor: Create `ChunkPredicate` predicate in the Server crate

* refactor: move chunk predicate out of chunk

* fix: remove comment in server/src/db/pred.rs

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-04 23:23:43 +00:00
Andrew Lamb 3be5c26f92
refactor: Remove impl of `query::Database` in mutable_buffer (#914) 2021-03-04 22:02:42 +00:00
Andrew Lamb 8b1f100df3
feat: make read_group and read_window_aggregate work across chunks (#905)
* feat: make read_group and read_window_aggregate work across chunks

* refactor: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* refactor: Update query/src/frontend/influxrpc.rs

Improve logic and use strings directly

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: fmt

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-04 17:06:31 +00:00
Andrew Lamb c23328d57d
refactor: Use `vec![]` instead of inlined strings, for legibility, in tests (#911)
* refactor: Use vec![] instead of inlined strings, for legibility

* refactor: use standardized names
2021-03-03 15:16:37 +00:00
Andrew Lamb 94bd200e60
refactor: Add Predicate::is_empty() and EMPTY_PREDICATE to avoid unecessary construction (#891)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-01 21:03:05 +00:00
Andrew Lamb 7d8d00781c
feat: Make read_filter work for mutable buffer and read buffer (#882)
* feat: port read_filter to InfluxRPCPlanner

* fix: remove commented out vestigal test

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: fmt

* fix: Update arrow_deps/src/util.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-01 16:50:29 +00:00
Nga Tran 18de3bdcab chore: merge main into branch
Merge branch 'main' into ntran/optimize_column_selection
2021-02-26 15:29:43 -05:00
Nga Tran f37e5846aa feat: fmt auto fix 2021-02-26 14:56:10 -05:00
NGA TRAN eb81975151 feat: Optimize Column Selection 2021-02-26 14:28:46 -05:00
Andrew Lamb 12deacd8a0
refactor: move SeriesSetPlans into its own module (#878)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-25 23:12:39 +00:00
Andrew Lamb 8fb7651719
feat: Port tag_values to the InfluxRPCPlanner (#859)
* feat: Port tag_values to the InfluxRPCPlanner

* refactor: merge imports

* refactor: rename column_names to tag_column_names for clarity

* fix: Update query/src/frontend/influxrpc.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* refactor: use ensure!

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* refactor: less silly whitespace

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: code review comments

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-24 23:11:22 +00:00
Andrew Lamb ed7859e182
refactor: rename tag_column_names --> tag_keys in InfluxRPCPlanner (#860) 2021-02-23 17:04:53 +00:00
Carol (Nichols || Goulding) 0aee605544 Merge remote-tracking branch 'origin/main' into pd-mutable-buffer-data-eviction 2021-02-22 14:13:34 -05:00
Jake Goulding 6e6cc616a0 refactor: Switch to parking_lot::Mutex 2021-02-22 13:51:31 -05:00
Carol (Nichols || Goulding) c6becbd544 fix: Use structured log output for partition dropping 2021-02-22 12:46:48 -05:00
Carol (Nichols || Goulding) 45f4855d2e fix: Sort data in tests to avoid spurious failures
The implementation of this function is ultimately the keys from a
HashMap, and that order isn't guaranteed. Not sure if the function needs
a stable ordering, so I didn't want to add the performance hit to the
function. Adding it to the test ensures it keeps passing (I saw it fail
a few times locally).
2021-02-22 12:34:08 -05:00
Carol (Nichols || Goulding) eec13912dd fix: Update test that calls a fn that's no longe async 2021-02-22 11:06:54 -05:00
Edd Robinson cb335810e2 refactor: use new column_name API 2021-02-22 15:32:55 +00:00
Edd Robinson 4caa1d732f refactor: make Selection Copy 2021-02-22 15:32:55 +00:00
Carol (Nichols || Goulding) d0707725cf Merge remote-tracking branch 'origin/main' into pd-mutable-buffer-data-eviction 2021-02-22 10:21:59 -05:00
Andrew Lamb aa09cd3579
fix: remove workaround query code (#842)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-22 13:29:27 +00:00
kodiakhq[bot] fc0c79a84a
Merge branch 'main' into er/feat/read_buffer/string_fields 2021-02-19 23:10:07 +00:00
Edd Robinson 92eb8b9e85 refactor: make certain Database method sync
A couple of methods don't seem to have any await points in their
implementations, so it feels like they could just be `sync`.
2021-02-19 17:14:17 +00:00
Marko Mikulicic 536c1724bd feat: Allow to put streams of unknown length to objectstore
Addresses the API aspect of #818

Adds a utility module that helps computing the length of a stream while buffering it
for later replay (in-memory or spilling it in a temporary file).
2021-02-18 16:49:18 +00:00
Edd Robinson 58bb79eef2 test: enable scenario 2021-02-17 21:17:56 +00:00
Andrew Lamb 9b91e0624c
feat: implement field_columns plan (#819)
* feat: implement field_columns plan

* fix: fix doc tests

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-17 20:43:24 +00:00
Edd Robinson b705baa223 refactor: apply clippy clone lint 2021-02-16 13:34:56 +00:00
Andrew Lamb 94a93e56ff
feat: implement `tag_keys` in gRPC planner and across mutable buffer (#795)
* feat: move tag_column_names into rpc planner

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: compile error

* refactor: remove PassThrough error type

* fix: Avoid extra layers of errors in mutable buffer chunk

* fix: use HashMap::get rather than values() and find

* fix: push filtering down to chunk in gRPC planner

* fix: fixup trait bounds to be non-silly

* fix: remove incorrect comment

* fix: remove cruft

* fix: clippy + fmt

* fix: correct comment

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-02-15 16:47:52 +00:00
Paul Dix dc465e5d02 feat: Add function to check db size and drop partitions
Adds functionality to the server Db to check the mutable buffer size and drop partitions based on the database rules.
2021-02-13 17:19:40 -06:00
Andrew Lamb b8f85967dd
feat: Enable/Disable logging in tests via RUST_LOG environment variable (#793)
* feat: Enable/Disable logging in tests via RUST_LOG environment variable

* docs: Add section to contributing

* docs: tweak readme

* fix: Use same logging system in tests as in influxdb_ioxd
2021-02-12 13:43:12 +00:00
Andrew Lamb a03598dfe2
feat: Implement Cross Chunk Schema / RecordBatch merging at query time (#783)
* feat: feat: Implement Cross Chunk Schema / RecordBatch merging at query time

* docs: update comments about NullArray::new_with-type

* docs: Update comments based on code review

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-11 18:26:38 +00:00
Raphael Taylor-Davies 7debe94ee6 feat: add background task tracking (#655) 2021-02-11 10:30:19 +00:00
Paul Dix 4c43c311f0
feat: Add database rules configuration for mutable buffer (#779)
* feat: Add database rules configuration for mutable buffer

* refactor: change all database rules usage to use mutable_buffer_config rather than store_locally
2021-02-10 18:36:29 -05:00
Andrew Lamb 46dfd58e6e
test: add a SQL test case for selection (of columns) (#778)
* test: add a SQL test case for selection (of columns)

* test: test with selection + pred
2021-02-10 12:09:19 -05:00
Andrew Lamb a316b16960
feat: Change table_names to return either Some(set) or None, rather than a plan (try 2) (#776)
* feat: Change table_names to return either Some(set) or None, rather than a plan

* docs: improve comments

* docs: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: merge conflict

* fix: don't clone a string unless needed

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-02-09 12:20:59 -05:00
Raphael Taylor-Davies 143488fae9 feat: add WAL metadata endpoint (#724) 2021-02-08 16:21:34 +00:00
Andrew Lamb 8399c56587
feat: remove RwLock on entire ReadBuffer (#761) 2021-02-05 16:58:17 -05:00
Andrew Lamb 74b8e8ec7d
refactor: Break Database query tests into their own files + add basic SQL tests (#760)
* refactor: break query tests into separate modules

* test: Add basic sql tests
2021-02-05 16:04:41 -05:00
Paul Dix 47bc28460e
refactor: rename partition, table, and column in parition_meta for clarity (#757)
* refactor: rename partition, table, and column in parition_meta for clarity
2021-02-05 08:00:22 -05:00
Andrew Lamb 3ec483b769
refactor: Reduce async in mutable buffer, use std::sync (#749)
* refactor: Reduce async in mutable buffer, use std::sync

* fix: logical confict with new code
2021-02-05 06:47:40 -05:00
Carol (Nichols || Goulding) fbf776c6b3
chore: Clean up Cargo.tomls (#754)
* fix: test_helpers crate should only be a dev-dep

* fix: object_store no longer has a build script, so no longer needs a build dep

* chore: Alphabetize all Cargo.tomls
2021-02-04 18:56:02 -05:00
Marko Mikulicic 202a4187a1
feat: Implement list databases API
Ref #722
2021-02-04 16:50:40 +01:00
Andrew Lamb d66eae1a44
feat: Implement TableProvider for Trait for `Db` (#730)
* feat: Implement TableProvider for Db

Gets us selection pushdown in plans, sets us up for predicate pushdown

Includes: SendableRecordBatchStreams for mutable buffer and read buffer results

fixup snapshots

* docs: comments
2021-02-03 14:18:47 -05:00
Carol (Nichols || Goulding) 8975b1cbf5 fix: into_iter in the assert_table_eq macro
This makes it so the macro can take `&[String]` or `&[&str]`.
2021-02-03 09:56:56 -05:00
Carol (Nichols || Goulding) 5c8b351f57 fix: Address clippy suggestions 2021-02-01 14:56:49 -05:00
Carol (Nichols || Goulding) ff6955a433 refactor: Extract a trait for ObjectStoreApi with associated path
This is the promised cleanup. This structure gets rid of a lot of
intermediate structures and encodes through associated types how the
object stores and path types are related.

The enums are still necessary to avoid having generics leak all over
the place, but the object store variants and path variants should always
match because they'll always come from the object store trait
implementations that use the associated types.
2021-02-01 14:56:47 -05:00
Carol (Nichols || Goulding) 7d3b4db234 fix: InMemory doesn't need pagination 2021-02-01 14:35:47 -05:00
Carol (Nichols || Goulding) fdbe602e57 refactor: Always get a path to build from the object store 2021-02-01 14:30:21 -05:00
Carol (Nichols || Goulding) 5e74ae2ee7 refactor: Segments should know how to persist themselves to object storage 2021-02-01 14:30:20 -05:00
Andrew Lamb 288861e646
feat: implement table_schema in partition chunk, mutable buffer, read buffer (#705)
fix: sort output schema by name

fix: Update data_types/src/schema.rs

Co-authored-by: Edd Robinson <me@edd.io>

refactor: Update read_buffer/src/lib.rs

Co-authored-by: Edd Robinson <me@edd.io>

Co-authored-by: Edd Robinson <me@edd.io>
2021-02-01 13:54:58 -05:00
Andrew Lamb f3bd8bd0e3
chore: update deps (tokio 1.0 and ecosystem) (#707)
* chore: Update arrow + tokio deps

* chore: Use bleeding edge azure

* chore: Update aws + other deps

* fix: fmt

* fix: Switch to in-house version of routerify

* fix: Upgrade to hyper 0.14

The hyper::error module is now private; hyper::Error is the public
re-export

* fix: Upgrade cloud storage to get tokio upgrade

* fix: Upgrade open_telemetry

* fix: Do not call `panic::set_hook` during another panic

Doing so leads to a double panic which aborts the process.

* fix: new h2 error who dis

Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@integer32.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2021-01-29 16:11:55 -05:00
Andrew Lamb 2282a68e65
refactor: Move selection to the data_types crate and remove redundant implemenation (#704) 2021-01-29 13:35:07 -05:00
Andrew Lamb efb1e0f8ae
feat: Add selection interface to mutable buffer and query interface (#700)
* feat: Add selection interface to mutable buffer and query interface

* docs: Update mutable_buffer/src/table.rs

* refactor: rename for consistency

* refactor: use map and filter_map  rather than fold
2021-01-27 14:31:10 -05:00
Andrew Lamb df1367bcd1
feat: gRPC measurement_names support for read buffer (#697)
* test: Add tests for reading data from read buffer in different scenarios

* feat: gRPC measurement_names support for read buffer
2021-01-26 13:15:56 -05:00
Andrew Lamb 504ca67532
test: revamp rpc query testing so it works in multiple chunk scenarios (#696)
* test: revamp testing so it works in multiple scenarios, fix bug found by same

* fix: Update docs in server/src/db.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* refactor: use tsp rather than different functions

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-01-25 16:34:19 -05:00
Carol (Nichols || Goulding) f7fdadc3b3
Merge branch 'main' into pd-load-database-configs 2021-01-25 09:50:16 -05:00
Andrew Lamb c3b0371c84
feat: Initial RPC Query Frontend (#692)
* feat: Initial RPC Query Frontend

* docs: s/immutable buffer/mutable buffer

* docs: Correct type in docstring
2021-01-25 08:33:39 -05:00
Paul Dix b3454daf70 chore: pr fixup 2021-01-22 16:06:10 -05:00
Paul Dix 45d25fcbcd feat: add loading of database rules on startup
This adds functionality to the server to load database rules on startup. Follow on work will update the rules to store additional data (the catalog) and ensure that updates to the catalog can occur as outlined in #651. This work also updated the configuration to not require a database directory so the server can run entirely in memory. I needed this to get the end-to-end test passing since the file object store API doesn't yet have the functionality needed. I've logged #688 to track adding that in.
2021-01-22 16:06:10 -05:00
Andrew Lamb 75b0a62fa5
refactor: Delete remove dead code (#686) 2021-01-21 19:20:39 -05:00
Andrew Lamb a967e2f1dd
fix: disallow control characters in Database names (#684) 2021-01-21 17:55:55 -05:00
Carol (Nichols || Goulding) 6bb91653c1
refactor: Some tiny cleanups (#680)
* refactor: Remove import of unimplemented macro that's in the prelude

* refactor: Remove allowing of dead code that isn't dead anymore

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-01-21 07:27:17 -05:00
Andrew Lamb 7969808f09
feat: Chunk Migration APIs and query data in the read buffer via SQL (#668)
* feat: Chunk Migration APIs and query data in the read buffer via SQL

* fix: Make code more consistent

* fix: fmt / clippy

* chore: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* refactor: Remove unecessary Result and make chunks() infallable

* chore: Apply more suggestions from code review

Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Edd Robinson <me@edd.io>
2021-01-19 13:28:26 -05:00
Paul Dix a4848d5e8b
feat: Add persistence of database rules (#671)
This pulls the server configuration into its own module. It updates the config so that databases can be created and saved without holding a write lock across an async call. Follow on work will add reading of these rules from object store on server initialization.
2021-01-19 10:31:04 -05:00
Carol (Nichols || Goulding) ac22d2bb44 refactor: Extract a cloud module to deal with cloud paths 2021-01-15 15:24:49 -05:00
Andrew Lamb 628405bd53
refactor: move db predicate conversion to its own module (#664) 2021-01-15 13:21:00 -05:00
Edd Robinson bdb8a78569 refactor: reduce expose API 2021-01-14 21:26:50 +00:00
Edd Robinson d7073efa72 feat: add support for query pred -> read buffer pred 2021-01-14 13:46:17 +00:00
Paul Dix e53a65b49a chore: fix failing tests after rebase 2021-01-13 18:15:35 -05:00
Paul Dix 9377dc1943 chore: address pr feedback 2021-01-13 18:15:35 -05:00
Paul Dix 39cc833ec7 chore: change wal buffer to use sync mutex instead of tokio 2021-01-13 18:15:35 -05:00
Paul Dix 26c4003e43 feat: add wal segment persistence to object store
This hooks the wal buffer up to the server. When segments go over their max size, they will be persisted if that configuration was set. A follow on commit will add persistence based on time (how long the segment has been open).
2021-01-13 18:15:35 -05:00
Paul Dix 3b1f045f44 feat: add serialization to wal buffer segments
Adds serialization with compression and checksum for WAL buffer segments.

This required a weird structure where the flatbuffer bytes of ReplicatedWrite were kept as a raw payload. I did this because otherwise each of the replicated writes would have been rebuilt in the segment.

The other thing that isn't ideal is that deserializing a segment actually marshals it into a Rust struct as opposed to keeping the entire thing as raw flatbuffers. We could update this later to have a concept of an open segment (regular rust stuct) and closed segments that are just the flatbuffers.
2021-01-13 18:15:34 -05:00
Paul Dix 29444ad06b feat: add function to get segment object store location 2021-01-13 18:15:34 -05:00
Carol (Nichols || Goulding) 6b67498533
Merge branch 'main' into cn/better-osp-api 2021-01-13 15:05:08 -05:00
Andrew Lamb 6376891da3
feat: implement query planning in terms of chunks (#647) 2021-01-12 16:04:45 -05:00
Dom 1a1a14308d feat: API set writer ID endpoint
Adds a new endpoint /iox/api/v1/id that accepts a JSON object in the form:

    {"id":42}

And calls into the server's set_id method to assign the writer ID to the server.
2021-01-12 15:16:59 +00:00
Dom e60906c5d5 refactor: remove async from ID methods
Both set / get ID methods are not async, so this removes the annotation &
awaits.
2021-01-12 15:01:51 +00:00
Dom e38df076b4
fix: do not overwrite databases (#644)
* fix: do not overwrite databases

Do not overwrite an existing database when attempting to create a DB with an
existing name.

This should probably update the existing database config, but for the time being
this prevents the existing database from being silently dropped.

Fixes #643

* test: ensure duplicate DB names returns an error

Covers #643 with a test case.
2021-01-12 08:55:29 -05:00
Carol (Nichols || Goulding) 06f1358e2d feat: Change ObjectStorePath API to be more explicit
Now you have to designate whether you're adding a directory or a file
name, with some assumptions based on paths coming from a cloud object
storage or the file system.

A notable difference: checking to see if "apple/b" is a prefix of
"apple/bear/cow.json" will now say no; only whole directories are
matched.
2021-01-11 16:57:37 -05:00
Andrew Lamb fd28d8a01b
refactor: Use u32 for Chunk ids consistently (#639) 2021-01-11 16:07:22 -05:00
Carol (Nichols || Goulding) 2be992b2f6
Merge branch 'main' into cn/fewer-servers 2021-01-08 12:41:48 -05:00
Andrew Lamb 6d0c538eca
refactor: pull DBChunk into its own module (#623)
* refactor: pull DBChunk into its own module

* refactor: consolidate impl blocks
2021-01-08 12:27:21 -05:00
Carol (Nichols || Goulding) cd03f39280 refactor: Move code in the server::server module to the root
This gets rid of the redundant server::server imports.
2021-01-08 12:19:58 -05:00
Andrew Lamb a4be6f74c7
refactor: Remove partition key from the Chunk trait (#622) 2021-01-08 06:11:07 -05:00
Andrew Lamb 8219403fab
feat: Instantiate ReadBuffer as part of server creation (#620)
* feat: Instantiate ReadBuffer as part of server creation

* refactor: remove Store from read_buffer
2021-01-07 13:25:42 -05:00
Andrew Lamb c672bb341d
feat: Extract SQL planning out of databases (#618) 2021-01-07 13:13:30 -05:00
Paul Dix d17ef800c5
feat: add create and get database to API (#619)
* feat: add create and get database to API

This commit is start of the IOx specific API. It puts everything under /iox/api/v1 as this is the beginning of the IOx API. Creating a database is done with a PUT and a GET request can retrieve the DatabaseRules details.

* feat: add defaults for DatabaseRules for create_database

* feat: add create and get database to API

This commit is start of the IOx specific API. It puts everything under /iox/api/v1 as this is the beginning of the IOx API. Creating a database is done with a PUT and a GET request can retrieve the DatabaseRules details.
2021-01-07 12:25:37 -05:00
Carol (Nichols || Goulding) 18ee1b561b feat: Use ObjectStorePath everywhere to feel out the API needed 2021-01-07 10:48:22 -05:00
Paul Dix d1ab5c0ee9 chore: refactor object_store crate
This pulls the different backing implmenetations into their own modules. They're about to get more complex so it felt like it was time to separate them out rather than building towards a single multi-thousand line lib.rs. The error type is only defined in lib and imported by the individual modules, which I think makes it easier to work with.
2021-01-07 09:19:07 -05:00
Andrew Lamb 654b520005
feat: Interface for writing and querying mutable buffer, read buffer and parquet (#615)
* refactor: Create database with mutable buffer, read buffer and parquet files

* docs: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: rename planners to clarify what they are

* refactor: simplify traits

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-01-06 17:25:46 -05:00
Andrew Lamb 08d52ea043
feat: implement partition chunk rollover + ids and timestamps (#601)
* feat: implement partition chunk rollover + ids and timestamps

* feat: add last_write_timestamp

* refactor: Use DateTime<Utc> rather than Instant

* refactor: avoid use of structure to generate ids
2020-12-29 11:00:18 -05:00
Andrew Lamb 0d0ec0ce69
chore: Upgrade arrow dependencies (#603)
* chore: Update arrow dependencies to latest

* refactor: Update code to conform to new arrow api
2020-12-28 16:08:09 -05:00
Andrew Lamb 5fa77c32cc
feat: Add "Chunks" to the Mutable Buffer (#596)
* refactor: Update docs, remove unused field

* refactor: rename partition -> chunk

* feat: Introduce new partition, which is a holder for Chunks

* refactor: Remove use of wal from mutable database

* refactor: cleanups, remove last direct use of chunks

* fix: delete old benchmarks

* fix: clippy sacrifice

* docs: tidy up comments

* refactor: remove unused error types

* chore: remove commented out tests
2020-12-28 07:10:25 -05:00
Paul Dix 1d200c5c77 chore: move http API over to Routerify
This moves the HTTP API over to Routerify, which has the basic route parsing logic that will enable the API design for IOx.

I had a little trouble with the error handling in Routerify so I ended up creating a macro for constructing error responses in the HTTP API. I'm not sure what I think of this pattern so I'm interested in what others think. Another option would be to have two functions for each API endpoint. One which is x_handler with a Routerify function signature. Then another which is just x that has the Result<Response<Body>, ApplicationError> return type, which would make using the ? operator work in those functions. That would eliminate the need for the return_err macro.

I'm happy to refactor to that if people prefer it.
2020-12-24 16:45:20 -06:00
Andrew Lamb 48c43b136c
refactor: rename write_buffer --> mutable_buffer (#595)
* refactor: git mv write_buffer mutable_buffer

* refactor: update crate name references

* refactor: update some more references
2020-12-22 10:49:53 -05:00
Dom 558cf6fe1a perf: avoid locking to load server ID
Moves the server ID value outside of the mutex-wrapped `Config` type and
into the `Server` struct, synchronising threads using an atomic instead.

This prevents any calls to `require_id()` from having to contend for a
read lock on the config mutex while maintaining lineraisaion of ID
values - all threads see the change as soon as it is set.

On x86 this synchronisation is actually completely "free" - memory
accesses are totally ordered, so this turns into a compile fence to
prevent LLVM changing program order at compile time with no runtime
overhead at all.

Previously the ID value was wrote as part of the config, but this is no
longer the case. It should be easy to add back in, but it appears to be
redundant as the ID is required to construct the path to the config when
loading in the first place.
2020-12-18 11:12:41 +00:00
Paul Dix 70ff4267ce
chore: Connect server crate to http routes and server command (#569)
* chore: Connect server crate to http routes and server command

This updates the http_routes and main server to use the Server crate. This is the first step in a larger effort to start hooking up the initial IOx API and get things running end to end with in-memory database, WAL buffer, and object storage.

For the time being, this disables the previous disk based WAL. Or rather, it uses the WriteBufferDb without it. That means that this IOx server has no persistence until later. Because of this, the restart in the end-to-end was removed.

Later PRs will add the WAL buffer and restart logic that loads from object store. We can opt to bring the local disk based WAL back later, but it will likely require some refactoring to work with how the WAL Buffer will operate.
2020-12-17 18:48:41 -05:00
Dom 6f473984d0 style: wrap comments
Runs rustfmt with the new config.
2020-12-11 18:22:26 +00:00
Paul Dix fa3ecbd4ed
feat: Implement write buffer to Parquet snapshotting (#526)
* feat: Implement write buffer to Parquet snapshotting

This introduces snapshot to the server packages to manage snapshotting. It also introduces a new trait for representing a Partition. There is a very crude API wired up in http_routes for testing purposes. Follow on work will bring the server package into http_routes and rework the snapshot API.
2020-12-08 14:20:43 -05:00
Dom 87573256a7 chore: fmt 2020-12-03 16:10:16 +00:00
Dom 13f391e2b9 refactor: ignore destructured fields
I temporarily forgot I can do this.

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-12-03 16:10:16 +00:00
Dom 234df612ec refactor: avoid clones for errors
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-12-03 16:10:16 +00:00
Dom b03de0e7ef refactor: remove needless lifetimes 2020-12-03 16:10:15 +00:00
Dom f90a95fd80 fix: unambigious bucket/org to DB mappings
Previosuly the $ORG and $BUCKET was joined as:

	$ORG + "_" + $BUCKET

Which is fine unless either $ORG or $BUCKET includes a "_", such as:

	$ORG = "org_a"
	$BUCKET = "bucket"

	and

	$ORG = "org"
	$BUCKET = "a_bucket"

This change continues to join $ORG and $BUCKET with an underscore, but
disallows underscores in either $ORG or $BUCKET. It appears these values
are non-zero u64s in the gRPC protocol converted to their base-10 string
representations for the DB name, so this seems safe to enforce.

In addition, this change introduces a `DatabaseName` type to avoid
passing bare strings around, and allow consuming code to ensure only
valid database names are provided at compile type. This type works with
both owned & borrowed content so doesn't force a string copy where we
can avoid it, and derefs to `str` to make it easier to use with existing
code.

I've been minimally invasive in pushing the `DatabaseName` through the
existing code and figured I'd see what the sentement is first.
Candidates for conversion from `str` to `DatabaseName` that seem to make
sense to me include:

	- `DatabaseStore` trait
	- `RemoteServer` trait
	- Others? Basically anywhere other than the "edge" API inputs

Fixes #436 (thanks @zeebo)
2020-12-03 16:10:15 +00:00
Andrew Lamb ecc4eee8e1
refactor: Move SQL functions into is own trait (#511)
* refactor: remove uneeded function table_to_arrow from Trait

* refactor: Move SQL functions into is own trait
2020-12-02 08:23:37 -05:00
Andrew Lamb 5ef499bb63
refactor: rename Database --> TSDatabase to better reflect its purpose (#510)
* refactor: rename Database --> TSDatabase to better reflect its purpose

* refactor: rename field_columns to field_column_names

* fix: clippy?
2020-12-01 12:37:11 -05:00
Dom 37fea83ca9 test: add newline test case 2020-11-30 11:57:48 +00:00
Dom f08846ca50 chore: validate database names
Ensure database names contain only {alphanumeric, -, _} characters.

Fixes #278.
2020-11-30 11:27:33 +00:00
Matt Freitas-Stavola 7e2df1fc59
chore(server): add logs for dropped WAL segments (#478)
* chore(server): add logs for dropped WAL segments

Added logging for dropped writes and old segments in rollover scenarios

Also including a dep on tracing and dev-dep on test_helpers

Refs: #466

* chore(server): Add more context to logs

Minor cleanup around remove_oldest_segment usage

Suggestions from @alamb's review
2020-11-24 16:37:09 -05:00
Andrew Lamb cdb26e60e4
refactor: rename `storage` crate to `query` to better reflect what it is (#475)
* refactor: rename storage --> query

* refactor: update a few more referenes
2020-11-24 14:19:29 -05:00
Paul Dix 59e2024d01 chore: rename cluster crate to server 2020-11-20 12:14:12 -05:00