Commit Graph

493 Commits (be9b3a48534e2292d9fc26c4fd1c88c1db72254f)

Author SHA1 Message Date
Andrew Lamb 27e5b8fabf
refactor: Remove multiple table support from Parquet Chunk (#1541) 2021-05-24 08:40:31 -04:00
Nga Tran 1f70d1f9c8 chore: remove a couple more comments 2021-05-21 17:06:53 -04:00
Nga Tran f113abacb5 feat: more unit & e2e tests plus cleanup and addressing review comments of Andrew and Edd 2021-05-21 16:48:43 -04:00
Nga Tran 1093542578 fix: now all tests pass. Next step is cleaning up and addressing review comments 2021-05-21 13:29:20 -04:00
Nga Tran 784ef88fcd chore: merge main to branch and add more tests that expose a wrong result bug on unsigned int 2021-05-21 12:38:06 -04:00
Nga Tran 93afc9c213 chore: more tests 2021-05-21 11:39:12 -04:00
Raphael Taylor-Davies 5b619733d9
refactor: split lifecycle tracking from chunk state (#1361) (#1099) (#1397)
* refactor: split lifecycle tracking from chunk state (#1361) (#1099)

* chore: namespace internal errors

* chore: fix logical conflict

* chore: don't remove moving chunk size metric
2021-05-21 09:27:44 +00:00
Nga Tran e44a3a87db feat: fnow predicate is actuallu pushed down to RUB but there are bugs and not working yet 2021-05-20 16:56:15 -04:00
kodiakhq[bot] f028a356f4
Merge branch 'main' into crepererum/issue1382-c 2021-05-20 15:51:47 +00:00
kodiakhq[bot] aac00d4fa6
Merge branch 'main' into crepererum/remove_snapshotting 2021-05-20 14:14:58 +00:00
Marco Neumann 0e37d500eb feat: remove snapshot feature
The parquet files produced by this code path are only semi-specified and
will miss many important metadata aspects that we will require for data
lineage.
2021-05-20 14:59:04 +02:00
Marko Mikulicic 462a5590c6
fix: fmt 2021-05-20 14:58:50 +02:00
Marko Mikulicic c908cf0f98
fix: review suggestion
Co-authored-by: Edd Robinson <me@edd.io>
2021-05-20 14:40:02 +02:00
Marko Mikulicic aa90329c1f
feat: Add remote_template for simpler remote configuration 2021-05-20 12:45:08 +02:00
Marco Neumann 7e55544eef fix: correctly track chunk ID counter during catalog replay 2021-05-20 10:32:40 +02:00
Marco Neumann 93251f22c7 feat: read perserved catalog during DB startup
Closes #1382.
2021-05-20 10:28:31 +02:00
Marko Mikulicic 91d7189e6d
feat: Log cached connections 2021-05-20 10:27:20 +02:00
Raphael Taylor-Davies 37880ee89a
refactor: store chunk IDs only in catalog (#1521)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-20 04:07:14 +00:00
Nga Tran 00dacb5394 feat: add tests to verify the correctness as well as the explain of the plan 2021-05-19 17:31:16 -04:00
Nga Tran 11561111d5 chore: merge main to branch 2021-05-19 15:11:15 -04:00
Nga Tran 087d61f229 feat: Part 1 of predicate push down - Send predicates to MUB, RUB, and Parquet File. Note that MUB has not handled predicates yet 2021-05-19 13:59:51 -04:00
Marko Mikulicic ce2f8351be
fix: Cache outbound gRPC connections 2021-05-19 18:28:45 +02:00
Marco Neumann 8db26485a4 refactor: empty transaction during catalog creation
That involves some refactoring which we are going to need anyway for
hooking up the "read" path of the catalog into the DB startup, namely:

- make `Db::new` require a preserved catalog
- introduce a helper function that can provide that
- as a consequence, all test-creations of a Db are now async

This prepares for #1382.
2021-05-18 17:42:07 +02:00
kodiakhq[bot] c3cc58b2ff
Merge branch 'main' into crepererum/issue1382 2021-05-17 17:57:26 +00:00
Raphael Taylor-Davies 4f0e46bcd5
refactor: track ingest metrics in one place (#1503)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-17 16:25:01 +00:00
Marco Neumann 18f0a7f614 docs: reference open issue 2021-05-17 14:01:51 +02:00
Marco Neumann cdf0ada6a6 test: test preserved catalog <-> Db write wiring 2021-05-17 13:57:31 +02:00
Raphael Taylor-Davies 91a45fd380
feat: simplify shutdown (#1502)
* feat: simplify shutdown

* chore: fix lint

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-17 11:50:14 +00:00
Marco Neumann 4299371cf2 refactor: remove some code 2021-05-17 12:32:48 +02:00
Marco Neumann 840c11dab2 feat: wire up catalog preservation write path
Required a bit of refactoring:

- Add an extra layer between DB an catalog which is the "preserved
  catalog" wrapper. This is required to make the ownership model
  somewhat sane, because during the read operations the "preserved
  catalog" is going to act on the in-mem catalog.
- Move "parquet file written" logic into binding `preserved catalog <->
  catalog state`, so we have a single place where new parquet files are
  announced. For now this only works for chunks that are already known
  (i.e. the writing->written transation when coming from read buffer),
  however in the next PR this will be extended to also handle totally
  new parquet files during transaction playback.

**NOTE: This does NOT include the read path yet!**

Issue: #1382.
2021-05-17 11:33:22 +02:00
Andrew Lamb 07db4932ee
refactor: rename data_types/src/chunk.rs -> data_types/src/chunk_metadata.rs (#1500) 2021-05-15 10:18:01 +00:00
Raphael Taylor-Davies f9178dbb5f
feat: push metrics into catalog (#1488)
* feat: push metrics into catalog

* chore: minor cleanup

* fix: include db labels in chunk metric domains

* chore: fmt

* fix: don't allow dropping moving chunks

* chore: further tweaks

* chore: review feedback

* feat: use new_unregistered() for metric instruments instead of default

* chore: use &[KeyValue] instead of &Vec<KeyValue>

* refactor: make GauageValue non default constructible
2021-05-14 17:37:39 +00:00
kodiakhq[bot] fdc8461c7f
Merge branch 'main' into cn/wb-clock 2021-05-14 13:00:06 +00:00
Marko Mikulicic 35c2ca17fc
fix: Add ingest_fields_total
ingest_lines_total count lines (which apparently are the same as points, quite confusingly)

No yaks harmed in the making of this PR.

(NOTE: the code around metric, especially dealing with happy and error paths is very painful;
to be done in another PR)
2021-05-13 17:55:07 +02:00
Nga Tran 9583636748 feat: we now can read parquet files form all kind of object stores 2021-05-12 18:05:34 -04:00
Carol (Nichols || Goulding) 8be95856ab test: Add a test with multiple threads using a process clock 2021-05-12 13:31:26 -04:00
Carol (Nichols || Goulding) cecb4afc58 docs: Add some documentation on the assumptions around this design 2021-05-12 13:31:26 -04:00
Carol (Nichols || Goulding) b3fb61a0b3 refactor: Rename now_nanos to system_clock_now for clarity 2021-05-12 13:31:26 -04:00
Carol (Nichols || Goulding) 425aacc391 refactor: Extract ProcessClock into its own type 2021-05-12 13:31:26 -04:00
Carol (Nichols || Goulding) b749353d21 refactor: Use a compare_exchange loop instead of Arc Mutex 2021-05-12 10:58:08 -04:00
Carol (Nichols || Goulding) 5dfd152549 test: Use the now_nanos helper function more in tests 2021-05-12 10:58:08 -04:00
Carol (Nichols || Goulding) f28c9ae04c docs: Add unit and semantic information about the process clock 2021-05-12 10:58:08 -04:00
Carol (Nichols || Goulding) 513d4731be feat: Add a process clock to Db and use it for Sequenced Entries
Connects to #1157.
2021-05-12 10:58:06 -04:00
Carol (Nichols || Goulding) f98807936d test: Some tests don't call await, so they don't need to be async 2021-05-12 10:57:05 -04:00
Edd Robinson 696e4e0cfd fix: ensure metrics not overwriting 2021-05-11 20:57:31 +01:00
Raphael Taylor-Davies 4409d2c8af
feat: instrument catalog locks (#1464)
* feat: instrument catalog locks (#1355)

* chore: add metrics test

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-11 18:59:11 +00:00
Andrew Lamb 9d0c3a2b1a
refactor: Remove multi-table per chunk code in MUB (#1471)
* refactor: Remove multi-table per chunk code in MUB

* fix: clippy

* fix: bench build

* fix: merge conflicts
2021-05-11 17:49:07 +00:00
Raphael Taylor-Davies d1da954fe4
feat: don't store encoded strings twice in RLE dictionaries (#1469) 2021-05-11 15:22:25 +00:00
Edd Robinson 3622a92c8b feat: wire in rb column metrics 2021-05-11 13:00:52 +01:00
Marco Neumann 795f5bfcb7 refactor: make `StatValues::{min,max}` optional + handle NaNs
This will allow us to:

- handle all-NULL columns correctly
- be in-line with Parquet (where min/max are optional)
- handle NaNs at least somewhat sane (they do not "poison" stats
  anymore)
2021-05-10 17:12:25 +02:00
Andrew Lamb f037c1281a
feat: Calculate all system tables "on demand" (#1452)
* feat: compute system.columns table on demand

* feat: compute system.chunk_columns on demand

* feat: compute system.operations on demand

* fix: fixup schemas

* fix: Log errors

* fix: clippy

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-10 14:43:55 +00:00
Marko Mikulicic 9f5350a6c5
fix: Load only databases for which a config exists
Closes #1450
2021-05-10 13:14:22 +02:00
Nga Tran c6b933eb63 chore: merge main to branch 2021-05-07 18:40:17 -04:00
Nga Tran 971500681f refactor: address Andrew's and Carol's comment 2021-05-07 17:33:19 -04:00
Nga Tran ba015ee4df refactor: clean up and add comments 2021-05-07 09:31:41 -04:00
Edd Robinson eae3fec571 feat: wire up regex UDF as predicate filter expr 2021-05-07 13:44:51 +01:00
Andrew Lamb b5ea71f45f
feat: Expose the storage usage for each column in system.chunk_columns (#1441)
* feat: Expose the storage usage for each column in system.chunk_columns

* fix: fixup logical conflicts

* refactor: move coalsce logic into the read buffer

* fix: Update system_tables to not use coalese

* fix: Improve comments

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-05-07 12:36:49 +00:00
Raphael Taylor-Davies 9320f59de0
feat: add shard sink indirection (#1447)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-07 11:04:51 +00:00
Andrew Lamb d7253c72c0
feat: Only calculate system.chunks table "on demand" (#1446)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-07 10:26:56 +00:00
Carol (Nichols || Goulding) febc1538ff
chore: Update Rust version (#1445)
* chore: Update Rust version

* refactor: Make struct constructor field orderings consistent

Sometimes I changed the struct definition, sometimes changed the struct
construction instance, depending on consistency with code around each
(other similar structs, function argument orders, etc)

More info: https://rust-lang.github.io/rust-clippy/master/index.html#inconsistent_struct_constructor

* refactor: Use flatten where appropriate

One instance is a false positive with a clippy bug.

More info:

- https://rust-lang.github.io/rust-clippy/master/index.html#filter_map_identity
- https://rust-lang.github.io/rust-clippy/master/index.html#manual_flatten

* refactor: Use Option map instead of match

More info: https://rust-lang.github.io/rust-clippy/master/index.html#manual_map

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 22:07:10 +00:00
Nga Tran 55bf848bd2 feat: Now we can query directly from files in object store 2021-05-06 18:02:17 -04:00
Raphael Taylor-Davies 7f6b11266d
feat: instrument catalog locks (#1355) (#1439)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 17:09:52 +00:00
Raphael Taylor-Davies 44de42906f
refactor: use Arc<str> instead of Arc<String> (#1442) 2021-05-06 17:05:08 +00:00
Raphael Taylor-Davies 49c0b8b90c
feat: pull-based metrics (#1355) (#1414)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 15:54:30 +00:00
Raphael Taylor-Davies 216903a949
refactor: move protobuf conversion logic to generated_types (#1437)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 15:49:27 +00:00
Andrew Lamb 884baf7329
feat: add column_type and influxdb_column_type, remove row_count from system.columns (#1415)
* feat: add column_type and influxdb_column_type, remove row_count from system.columns

* fix: update tests

* fix: more test update

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: fmt

* fix: copy/paste type conversion to avoid cross dependency between data_types and internal_types

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-05-06 12:59:30 +00:00
Marko Mikulicic 578dc0db25
feat: Add more logs to shed light on the curious incident with missing metrics in the nighttime 2021-05-06 14:42:48 +02:00
Raphael Taylor-Davies 10f89a3e8d
refactor: split entry out into separate crate (#1428)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 11:36:23 +00:00
Nga Tran a5c92fae8a chore: merge main to branch 2021-05-05 13:48:42 -04:00
Raphael Taylor-Davies 411cf134e9
refactor: explode arrow_deps (#1425)
* refactor: explode arrow_deps

* chore: workaround doctest bug
2021-05-05 16:59:12 +00:00
kodiakhq[bot] 4395ede244
Merge branch 'main' into debug-chunk-metrics 2021-05-05 15:43:32 +00:00
Marko Mikulicic 2b0d7cfb91
feat: Add debug to update_chunk_state metrics 2021-05-05 17:37:57 +02:00
Nga Tran fcb37a0b1d feat: more testing scenarios for quering parquet files 2021-05-05 10:57:02 -04:00
Carol (Nichols || Goulding) 4a64e22e64 refactor: Use trait object and deref instead of cloning Arc in tests 2021-05-05 10:55:12 -04:00
Carol (Nichols || Goulding) e32fa43a53 docs: Add note about implication of write buffer errors 2021-05-05 10:55:12 -04:00
Carol (Nichols || Goulding) 7d5c988fba feat: Actually route SequencedEntry to the Write Buffer, if present
Connects to #1157.

Rearrange some code and comments to be consistent with the design. Make
some more places not care whether they're getting an owned or borrowed
SequencedEntry.
2021-05-05 10:55:11 -04:00
Carol (Nichols || Goulding) 54c5f984d5 fix: Use stdlib's path manipulation rather than format
The syntax highlighting in my editor broke because of the unmatched
double quote, which got me to look a bit closer at this test. These
tests would have failed on Windows.
2021-05-05 10:55:11 -04:00
Carol (Nichols || Goulding) 231abd221f refactor: Extract a TestDbBuilder 2021-05-05 10:55:11 -04:00
Carol (Nichols || Goulding) 62dfb47825 refactor: Reorganize test imports 2021-05-05 10:55:11 -04:00
Marco Neumann 9e61b470e7 feat: change MemoryStream to accept multiple record batches 2021-05-05 13:29:16 +02:00
Marco Neumann 34754ebcdb refactor: move MemoryStream to arrow_deps 2021-05-05 13:29:16 +02:00
Edd Robinson 733d502350 refactor: fix tests 2021-05-04 18:38:42 +01:00
Edd Robinson 9aa144e0f4 feat: add per-stage current chunk storage 2021-05-04 17:43:53 +01:00
Andrew Lamb 3b7c5ac350
fix(storage rpc): do not send back tags with empty values (#1403) 2021-05-04 10:35:24 +00:00
Marko Mikulicic b579ef8646
feat: Add jemalloc stats 2021-05-03 12:10:48 +02:00
kodiakhq[bot] 3c5595d046
Merge branch 'main' into ntran/unload_chunks 2021-04-30 22:02:38 +00:00
Paul Dix 979f5f9347 refactor: write buffer to use sequenced entry and new segment
This refactors the write buffer to use the sequenced entry structure and the new segment definition. It removes the old replicated write and write_buffer.fbs.

Finally, it updates the SequencedEntry wrapper type around the Flatbuffer structure to be a trait so that SequencedEntry can be initialized from a borrowed Flatbuffer or an owned Vec<u8>.

How writes go into segments in the buffer and any kind of validation will likely have to be updated based on what kinds of guarantees we want to make in the buffer. However, that should probably come after we've rethought the design a bit around the new layout of chunks in the Parquet persistence.
2021-04-30 17:00:23 -04:00
Raphael Taylor-Davies a967ebfabd
refactor: rename closing to closed (#1396)
* refactor: rename closing to closed

* refactor: further renames
2021-04-30 20:59:45 +00:00
Nga Tran 34a3388a49 feat: unload chunks from read buffer but keep them in object store 2021-04-30 16:12:02 -04:00
Raphael Taylor-Davies c2f7e7efea
feat: warn on dropping from open partition (#1395)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-30 18:56:49 +00:00
Edd Robinson ebe671e59e refactor: include default labels 2021-04-30 14:03:09 +01:00
Andrew Lamb 40b9b09cdc
refactor: rename assert_table_eq to assert_batches_eq (#1368) 2021-04-30 10:51:08 +00:00
Nga Tran c9b33c6b7d chore: Merge branch 'main' into ntran/test_query_parquets 2021-04-29 14:22:34 -04:00
Nga Tran f89d56af8a chore: merge mian to branch 2021-04-29 14:14:04 -04:00
Andrew Lamb 2d643d4407
fix: set nullability correctly for system tables (#1387) 2021-04-29 18:09:27 +00:00
Nga Tran e87973babe refactor: address review comments 2021-04-29 13:15:43 -04:00
Edd Robinson ea476c86be feat: add chunk size histogram 2021-04-29 17:10:57 +01:00
Andrew Lamb 5ef9be4abd
refactor: use RecordBatch::try_from_iter to make system tables (#1374)
* refactor: use RecordBatch::try_from_iter to make system tables

* fix: clippy
2021-04-29 15:32:35 +00:00
Andrew Lamb 0788892413
feat: add row_count to system.chunks and Chunk managment API (#1373)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-29 13:53:25 +00:00
Marko Mikulicic 3672a20570
feat: Add ingest_entries_bytes_total counter 2021-04-29 14:03:05 +02:00
Raphael Taylor-Davies 262bf446ab
feat: change background worker to use backoff instead of poll (#1339) (#1347)
* feat: change background worker to use backoff instead (#1339)

* chore: fix comment

* chore: fix tests

* chore: review comments

* chore: review feedback
2021-04-29 11:10:51 +00:00
Andrew Lamb b1924d10b4
fix: Only show current database's system.operations table (#1310)
* test: add test for operations.system_tables

* fix: only show operations for current database

* fix: update test

* fix: improve test

* refactor: filter in Schema provider rather than in job tracker

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-28 21:06:19 +00:00
Nga Tran 402d9c748c chore: cargo fmt 2021-04-28 16:52:52 -04:00
Nga Tran 0ea8861a65 chore: fix merge that causes test failures 2021-04-28 16:38:29 -04:00
Nga Tran 2a2760bd18 feat: complete tests where data in both RUB and OS 2021-04-28 16:14:07 -04:00
Nga Tran 140d96dbea feat: tests ffor loading data to object store and make sure twe still query read buffer 2021-04-28 15:59:17 -04:00
Carol (Nichols || Goulding) 9aefcd216f fix: Validate that ClockValue is never 0 2021-04-28 13:54:55 -04:00
kodiakhq[bot] dcc732b154
Merge branch 'main' into cn/server-id-fix 2021-04-28 17:19:29 +00:00
Carol (Nichols || Goulding) 2f4d7189ff fix: Validate ServerId when creating structs from flatbuffers
When we get the flatbuffers, we won't have the server ID in addition to
the flatbuffers-- it's in the flatbuffers. But we want to validate the
`ServerId` once when the `SequencedEntry` is created so that its
`server_id` method can assume it has a valid `ServerId`.
2021-04-28 13:06:12 -04:00
Edd Robinson c4259d5336 refactor: update benchmarks 2021-04-28 16:11:57 +00:00
Edd Robinson a9ef604ef6 perf: avoid using channels for query execution
Pre-sized channels get full when the results to send over them are larger than the capacities. This causes significant runtime overhead and slows down query performance.

This commit removes the intermediate channels. The potential downside to this approach is there may be more buffering which could increase memory usage during query and also block a thread for longer periods of time.
2021-04-28 16:11:57 +00:00
Edd Robinson d6c0f40630 test: add read_group benchmark 2021-04-28 16:11:57 +00:00
Edd Robinson 5dfe312867 test: add benchmark for influxrpc read_filter 2021-04-28 16:11:57 +00:00
Marko Mikulicic b6054f9df9
feat: Make background task period configurable 2021-04-28 15:47:31 +02:00
Raphael Taylor-Davies 9b57214344
feat: move closing chunks as soon as possible (#1329)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-28 07:50:15 +00:00
Raphael Taylor-Davies 6bdc153361
feat: sort RUB (read buffer) chunks (#1308)
* feat: sort chunks before upserting to read buffer (#1216)

* chore: review feedback

* chore: fix merge conflict

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-27 16:50:48 +00:00
Raphael Taylor-Davies 7ca1da3fcd
feat: pushdown table and partition key predicates to catalog (#736) (#1327)
* feat: catalog predicate pushdown (#736)

* chore: fix lints

* chore: review comments

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-27 15:31:47 +00:00
Marco Neumann 91bccdfca3 ci: pass `--document-private-items` to `cargo doc` 2021-04-27 15:42:07 +02:00
kodiakhq[bot] e78a81ee59
Merge branch 'main' into crepererum/ci_doc 2021-04-27 12:03:04 +00:00
kodiakhq[bot] c3cb86fc1a
Merge branch 'main' into crepererum/issue1296 2021-04-27 11:58:10 +00:00
Marco Neumann eddc9319ff docs: deny broken intradoc links 2021-04-27 13:22:28 +02:00
Raphael Taylor-Davies 20117de078
feat: string dictionary encoding (#1220) (#1262)
* feat: string dictionary encoding (#1220)

* chore: review comments

* chore: fix lint

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-27 09:36:58 +00:00
Marco Neumann c8594ce743 refactor: chunks shall always have data (= 1 table)
Chunks now always have "data" (aka exactly 1 table including
schema/columns). Open chunks can only be created with data. Rollovers do
NOT create open chunks anymore (this is now only done for incoming
data).

Closes #1296.
2021-04-27 10:28:57 +02:00
Andrew Lamb 35bc09ef7c
refactor: use `FromIter` rather than array builders to create system table arrays (#1246)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-26 19:47:11 +00:00
Carol (Nichols || Goulding) 272cdb85ce fix: Use the ServerId type everywhere, for writing, querying, anything 2021-04-26 18:44:32 +00:00
Carol (Nichols || Goulding) b8face3335 refactor: Organize use statements 2021-04-26 18:44:32 +00:00
Jake Goulding 67f5ad841d refactor: Introduce ServerId and CurrentServerId types 2021-04-26 18:44:32 +00:00
Raphael Taylor-Davies 0a835436ac
feat: use bitmasks within MUB (#1274) (#1289)
* feat: use bitmasks within MUB (#1274)

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-26 18:00:16 +00:00
Andrew Lamb e33af0c084 fix: remove wait_for_complete 2021-04-26 16:59:36 +00:00
Andrew Lamb fcf5c304a6 fix: fixup merge conflicts 2021-04-26 16:59:36 +00:00
Andrew Lamb 348f38c2c9 feat: add status column 2021-04-26 16:59:36 +00:00
Andrew Lamb 9c74074095 feat: List background operations as system.operations table 2021-04-26 16:59:36 +00:00
Edd Robinson faec98eab9 refactor: remove time column from row group 2021-04-26 09:51:06 +00:00
Edd Robinson dd2bee3343 fix: don't over count new chunks 2021-04-23 20:25:37 +01:00
Edd Robinson 552409ea70 feat: add metrics for chunk lifecycle 2021-04-23 17:13:43 +00:00
Edd Robinson 0e3b7bff49 refactor: plumb metrics registry to catalog 2021-04-23 17:13:43 +00:00
Edd Robinson eacfe9af6a refactor: address PR feedback 2021-04-23 15:58:48 +00:00
Edd Robinson d3218802ab refactor: more API metric coverage 2021-04-23 15:58:48 +00:00
Edd Robinson 97b2369140 refactor: swap existing metrics for THE NEW WAY 2021-04-23 15:58:48 +00:00
Marco Neumann f12f1eeef7 refactor: make chunks per-table
This changes the hierarchy from

```
database -> partition -> chunk -> table
```

to

```
database -> partition -> table -> chunk
```

Only the high-level APIs are changed for now. The chunk states (like
MutableBuffer and ReadBuffer) still multiplex tables, although they will
always only get a single table assigned (or no table if no data was
presented yet).

Closes #1256.
2021-04-23 09:18:55 +02:00
Raphael Taylor-Davies 74c25f541d
feat: fast MUB dictionary arrow conversion (#1273)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-22 20:31:44 +00:00
Marko Mikulicic 83d6550316 feat: Implement write_entry_downstream 2021-04-21 20:50:46 +00:00
Carol (Nichols || Goulding) 88ca1a5245 fix: Rename wal.fbs to write_buffer.fbs 2021-04-21 17:43:03 +00:00
Carol (Nichols || Goulding) 80995afb70 fix: Change WAL to Write Buffer in comments and documentation 2021-04-21 17:43:03 +00:00
Carol (Nichols || Goulding) 5f024401a4 refactor: Rename data_types::wal module to data_types::write_buffer 2021-04-21 17:43:03 +00:00
Carol (Nichols || Goulding) 46f9360c58 fix: Rename WAL_DIR to WRITE_BUFFER_DIR and redefine it as wb, not wal 2021-04-21 17:43:03 +00:00
Carol (Nichols || Goulding) 8baeaab8ca refactor: Rename Db.wal_buffer to write_buffer 2021-04-21 17:43:03 +00:00
Carol (Nichols || Goulding) 4bfcb68536 refactor: Remove a WalError that doesn't appear to be used 2021-04-21 17:43:03 +00:00
Carol (Nichols || Goulding) 0677c544bb refactor: Rename WalBufferRollover to WriteBufferRollover 2021-04-21 17:43:03 +00:00
Carol (Nichols || Goulding) ee45db402b refactor: Rename WalBufferConfig to WriteBufferConfig 2021-04-21 17:43:03 +00:00