Commit Graph

155 Commits (62d3305923c9e7d1af4752f3e805c5fed7123667)

Author SHA1 Message Date
kodiakhq[bot] e10173124f
Merge branch 'main' into crepererum/mb_checkpoint 2021-06-29 15:25:24 +00:00
Andrew Lamb 3ee96c4618
fix: Do not sequence local writes (avoid panic under load) (#1826)
* fix: Do not sequence local writes

* fix: Update server/src/db.rs

Co-authored-by: Edd Robinson <me@edd.io>

* fix: review comments

* fix: restore passing sequence information down to mutable buffer

* fix: store min/max times even when there are no sequence numbers

Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-29 10:39:37 +00:00
Raphael Taylor-Davies f145c5d760
feat: add ability to optimize record batch dictionaries (#1827) (#1830)
* feat: add ability to optimize record batch dictionaries (#1827)

* chore: correct capacity

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-29 09:23:56 +00:00
Raphael Taylor-Davies 32c683ddc1
fix: remove shared dictionary concept from ChunkSnapshot (#1570) (#1820) (#1825)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-28 15:26:49 +00:00
Marco Neumann 7b12426602 feat: implement persistence window checkpointing 2021-06-28 11:34:41 +02:00
Marco Neumann 25a912d4eb docs: fix mutable buffer intro docs 2021-06-28 11:34:41 +02:00
Paul Dix af93bb6573 chore: PR cleanup 2021-06-25 15:45:00 -04:00
Paul Dix de236c5a6f feat: update persistence windows to support late arrival less than 30 seconds 2021-06-25 15:34:11 -04:00
Paul Dix 435b4b6a94 feat: add persistence windows to partition and update on write
This brings the persistence windows into the catalog partition. It adds a helper method on TableBatch to get the min and max times for a given write. Finally, it adds this logic to the db to update persistence windows on every write while the partition write lock is being held.
2021-06-25 15:34:11 -04:00
Andrew Lamb bed6ec8c31
feat: Handle merging chunks that have different schemas (#1761)
* feat: Handle merging chunks that have different schemas

* test: print out original (non deduplicated) data in tests
2021-06-21 15:52:13 +00:00
Andrew Lamb 634c9b4c66
refactor: delete unused code (#1759)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-19 08:15:33 +00:00
Andrew Lamb de67bd3efe
refactor: Remove PartitionChunk::table_schema (#1756)
* refactor: Remove PartitionChunk::table_schema

* docs: update comments
2021-06-18 16:13:16 +00:00
Paul Dix ecfabe3408 chore: address PR feedback, changed closed persistence window to dequeue. 2021-06-16 15:28:37 -04:00
Paul Dix 6bc4b383a4 chore: update based on PR feedback, add more comments 2021-06-16 15:28:37 -04:00
Paul Dix 802ecacf61 feat: Add persistence windows ingestion tracking
This adds a new module, persistence_windows, to the mutable buffer crate. Later PRs will add this into the mutable buffer chunk where it can be used to track when the lifecycle for persistence should be triggered.
2021-06-16 15:28:37 -04:00
Andrew Lamb a14e9ab27c
refactor: rename mutable_buffer::Chunk --> mutable_buffer::MBChunk (#1711)
* refactor: rename mutable_buffer::Chunk --> mutable_buffer::MBChunk

* fix: fmt
2021-06-14 13:35:20 +00:00
Andrew Lamb 8c21eb79af
refactor: inline Table into mutable_buffer::chunk::Chunk (#1704)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-14 13:04:06 +00:00
Andrew Lamb a614fef5bc
chore: remove more unused dependencies (#1658)
* chore: remove more unused deps

* refactor: move benchmarks into server_benchmarks crate
2021-06-09 10:17:20 +00:00
kodiakhq[bot] 87297f7db4
Merge branch 'main' into cn/delete 2021-06-07 13:32:42 +00:00
Raphael Taylor-Davies 1e7ef193a6
refactor: use field metadata to store influx types (#1642)
* refactor: use field metadata to store influx types

make SchemaBuilder non-consuming

* chore: remove unused variants

* chore: fix lints
2021-06-07 13:26:39 +00:00
Carol (Nichols || Goulding) 4d6569583e fix: Partially restore SequencedEntry as Entry+sequencer_id+sequence_num 2021-06-04 14:40:19 -04:00
Andrew Lamb 32c6ed1f34
refactor: More cleanup related to multi-table chunks (#1604)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-02 17:00:23 +00:00
Andrew Lamb d8fbb7b410
refactor: Remove last vestiges of multi-table chunks from PartitionChunk API (#1588)
* refactor: Remove last vestiges of multi-table chunks from PartitionChunk API

* fix: remove test that can no longer fail

* fix: update tests + code review comments

* fix: clippy

* fix: clippy

* fix: restore test_measurement_fields_error test
2021-06-01 16:12:33 +00:00
Andrew Lamb 64328dcf1c
feat: cache schema on catalog chunks too (#1575) 2021-06-01 12:42:46 +00:00
Andrew Lamb 00e735ef0d
chore: remove unused dependencies (#1583) 2021-05-29 10:31:57 +00:00
Raphael Taylor-Davies db432de137
feat: add distinct count to StatValues (#1568) 2021-05-28 17:41:34 +00:00
Raphael Taylor-Davies d8f19348bf
feat: per-column dictionaries in MUB (#1570)
* feat: per-column dictionaries in MUB

* chore: fmt

* refactor: remove chunk-level dictionary

* chore: remove redundant sort

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-28 13:51:56 +00:00
Raphael Taylor-Davies 792bff07d1
feat: only store ChunkSnapshot in Closed state (#1560)
* feat: only store ChunkSnapshot in Closed state

* chore: review feedback

* feat: record MUB size as closed size

* chore: document column ordering assumption

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-27 10:36:47 +00:00
Raphael Taylor-Davies 4fcc04e6c9
chore: enable arrow prettyprint feature (#1566) 2021-05-27 10:28:14 +00:00
Raphael Taylor-Davies c03b8a3963
refactor: remove tables from ChunkSnapshot (#1295) (#1558) 2021-05-26 10:37:40 +00:00
Andrew Lamb 14ba25f86d
chore: Update datafusion and use released version of arrow crates (#1546)
* chore: Update datafusion and use released version of arrow crate

* fix: Update for change in API
2021-05-24 15:37:22 +00:00
Andrew Lamb ee06ca4c7d
refactor: remove MutableChunk::has_table (#1539)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-22 12:02:42 +00:00
Raphael Taylor-Davies 37880ee89a
refactor: store chunk IDs only in catalog (#1521)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-20 04:07:14 +00:00
Andrew Lamb c1ea6be101
fix: Add context to panic error on tools (#1501)
* fix: Add context to panic error on tools

* fix: log new unwrap that has snuck in

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-17 21:05:01 +00:00
Raphael Taylor-Davies f9178dbb5f
feat: push metrics into catalog (#1488)
* feat: push metrics into catalog

* chore: minor cleanup

* fix: include db labels in chunk metric domains

* chore: fmt

* fix: don't allow dropping moving chunks

* chore: further tweaks

* chore: review feedback

* feat: use new_unregistered() for metric instruments instead of default

* chore: use &[KeyValue] instead of &Vec<KeyValue>

* refactor: make GauageValue non default constructible
2021-05-14 17:37:39 +00:00
Raphael Taylor-Davies cfa8160644
fix: don't test snapshotting when disabled (#1496) (#1498) 2021-05-14 16:15:55 +00:00
Raphael Taylor-Davies 8ab71be5eb
feat: use PackedStringArray in mutable buffer (#1476)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-13 15:25:19 +00:00
Andrew Lamb 9d0c3a2b1a
refactor: Remove multi-table per chunk code in MUB (#1471)
* refactor: Remove multi-table per chunk code in MUB

* fix: clippy

* fix: bench build

* fix: merge conflicts
2021-05-11 17:49:07 +00:00
Raphael Taylor-Davies c85d1574eb
feat: move dictionary and bitset into arrow_utils (#1459)
* feat: move dictionary and bitset into arrow_utils

* chore: review feedback

* chore: remove redundant dictionary methods

* chore: consistent type parameter name in PackedStringArray

* chore: review comments

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-11 16:43:38 +00:00
Marco Neumann 795f5bfcb7 refactor: make `StatValues::{min,max}` optional + handle NaNs
This will allow us to:

- handle all-NULL columns correctly
- be in-line with Parquet (where min/max are optional)
- handle NaNs at least somewhat sane (they do not "poison" stats
  anymore)
2021-05-10 17:12:25 +02:00
Andrew Lamb b5ea71f45f
feat: Expose the storage usage for each column in system.chunk_columns (#1441)
* feat: Expose the storage usage for each column in system.chunk_columns

* fix: fixup logical conflicts

* refactor: move coalsce logic into the read buffer

* fix: Update system_tables to not use coalese

* fix: Improve comments

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-05-07 12:36:49 +00:00
Andrew Lamb 884baf7329
feat: add column_type and influxdb_column_type, remove row_count from system.columns (#1415)
* feat: add column_type and influxdb_column_type, remove row_count from system.columns

* fix: update tests

* fix: more test update

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: fmt

* fix: copy/paste type conversion to avoid cross dependency between data_types and internal_types

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-05-06 12:59:30 +00:00
Raphael Taylor-Davies 10f89a3e8d
refactor: split entry out into separate crate (#1428)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 11:36:23 +00:00
Raphael Taylor-Davies ca1c698fd0
chore: update hashbrown (#1430)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-05 22:32:46 +00:00
Raphael Taylor-Davies 411cf134e9
refactor: explode arrow_deps (#1425)
* refactor: explode arrow_deps

* chore: workaround doctest bug
2021-05-05 16:59:12 +00:00
Andrew Lamb 40b9b09cdc
refactor: rename assert_table_eq to assert_batches_eq (#1368) 2021-04-30 10:51:08 +00:00
Andrew Lamb 0788892413
feat: add row_count to system.chunks and Chunk managment API (#1373)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-29 13:53:25 +00:00
Carol (Nichols || Goulding) 9aefcd216f fix: Validate that ClockValue is never 0 2021-04-28 13:54:55 -04:00
Marco Neumann eddc9319ff docs: deny broken intradoc links 2021-04-27 13:22:28 +02:00
Raphael Taylor-Davies 20117de078
feat: string dictionary encoding (#1220) (#1262)
* feat: string dictionary encoding (#1220)

* chore: review comments

* chore: fix lint

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-27 09:36:58 +00:00