4da8a16c18
chore: update to arrow 5.0 and master datafusion ( #2049 )
...
* chore: update to arrow 5.0 and master datafusion
* fix: Update test for change in object size
2021-07-19 12:49:51 +00:00
1d00fa2fd8
refactor: track memory metrics in catalog ( #1995 )
...
* refactor: track memory metrics in catalog
* chore: update comment
2021-07-14 16:23:00 +00:00
7ccbab8c90
feat: Make a TableSummaryAndTimes to use to slowly replace TableSummary
...
And use TableSummaryAndTimes with the mutable buffer chunks when turning
them into catalog chunks.
It's proving too big to switch over everything using TableSummary at
once, so this will let us switch over more incrementally.
2021-07-14 10:28:12 -04:00
d35b74c226
fix: Fix doc build warnings ( #1945 )
...
* fix: Fix doc build warnings
* refactor: add deny bare_urls to crates
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-13 08:03:42 +00:00
b9a6a11b34
feat: Record time of first/last write on MBChunk
2021-07-08 16:39:35 -04:00
22495dd355
fix: Take a TableBatch in the MBChunk constructor
...
Thus ensuring all MBChunks will have data in them.
2021-07-08 16:39:35 -04:00
57501e7f59
refactor: Remove a test function that's only slightly different from another test function
2021-07-08 16:39:35 -04:00
2f9cf857af
fix: MBChunk.write_columns doesn't need to be pub
2021-07-08 16:39:35 -04:00
74c0a6cb00
fix: Arrange use statements so rustfmt can manage their order
2021-07-08 16:39:02 -04:00
4ca2d3e148
chore: move persistence windows related code into own crate
...
The entire persistence windows data structures (including the
checkpoints) have nothing to do with the mutable buffer per se. So lets
move them into their own crate. This also makes `parquet_file` not
longer depend on `mutable_buffer`.
2021-07-05 10:23:58 +02:00
cdab1bed05
feat: persist part+db checkpoint in parquets and catalog
...
This will be required for replay on server startup.
2021-07-05 09:42:46 +02:00
5fe49aa017
feat: add flush guard to PersistenceWindows ( #1883 )
...
* feat: add flush guard to PersistenceWindows
* docs: Update comments based on code review
* fix: fmt
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2021-07-02 20:15:33 +00:00
0779b0d9bd
feat: add gRPC listener for new write protocol ( #1842 )
...
* feat: add gRPC listener for new write protocol
* chore: clippy happy
* chore: lint
* chore: cargo fmt --all
* chore: cargo clippy
* chore: protobuf-lint
* chore: more formatting
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-01 16:15:12 +00:00
043890369f
refactor: make `MinMaxSequence` safer to use
2021-06-30 16:37:48 +02:00
e10173124f
Merge branch 'main' into crepererum/mb_checkpoint
2021-06-29 15:25:24 +00:00
3ee96c4618
fix: Do not sequence local writes (avoid panic under load) ( #1826 )
...
* fix: Do not sequence local writes
* fix: Update server/src/db.rs
Co-authored-by: Edd Robinson <me@edd.io>
* fix: review comments
* fix: restore passing sequence information down to mutable buffer
* fix: store min/max times even when there are no sequence numbers
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-29 10:39:37 +00:00
f145c5d760
feat: add ability to optimize record batch dictionaries ( #1827 ) ( #1830 )
...
* feat: add ability to optimize record batch dictionaries (#1827 )
* chore: correct capacity
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-29 09:23:56 +00:00
32c683ddc1
fix: remove shared dictionary concept from ChunkSnapshot ( #1570 ) ( #1820 ) ( #1825 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-28 15:26:49 +00:00
7b12426602
feat: implement persistence window checkpointing
2021-06-28 11:34:41 +02:00
25a912d4eb
docs: fix mutable buffer intro docs
2021-06-28 11:34:41 +02:00
af93bb6573
chore: PR cleanup
2021-06-25 15:45:00 -04:00
de236c5a6f
feat: update persistence windows to support late arrival less than 30 seconds
2021-06-25 15:34:11 -04:00
435b4b6a94
feat: add persistence windows to partition and update on write
...
This brings the persistence windows into the catalog partition. It adds a helper method on TableBatch to get the min and max times for a given write. Finally, it adds this logic to the db to update persistence windows on every write while the partition write lock is being held.
2021-06-25 15:34:11 -04:00
bed6ec8c31
feat: Handle merging chunks that have different schemas ( #1761 )
...
* feat: Handle merging chunks that have different schemas
* test: print out original (non deduplicated) data in tests
2021-06-21 15:52:13 +00:00
634c9b4c66
refactor: delete unused code ( #1759 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-19 08:15:33 +00:00
de67bd3efe
refactor: Remove PartitionChunk::table_schema ( #1756 )
...
* refactor: Remove PartitionChunk::table_schema
* docs: update comments
2021-06-18 16:13:16 +00:00
ecfabe3408
chore: address PR feedback, changed closed persistence window to dequeue.
2021-06-16 15:28:37 -04:00
6bc4b383a4
chore: update based on PR feedback, add more comments
2021-06-16 15:28:37 -04:00
802ecacf61
feat: Add persistence windows ingestion tracking
...
This adds a new module, persistence_windows, to the mutable buffer crate. Later PRs will add this into the mutable buffer chunk where it can be used to track when the lifecycle for persistence should be triggered.
2021-06-16 15:28:37 -04:00
a14e9ab27c
refactor: rename mutable_buffer::Chunk --> mutable_buffer::MBChunk ( #1711 )
...
* refactor: rename mutable_buffer::Chunk --> mutable_buffer::MBChunk
* fix: fmt
2021-06-14 13:35:20 +00:00
8c21eb79af
refactor: inline Table into mutable_buffer::chunk::Chunk ( #1704 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-14 13:04:06 +00:00
a614fef5bc
chore: remove more unused dependencies ( #1658 )
...
* chore: remove more unused deps
* refactor: move benchmarks into server_benchmarks crate
2021-06-09 10:17:20 +00:00
87297f7db4
Merge branch 'main' into cn/delete
2021-06-07 13:32:42 +00:00
1e7ef193a6
refactor: use field metadata to store influx types ( #1642 )
...
* refactor: use field metadata to store influx types
make SchemaBuilder non-consuming
* chore: remove unused variants
* chore: fix lints
2021-06-07 13:26:39 +00:00
4d6569583e
fix: Partially restore SequencedEntry as Entry+sequencer_id+sequence_num
2021-06-04 14:40:19 -04:00
32c6ed1f34
refactor: More cleanup related to multi-table chunks ( #1604 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-02 17:00:23 +00:00
d8fbb7b410
refactor: Remove last vestiges of multi-table chunks from PartitionChunk API ( #1588 )
...
* refactor: Remove last vestiges of multi-table chunks from PartitionChunk API
* fix: remove test that can no longer fail
* fix: update tests + code review comments
* fix: clippy
* fix: clippy
* fix: restore test_measurement_fields_error test
2021-06-01 16:12:33 +00:00
64328dcf1c
feat: cache schema on catalog chunks too ( #1575 )
2021-06-01 12:42:46 +00:00
00e735ef0d
chore: remove unused dependencies ( #1583 )
2021-05-29 10:31:57 +00:00
db432de137
feat: add distinct count to StatValues ( #1568 )
2021-05-28 17:41:34 +00:00
d8f19348bf
feat: per-column dictionaries in MUB ( #1570 )
...
* feat: per-column dictionaries in MUB
* chore: fmt
* refactor: remove chunk-level dictionary
* chore: remove redundant sort
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-28 13:51:56 +00:00
792bff07d1
feat: only store ChunkSnapshot in Closed state ( #1560 )
...
* feat: only store ChunkSnapshot in Closed state
* chore: review feedback
* feat: record MUB size as closed size
* chore: document column ordering assumption
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-27 10:36:47 +00:00
4fcc04e6c9
chore: enable arrow prettyprint feature ( #1566 )
2021-05-27 10:28:14 +00:00
c03b8a3963
refactor: remove tables from ChunkSnapshot ( #1295 ) ( #1558 )
2021-05-26 10:37:40 +00:00
14ba25f86d
chore: Update datafusion and use released version of arrow crates ( #1546 )
...
* chore: Update datafusion and use released version of arrow crate
* fix: Update for change in API
2021-05-24 15:37:22 +00:00
ee06ca4c7d
refactor: remove MutableChunk::has_table ( #1539 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-22 12:02:42 +00:00
37880ee89a
refactor: store chunk IDs only in catalog ( #1521 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-20 04:07:14 +00:00
c1ea6be101
fix: Add context to panic error on tools ( #1501 )
...
* fix: Add context to panic error on tools
* fix: log new unwrap that has snuck in
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-17 21:05:01 +00:00
f9178dbb5f
feat: push metrics into catalog ( #1488 )
...
* feat: push metrics into catalog
* chore: minor cleanup
* fix: include db labels in chunk metric domains
* chore: fmt
* fix: don't allow dropping moving chunks
* chore: further tweaks
* chore: review feedback
* feat: use new_unregistered() for metric instruments instead of default
* chore: use &[KeyValue] instead of &Vec<KeyValue>
* refactor: make GauageValue non default constructible
2021-05-14 17:37:39 +00:00
cfa8160644
fix: don't test snapshotting when disabled ( #1496 ) ( #1498 )
2021-05-14 16:15:55 +00:00
Andrew Lamb
Raphael Taylor-Davies
Carol (Nichols || Goulding)
Marco Neumann
Jacob Marble
kodiakhq[bot]
Paul Dix