9d15798288
fix: Address or allow Clippy warnings new with Rust 1.54
2021-07-30 09:59:59 -04:00
4689b5e4e5
refactor: Remove first/last write times from MUB chunks
2021-07-28 11:02:57 -04:00
5fb3e00f2a
fix: Properly record total_count and null_count in statistics ( #2103 )
...
* fix: Properly record total_count and null_count in statistics
* fix: fix statistics calculation in mutable_buffer
* refactor: expose null counts in read_buffer
* refactor: expose null_count in parquet_file
* fix: update server crate tests
* fix: update query_tests tests
* docs: tweak comments
* refactor: Use storage_stats rather than adding `null_count`
* refactor: rename test data field for clarity
* fix: fixup merge conflicts
* refactor: rename initial_non_null_count to initial_total_count
* refactor: caculate null_count as row_count - to_add
2021-07-26 18:13:36 +00:00
0b88deea43
refactor: don't pass sequence to MUB ( #2107 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-26 14:40:39 +00:00
446af5eb15
fix: consistent write timestamps ( #2104 )
...
* fix: consistent write timestamps
* chore: fix benchmarks
2021-07-23 18:04:15 +00:00
71f3f1aba2
Merge branch 'main' into cn/refactorings
2021-07-22 19:44:18 +00:00
01c79f1a1a
fix: Print all timestamps using RFC3339 format ( #2098 )
...
* fix: Use IOx pretty printer rather than arrow pretty printer
* chore: update tests in the query crate
* chore: update influxdb_iox tests
* chore: Update end to end tests
* chore: update query_tests
* chore: update mutable_buffer tests
* refactor: update parquet_file tests
* refactor: update db tests
* chore: update kafka integration test output
* fix: merge conflict
2021-07-22 19:04:52 +00:00
799406be72
refactor: Have MUB chunk snapshots hold a TableSummaryAndTimes rather than only ColumnSummaries
...
This way we can have the first/last write times on MUB chunk snapshots
without needing to deconstruct/reconstruct TableSummaries as it was
doing.
2021-07-22 11:35:23 -04:00
956647d966
docs: Fix an outdated comment
2021-07-22 11:35:23 -04:00
0a724878e6
refactor: Organize uses
2021-07-22 11:15:18 -04:00
4da8a16c18
chore: update to arrow 5.0 and master datafusion ( #2049 )
...
* chore: update to arrow 5.0 and master datafusion
* fix: Update test for change in object size
2021-07-19 12:49:51 +00:00
1d00fa2fd8
refactor: track memory metrics in catalog ( #1995 )
...
* refactor: track memory metrics in catalog
* chore: update comment
2021-07-14 16:23:00 +00:00
7ccbab8c90
feat: Make a TableSummaryAndTimes to use to slowly replace TableSummary
...
And use TableSummaryAndTimes with the mutable buffer chunks when turning
them into catalog chunks.
It's proving too big to switch over everything using TableSummary at
once, so this will let us switch over more incrementally.
2021-07-14 10:28:12 -04:00
d35b74c226
fix: Fix doc build warnings ( #1945 )
...
* fix: Fix doc build warnings
* refactor: add deny bare_urls to crates
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-13 08:03:42 +00:00
b9a6a11b34
feat: Record time of first/last write on MBChunk
2021-07-08 16:39:35 -04:00
22495dd355
fix: Take a TableBatch in the MBChunk constructor
...
Thus ensuring all MBChunks will have data in them.
2021-07-08 16:39:35 -04:00
57501e7f59
refactor: Remove a test function that's only slightly different from another test function
2021-07-08 16:39:35 -04:00
2f9cf857af
fix: MBChunk.write_columns doesn't need to be pub
2021-07-08 16:39:35 -04:00
74c0a6cb00
fix: Arrange use statements so rustfmt can manage their order
2021-07-08 16:39:02 -04:00
4ca2d3e148
chore: move persistence windows related code into own crate
...
The entire persistence windows data structures (including the
checkpoints) have nothing to do with the mutable buffer per se. So lets
move them into their own crate. This also makes `parquet_file` not
longer depend on `mutable_buffer`.
2021-07-05 10:23:58 +02:00
cdab1bed05
feat: persist part+db checkpoint in parquets and catalog
...
This will be required for replay on server startup.
2021-07-05 09:42:46 +02:00
5fe49aa017
feat: add flush guard to PersistenceWindows ( #1883 )
...
* feat: add flush guard to PersistenceWindows
* docs: Update comments based on code review
* fix: fmt
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2021-07-02 20:15:33 +00:00
0779b0d9bd
feat: add gRPC listener for new write protocol ( #1842 )
...
* feat: add gRPC listener for new write protocol
* chore: clippy happy
* chore: lint
* chore: cargo fmt --all
* chore: cargo clippy
* chore: protobuf-lint
* chore: more formatting
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-01 16:15:12 +00:00
043890369f
refactor: make `MinMaxSequence` safer to use
2021-06-30 16:37:48 +02:00
e10173124f
Merge branch 'main' into crepererum/mb_checkpoint
2021-06-29 15:25:24 +00:00
3ee96c4618
fix: Do not sequence local writes (avoid panic under load) ( #1826 )
...
* fix: Do not sequence local writes
* fix: Update server/src/db.rs
Co-authored-by: Edd Robinson <me@edd.io>
* fix: review comments
* fix: restore passing sequence information down to mutable buffer
* fix: store min/max times even when there are no sequence numbers
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-29 10:39:37 +00:00
f145c5d760
feat: add ability to optimize record batch dictionaries ( #1827 ) ( #1830 )
...
* feat: add ability to optimize record batch dictionaries (#1827 )
* chore: correct capacity
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-29 09:23:56 +00:00
32c683ddc1
fix: remove shared dictionary concept from ChunkSnapshot ( #1570 ) ( #1820 ) ( #1825 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-28 15:26:49 +00:00
7b12426602
feat: implement persistence window checkpointing
2021-06-28 11:34:41 +02:00
25a912d4eb
docs: fix mutable buffer intro docs
2021-06-28 11:34:41 +02:00
af93bb6573
chore: PR cleanup
2021-06-25 15:45:00 -04:00
de236c5a6f
feat: update persistence windows to support late arrival less than 30 seconds
2021-06-25 15:34:11 -04:00
435b4b6a94
feat: add persistence windows to partition and update on write
...
This brings the persistence windows into the catalog partition. It adds a helper method on TableBatch to get the min and max times for a given write. Finally, it adds this logic to the db to update persistence windows on every write while the partition write lock is being held.
2021-06-25 15:34:11 -04:00
bed6ec8c31
feat: Handle merging chunks that have different schemas ( #1761 )
...
* feat: Handle merging chunks that have different schemas
* test: print out original (non deduplicated) data in tests
2021-06-21 15:52:13 +00:00
634c9b4c66
refactor: delete unused code ( #1759 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-19 08:15:33 +00:00
de67bd3efe
refactor: Remove PartitionChunk::table_schema ( #1756 )
...
* refactor: Remove PartitionChunk::table_schema
* docs: update comments
2021-06-18 16:13:16 +00:00
ecfabe3408
chore: address PR feedback, changed closed persistence window to dequeue.
2021-06-16 15:28:37 -04:00
6bc4b383a4
chore: update based on PR feedback, add more comments
2021-06-16 15:28:37 -04:00
802ecacf61
feat: Add persistence windows ingestion tracking
...
This adds a new module, persistence_windows, to the mutable buffer crate. Later PRs will add this into the mutable buffer chunk where it can be used to track when the lifecycle for persistence should be triggered.
2021-06-16 15:28:37 -04:00
a14e9ab27c
refactor: rename mutable_buffer::Chunk --> mutable_buffer::MBChunk ( #1711 )
...
* refactor: rename mutable_buffer::Chunk --> mutable_buffer::MBChunk
* fix: fmt
2021-06-14 13:35:20 +00:00
8c21eb79af
refactor: inline Table into mutable_buffer::chunk::Chunk ( #1704 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-14 13:04:06 +00:00
a614fef5bc
chore: remove more unused dependencies ( #1658 )
...
* chore: remove more unused deps
* refactor: move benchmarks into server_benchmarks crate
2021-06-09 10:17:20 +00:00
87297f7db4
Merge branch 'main' into cn/delete
2021-06-07 13:32:42 +00:00
1e7ef193a6
refactor: use field metadata to store influx types ( #1642 )
...
* refactor: use field metadata to store influx types
make SchemaBuilder non-consuming
* chore: remove unused variants
* chore: fix lints
2021-06-07 13:26:39 +00:00
4d6569583e
fix: Partially restore SequencedEntry as Entry+sequencer_id+sequence_num
2021-06-04 14:40:19 -04:00
32c6ed1f34
refactor: More cleanup related to multi-table chunks ( #1604 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-02 17:00:23 +00:00
d8fbb7b410
refactor: Remove last vestiges of multi-table chunks from PartitionChunk API ( #1588 )
...
* refactor: Remove last vestiges of multi-table chunks from PartitionChunk API
* fix: remove test that can no longer fail
* fix: update tests + code review comments
* fix: clippy
* fix: clippy
* fix: restore test_measurement_fields_error test
2021-06-01 16:12:33 +00:00
64328dcf1c
feat: cache schema on catalog chunks too ( #1575 )
2021-06-01 12:42:46 +00:00
00e735ef0d
chore: remove unused dependencies ( #1583 )
2021-05-29 10:31:57 +00:00
db432de137
feat: add distinct count to StatValues ( #1568 )
2021-05-28 17:41:34 +00:00
Carol (Nichols || Goulding)
Andrew Lamb
Raphael Taylor-Davies
kodiakhq[bot]
Marco Neumann
Jacob Marble
Paul Dix